Soure Code

C# webbrowser control - Synchronization for Page navigation/loading

We will face lot of difficulties/errors if we are not handling page synchronization properly when using .NET webbrowser control for scrapping/crawling web pages.

(i-e) We need to write a code to start other activities only when page navigation is completely done.

We can use the below function "waitTillLoad()" for this synchronization purpose.

It will wait till the browser readystate becomes "complete".

Since, initally the readystate will be "complete" there is a possibility of incorrectly exiting this function even before starting new page loading.

So to avoid this issue we have enhanced the function to wait for non-complete status before waiting for complete status.

(i-e) Page loading should occur only after stating the page navigation.

We need to mention timeout period (waittime), as the function may fall into infinite loop if we are calling it two times without initiating any further page navigation.

We can use the same function with little modifications in vb.net also.
It will be more useful and also I hope it will be more reliable as we are using it in many tools and applications for long time.

I can say that it is very essential if you are using webbrowser control for doing any page scrapping and web crawling.



private void waitTillLoad()
{
WebBrowserReadyState loadStatus;
//wait till beginning of loading next page
int waittime = 100000;
int counter = 0;
while (true)
{
loadStatus = webBrowser1.ReadyState;
Application.DoEvents();

if((counter > waittime)(loadStatus == WebBrowserReadyState.
           Uninitialized)  (loadStatus == WebBrowserReadyState.Loading)  
      (loadStatus == WebBrowserReadyState.Interactive))
                {
                    break;
                }
                counter++;
            }

            //wait till the page get loaded.
            counter = 0;
            while (true)
            {
                loadStatus = webBrowser1.ReadyState;
                Application.DoEvents();

                if (loadStatus == WebBrowserReadyState.Complete)
                {
                    break;
                }
                counter++;

            }

        }

If you have better solution, just tell me !

Finding broken links using Http WebRequest / Http WebResponse in C#

Status Code in the response will be used for finding whether the link is broken or not. But normally exception will be thrown if the link is broken. So the Timeout property of webrequest plays important role here.

(i-e) If we specify more timeout value, then total execution will take more time. If we specify less timout then there may a possiblity of declaring a valid link as a broken link. If anyone knows how to handle it appropriately, you can mention it in the comments.


private bool isBrokenLink(string url)
{

Boolean isBrokenLink = false;

try
{

WebRequest http = HttpWebRequest.Create(url);
http.Timeout = 5000;
HttpWebResponse httpresponse = (HttpWebResponse)http.GetResponse();

if (httpresponse.StatusCode == HttpStatusCode.OK)
{
isBrokenLink = false;
}
else
{
isBrokenLink = true;
}


}
catch (Exception ex)
{
isBrokenLink = true;

}
return isBrokenLink;

}

Making below two changes in the above code may increase the performance.

HttpWebRequest http = (HttpWebRequest) WebRequest.Create(url);
               http.UserAgent = "Mozilla/9.0 
              (compatible; MSIE 6.0; Windows 98)";
               http.Method = "HEAD";

Actually the HEAD method will allow verifying the link without downloading entire content. So the performance will be increased. Particularly, it will improve the performance significantly when verifying the missing images.

If you have better solution, just tell me !

Creating Captcha using PHP and Handling Captcha using C# web browser control

These days captcha is used widely in most of the websites for preventing automated entry of details in their websites.

For those who are hearing the word Captch first time - You might have seen an image with blurred alpha numeric content with different font style and font size and with different orientation. It is used in the forms of websites to prevent automated entry of details using software programs such as bots and crawlers. So only the human can read those contents and we need to enter that content in the text box provided near this image. This system is called as Captcha.

If you want to avoid dependency of this third-party service you can create your own captcha images dynamically using GDLibrary of PHP. Simple Google search will give you the code. Please find below the sample one.

class CaptchaImages {

 var $font = 'monofont.ttf';

 function generateCode($characters) {
  /* list all possible characters, similar looking characters and

                                          vowels have been removed */
  $possible = '23456789bcdfghjkmnpqrstvwxyz';
  $code = '';
  $i = 0;
  while ($i < $characters) { $code .= substr($possible, mt_rand(0,

                                             len($possible)-1), 1);

      $i++;

}   return $code;

}   function CaptchaImages($width='120',$height='40',$characters='6')

{
  $code = $this->generateCode($characters);
  /* font size will be 75% of the image height */
  $font_size = $height * 0.75;
  $image = @imagecreate($width, $height) or die('Cannot initialize new

                                                    GD image stream');
  /* set the colours */
  $background_color = imagecolorallocate($image, 255, 255, 255);
  $text_color = imagecolorallocate($image, 20, 40, 100);
  $noise_color = imagecolorallocate($image, 100, 120, 180);
  /* generate random dots in background */
  for( $i=0; $i<($width*$height)/3; $i++ )  {

  imagefilledellipse($image, mt_rand(0,$width), mt_rand(0,$height),

      1, 1, $noise_color);
  }
  /* generate random lines in background */
  for( $i=0; $i<($width*$height)/150; $i++ ){imageline($image, mt_rand

                                        (0,$width), mt_rand(0,$height),

 mt_rand(0,$width), mt_rand(0,$height), $noise_color);
  }
  /* create textbox and add text */
  $textbox = imagettfbbox($font_size, 0, $this->font, $code) or die

                     (               'Error in imagettfbbox function');
  $x = ($width - $textbox[4])/2;
  $y = ($height - $textbox[5])/2;
  imagettftext($image, $font_size, 0, $x, $y, $text_color,

      $this->font , $code) or die('Error in imagettftext function');
  /* output captcha image to browser */
  header('Content-Type: image/jpeg');
  imagejpeg($image);
  imagedestroy($image);
  $_SESSION['security_code'] = $code;
 }
}
$width = isset($_GET['width']) ? $_GET['width'] : '120';
$height = isset($_GET['height']) ? $_GET['height'] : '40';
$characters = isset($_GET['characters']) && $_GET['characters']

                                 > 1 ? $_GET['characters'] : '6';

$captcha = new CaptchaImages($width,$height,$characters);

Till now we have seen how to implement Captcha to avoid automated form filling using bots.

Sometimes we may need to navigate the websites using bots created using webbrowser control. As the bots can not read the captcha content, it is not possible to navigate the sites without interruption. In this case, we can create a code atleast for allowing the user to manually enter the captcha text while the bot continues the execution.


public partial class ModalesMsgBox : Form
{
public string captchaWord;
public string captchaurl;
public bool isEntered;
public ModalesMsgBox(string strcaptchaurl,string strMsg)
{
captchaurl = strcaptchaurl;
isEntered = false;
InitializeComponent();
lblMsg.Text = strMsg;

}

private void btnsubmit_Click(object sender, EventArgs e)
{
captchaWord = txtCaptcha.Text;
isEntered = true;
this.Close();
}



private void ModalesMsgBox_Load(object sender, EventArgs e)
{

try
{

webBrowser1.Navigate(captchaurl);

}
catch(Exception ex)
{

}

}

Find below the sample code for using the above class in the typical webbrowser control navigation code.

 string strMsgrd = "Please enter captcha letters and click

'Enter' in this Dialog box.\n Don't type 'Enter'  and don't click 'Submit' button in the browser control";
                string strCaptchImgrd = webBrowser1.Document.GetElementById

                                              ("capimage").GetAttribute("src");
                ModalesMsgBox msgrd = new ModalesMsgBox(strCaptchImgrd, strMsgrd);
                wait(70000);
                msgrd.Show();
                wait(500000);
                while (!msgrd.isEntered)
                {
                    Application.DoEvents();
                }
                string strCaptchard = msgrd.captchaWord.ToString();
                webBrowser1.Document.GetElementById("captcha").Focus();
                wait(70000);
                webBrowser1.Document.GetElementById("captcha").InnerText = strCaptchard;
                wait(700000);
                HtmlElementCollection SubmitButton = webBrowser1.Document.

                                                  GetElementsByTagName("button");
                SubmitButton[0].InvokeMember("click");

If you have better solution, just tell me !

Deleting Session Cookie in Webbrowser control.

We know that many websites are using cookies for storing some user data in browser to improve user experience.

C#.net is having webbrowser control for creating applications useful for automatically navigating/crawling websites.

It will be required to delete these cookies in webbrowser control when using it for different set of user logins.

Below C# function will be useful for deleting the cookie for IE (Internet Explorer) browser. As webbrowser control will share the cookie with IE, this code can be used for deleting webbrowser control cookie.

I heard that cookie will be stored somewhere temporarily also, and this function won't delete it. But I am not sure about. If anyone know more details about this temporary cookie you can share it here.


private void deletecookie()
{
string[] theCookies = System.IO.Directory.GetFiles(Environment.GetFolderPath(Environment.SpecialFolder.Cookies));

foreach(string currentFile in theCookies)
{

try
{

System.IO.File.Delete(currentFile);

}
catch (Exception ex)
{
//MessageBox.Show(ex.Message);
}

}

}

If you have better solution, just tell me !

Ajax and sample code for learning Ajax

Ajax is not an Technology in itself. It is a term used for representing the technique used for updating the webpage content asynchronously without affecting existing content of the webpage.

Ajax was made popular by Google by using it in the Google Suggest.

So, basically below steps are involved in using ajax for webdevelopment.

- Creating client side code which will call a javascript function on some events such as onChange.

- Creating a XMLHttpRequest which will be used for sending request to web server from javascript itself and used for receiving response from the webserver.

- A place such as div in the client side code for placing the server response got from the ajax call.

We will see the above steps in detail using below sample code.

Consider below HTML code which will be placed within a Form.


<select name="name" id="name" onChange="showUser(this.value);">

The above code will call a javascript function showUser() when changing the drop-down list value in the Select tag.

The selected value (i-e name in this example) will be passed as argument to the showUser.

The showUser javascript function will look like below mentioned code.



function showUser(str)
{
xmlHttp=GetXmlHttpObject();

if (xmlHttp==null)
 {
 alert ("Browser does not support HTTP Request");
 return;
 }
var url="getuser.php";
url=url+"?q="+str;

url=url+"&sid="+Math.random();
xmlHttp.onreadystatechange=stateChanged;
xmlHttp.open("POST",url,true);
xmlHttp.send(null);
}



This function creates xmlHttp object by calling a function GetXmlHttpObject()

You can refer the GetXmlHttpObject() below.


function GetXmlHttpObject()
{
var xmlHttp=null;
try
 {
 // Firefox, Opera 8.0+, Safari
 xmlHttp=new XMLHttpRequest();
 }
catch (e)
 {
 //Internet Explorer
 try
  {
  xmlHttp=new ActiveXObject("Msxml2.XMLHTTP");
  }
 catch (e)
  {
  xmlHttp=new ActiveXObject("Microsoft.XMLHTTP");
  }
 }
return xmlHttp;
}

The url to be sent as server request is created by specifying the server-side webpage (e.g getuser.php), the function argument "name", and a random number to avoid cache problem.

A function stateChanged is called on onreadystatechange event for getting the response from the webserver for the request of this created url.

You can refer the sample state Changed function below.

Find below the sample server side code (e.g getuser.php) which will be used for sending response according to the value selected in the drop-down list.

<?php
$link=project1_db_connect1();
//for connecting database.
    function project1_db_connect1()
    {
        $link=@mysql_connect("localhost","root","") or exit ();
        mysql_select_db ("my_db1") or exit ();

        return $link;
    }
?>
<table border="1" width="100%">
    <tr>
        <td>First name</td>
        <td>last name</td>
        <td>age</td>
    </tr>
    <?php
    $q=$_GET["q"];

        $sqluser="select * from persons where id=".$q;
        $result=mysql_query($sqluser);
        while($userrow=mysql_fetch_assoc($result))
        {
            $firstname=$userrow['FirstName'];
            $lastname=$userrow['LastName'];
            $age=$userrow['Age'];

            ?>
            <tr>
                <td><?php echo $firstname;?></td>
                <td><?php echo $lastname;?></td>
                <td><?php echo $age;?></td>
            </tr>
            <?php
        }
    ?>
</table>

The above code is receiving the selected drop-down value as "q" querystring parameter in the url. sql query will be used for fetching details such as firstname, lastname age for the corresponding "q" value.The xmlhttp object is receiving the output of this php page.

If you have better solution, just tell me !

AJAX Login System - AJAX Script

This is an example of a login system that does not require page refreshes, but is still very secure. Valid usernames and passwords for this demo are user1/pass1 and user2/pass2. Try these, and also incorrect passwords to see the results.

Please note that this is not a functional form, your input will not go anywhere.It is solely for demonstrating an XMLHttpRequest login system in javascript.

Advantages

User does not need to refresh the page to login.

User is notified instantly on incorrect username/password combination.

Overall user experience is more seamless.

Password is not sent in plain text ever (more secure than traditional system).

Javascript convenience with server-side security (uses PHP/MySQL).

Uses one-time use random seed to hash the password before sending (making interceptions useless).

Disadvantages

System is more prone to brute force attacks.

Can be minimized by adding a delay after a certain number of attempts per username or per client.

User may expect a login button.

One could still be added without reloading the page.

Older versions of Safari cannot disable a password field.

This code uses the MD5 encryption algorithm, which has since been proven to be less secure than previously thought. If you use this code, I strongly recommend you switch to a more secure encryption algorithm, such as SHA-1. For sites were security is not crucial, MD5 should suffice.

Source Download

login_controller.js

login_presentation.js

xml_http_request.js

md5.js

If you have better solution, just tell me !

User validation across pages using session after login in ASP.NET using C sharp

In this example i m showing how to validate a user across different pages whether user is logged in or not using session variables in Global.asax through Session_Start event and Application_OnPostRequestHandlerExecute event which checks for the login validation which occurs when ant asp.net event handler finish execution

Here is my login page , i've used hard coded values to login.

<%@ Page Language="C#" AutoEventWireup="true"
CodeFile="Login.aspx.cs" Inherits="_Default" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
."http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >
<head runat="server">
<title>Untitled Page</title>
</head>
<body>
<form id="form1" runat="server">
<div style="text-align:left" >
<table width="40%" style="text-align: center">
<tr><td style="width: 20%">
<asp:Label ID="lblUserName"
runat="server" Text="Enter UserName:">
</asp:Label></td>
<td style="width: 20%">
<asp:TextBox ID="txtUserName"
    runat="server">
</asp:TextBox></td>
</tr>
<tr>
<td style="width: 20%">
<asp:Label ID="lblPassword" runat="server"
  Text="Enter Password:">
</asp:Label></td>
<td style="width: 20%" >
<asp:TextBox ID="txtPassword" runat="server"
    TextMode="Password">
</asp:TextBox></td>
</tr><tr><td colspan="2" align="right">
<asp:Button ID="btnLogin" runat="server"
   Text="Sign in" OnClick="btnLogin_Click" />
</td></tr>
</table>
<asp:Label ID="Label1" runat="server"
      Text="Label">
</asp:Label><br />
</div>

</form>
</body>
</html>

After checking the username and password i m creating a new Session variable and setting the flag kindaa value in it , which is "Yes" in this example, this session value will be checked when ever user go to other pages and if it's null than user in not logged in.

protected void btnLogin_Click(object sender, EventArgs e)
{
if (txtUserName.Text == "amit" && txtPassword.Text == "amit")
{
Session["Authenticate"] = "Yes";
Response.Redirect("Default2.aspx");
}
else
Label1.Text = " login failed";
}

In Global.asax, in Session_Start event i m assigning null value to the session variable created at the time of Login and than calling the method to check the login, same is in Application_OnPostRequestHandlerExecute event as well.

void Session_Start(object sender, EventArgs e)
{
// Code that runs when a new session is started
Session["Authenticate"] = "";
CheckLogin();

}
void Application_OnPostRequestHandlerExecute()
{
CheckLogin();
}

void CheckLogin()
{
string Url = Request.RawUrl;
int count = Url.Length - 10 ;
string TestUrl = Url.Substring(count);
string SessionData = Session["Authenticate"].ToString();
if (SessionData == "" && TestUrl != "Login.aspx")
{
Response.Redirect("~/Login.aspx");
}
}

If you have better solution, just tell me !

Soure Code

C# webbrowser control - Synchronization for Page navigation/loading

Finding broken links using Http WebRequest / Http WebResponse in C#

Creating Captcha using PHP and Handling Captcha using C# web browser control

Deleting Session Cookie in Webbrowser control.

Ajax and sample code for learning Ajax

AJAX Login System - AJAX Script

Source Download

User validation across pages using session after login in ASP.NET using C sharp

Categories

Blog Archive