Inquisitive Mind

Web Crawlers

2010-12-20T02:20:00.001-06:00

How do search engines know what websites are linked? How do spammers get hold of our email addresses? Simple answer: Web crawlers! Yes these tiny programs do a wonderful job of skimming through websites and picking up website or email addresses. One can then build a network of web pages. These are the basis for search engines. Web crawlers, spiders and bots(robots) are similarly functioning software agents which differ with the functionality of application.
Web crawler follows the different hyperlinks and this data is used for Search Engines indexing and ranking.
Web Spiders download the web pages again by traversing the links
Bots are computer programs which visit websites & perform predefined task
Thus all of them are parsing through the html content of one web page to pick up the referenced hyperlinks.
Let us now build a small web crawler which takes an URI as an input and returns a list of all the websites that have been linked in given page. This can be recursively used to create a smarter program to crawl bigger websites.
webCrawler(String strURL){
URL url;
try {
//create the url from the string
url = new URL(strURL); // try opening the URL
URLConnection urlConnection = url.openConnection();
System.out.println("successful opening the URL " + strURL);
InputStream urlStream = url.openStream();
// read the entire content from URL
byte b[] = new byte[1000];
int numRead = urlStream.read(b);
content = new String(b, 0, numRead);
while (numRead != -1) {
numRead = urlStream.read(b);
if (numRead != -1) {
String newContent = new String(b, 0, numRead);
content += newContent;
}
}
urlStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
Here we are taking the URI in form of string. We convert it into a URL using the Java URL library function. If we are successful in opening this link, we read the entire content into a variable. This variable will be further used to look for the hyper links.
public ArrayList<String> getListOfPlatforms(){
ArrayList<String> platformList = new ArrayList<String>();
//covert all content to lowerCase
String lowerCaseContent = content.toLowerCase();
int index = 0;
//search for links referred in the html source
while((index = lowerCaseContent.indexOf("<a",index))!= -1)
{
if((index = lowerCaseContent.indexOf("href",index))== -1)
break;
if ((index = lowerCaseContent.indexOf("=", index)) == -1)
break;
index++;
String moreContent= content.substring(index);
StringTokenizer str
= new StringTokenizer(moreContent, "\t\n\"");
String strLink = str.nextToken();
//addit to the ArrayList of strings
platformList.add(strLink);
}
//return the list of folder names
return platformList;
}
Here we parse the content and search for all the href values. The links are added to an ArrayList object and returned. This list can be used for further crawling. So we can recursively crawl many webpages.

Common Language Runtime - CLR

2010-12-16T04:02:00.000-06:00

What is a CLR?

This is something you need to know if you say you are working on .Net framework.

Common Language Runtime (CLR) is an application virtual machine where in programmers need not consider the capabilities of the specific CPU that will execute the program. The .NET Framework provides a run-time environment, which runs the code and provides services that make the development process easier. The class library and the CLR together constitute the .NET Framework.

Developers using the CLR, write code in a language such as C# or VB.NET. At compile time, a .NET compiler converts such code into CIL code. At runtime, the CLR's just-in-time compiler converts the CIL code into code native to the operating system.

The common language runtime makes it easy to design components and applications whose objects interact across languages. Objects written in different languages can communicate with each other, and their behaviors can be tightly integrated. For example, you can define a class and then use a different language to derive a class from your original class or call a method on the original class. You can also pass an instance of a class to a method of a class written in a different language. This cross-language integration is possible because language compilers and tools that target the runtime use a common type system defined by the runtime, and they follow the runtime's rules for defining new types, as well as for creating, using, persisting, and binding to types.

CLR also provides some of the other services namely:

Memory management
Thread management
Exception handling
Garbage collection
Security

I will talk about these in detail in the coming blogs.

File Transfer using Socket Programming in C# with File Split

2010-11-22T15:39:00.005-06:00

Here we will see how to do multiple client server file transfer using Socket Programming. You can see a lot of documents for a simple file transfer. But this is different. Here, if the file size is large, say larger than 1 MB, I am splitting the file for sizes of 1MB and then transferring each file to the server. At the server side, the split files will be stored until all the files are received and once it is received, I am merging all the files to create the single file.

Right now this code can handle upto 14MB of files with a split size of 1MB. You can increase the split size and proportionally the file to be sent size.

Server Side code:

Please do remember to add the timer and the background worker in your form.

using System;

using System.Data;

using System.Text;

using System.Windows.Forms;

using System.Net;

using System.Net.Sockets;

using System.IO;

using System.Threading;

// File transfer protocol Server function by Vinay

namespace Server

{

public partial class Server : Form

{

public Server()

{

InitializeComponent();

FTServer.receivedPath = "";

}

// Called when the Server start button is clicked

private void btnStart_Click(object sender, EventArgs e)

{

//Checks if the destination path is set

if (FTServer.receivedPath.Length > 0)

backgroundWorker1.RunWorkerAsync();

else

MessageBox.Show("Please select file receiving path");

}

private void timer1_Tick(object sender, EventArgs e)

{

label5.Text = FTServer.receivedPath;

label3.Text = FTServer.curMsg;

}

FTServer obj = new FTServer();

private void backgroundWorker1_DoWork(object sender, DoWorkEventArgs e)

{

obj.StartServer();

}

// Sets the destination path

private void btnReceive_Click(object sender, EventArgs e)

{

FolderBrowserDialog fd = new FolderBrowserDialog();

if (fd.ShowDialog() == DialogResult.OK)

{

FTServer.receivedPath = fd.SelectedPath;

}

// Server is stopped and all the temporary files are deleted

private void btnStop_Click(object sender, EventArgs e)

{

FTServer.sock.Close();

FTServer.curMsg = "Server Stopped";

FTServer.deleteFiles();

}

//FILE TRANSFER USING C#.NET SOCKET - SERVER

class FTServer

{

IPEndPoint ipEnd;

public static Socket sock;

public FTServer()

{

//To accept any IP address with port number 5656

ipEnd = new IPEndPoint(IPAddress.Any, 5656);

// Creating a new socket with TCP protocol

sock = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);

//Bind the socket with the Ip

sock.Bind(ipEnd);

}

public static string receivedPath;

public static string curMsg = "Stopped";

// Method to receive the data sent from the client.

public void StartServer()

{

try

{

curMsg = "Starting...";

//The socket is waiting for client connection. The socket can handle maximum

// of 100 client connections at a time.

sock.Listen(100);

curMsg = "Running and waiting to receive file.";

//When a request comes from the client the socket is ready to accept

Socket clientSock = sock.Accept();

//5Mb of buffer space is allocated for the data transfer

byte[] clientData = new byte[1024 * 5000];

//The receive method returns the size of the data that is transferred.

//The size will be in bytes.

int receivedBytesLen = clientSock.Receive(clientData);

int fileNameLen = BitConverter.ToInt32(clientData, 0);

string fileName = Encoding.ASCII.GetString(clientData, 4, fileNameLen);

//If the file exists in the path, the existing file is deleted

if (File.Exists(receivedPath + "\\" + fileName))

{

File.Delete(receivedPath + "\\" + fileName);

}

curMsg = "Receiving data..." + fileName;

//Binary stream writer to save the data received from the client.

BinaryWriter bWrite = new BinaryWriter(File.Open(receivedPath + "/" + fileName, FileMode.Append));

bWrite.Write(clientData, 4 + fileNameLen, receivedBytesLen - 4 - fileNameLen);

Thread.Sleep(1000);

curMsg = "Saving file...";

//Closing the binary writer

bWrite.Close();

//Closing the socket connection

clientSock.Close();

//Calling the merger method to combine the split files into a single readable file.

mergeFiles(fileName);

StartServer();

}

catch (Exception ex)

{

curMsg = "File Receving error.";

}

// Merge method to combine the split files into a single readable file.

public void mergeFiles(string recvdfileName)

{

//Get all the files with the extension .vin

string[] filePaths = Directory.GetFiles(receivedPath, "*.vin");

string fileName = " ";

int parts = filePaths.Length;

for (int i = 0; i < parts; i++)

{

fileName = getFilename(0, filePaths[i]);

if (filePaths[i].IndexOf(recvdfileName) > -1)

{

//Deleting the file if the file is already present

if (File.Exists(receivedPath + "\\" + fileName))

{

File.Delete(receivedPath+ "\\" + fileName);

}

//Creating a new file with the filename

FileStream createFile = File.Create(receivedPath + "\\" + fileName);

createFile.Close();

}

string path = receivedPath + "\\" + fileName;

FileStream outFile = new FileStream(path, FileMode.OpenOrCreate, FileAccess.Write);

if (outFile != null)

{

outFile.Flush();

}

for (int i = 0; i < parts; i++)

{

if (filePaths[i].IndexOf(fileName) > -1)

{

FileInfo f = new FileInfo(filePaths[i]);

int size = (int)f.Length;

int data = 0;

byte[] buffer = new byte[1024 * 50000];

FileStream inFile = new FileStream(filePaths[i], FileMode.OpenOrCreate, FileAccess.Read);

//reading the data from the split file and putting it into a single file.

while ((data = inFile.Read(buffer, 0, size)) > 0)

{

outFile.Write(buffer, 0, data);

}

inFile.Close();

}

outFile.Close();

}

// Method to get the filenames with the extension

public string getFilename(int i, string fileName)

{

string[] array = fileName.Split('\\');

int size = array.Length;

string[] tempname = array[size - 1].Split('.');

return tempname[i] + "." + tempname[i+1];

}

// Method to delete all the files when the server stop button is clicked.

public static void deleteFiles()

{

string[] filePaths = Directory.GetFiles(receivedPath, "*.vin");

int parts = filePaths.Length;

for (int i = 0; i < parts; i++)

{

File.Delete(filePaths[i]);

}

Client Side:

using System;

using System.Data;

using System.Windows.Forms;

using System.Net;

using System.Net.Sockets;

using System.IO;

using System.Threading;

// File transfer protocol Client function by Vinay

namespace Client

{

public partial class Client : Form

{

public Client()

{

InitializeComponent();

}

// Called when the button is clicked.

private void btnSend_Click(object sender, EventArgs e)

{

string filePath = "";

FTClientCode.curMsg = "Idle";

//Loads the windows explorer to select a file.

FileDialog loadFile = new OpenFileDialog();

if (loadFile.ShowDialog() == DialogResult.OK)

{

FileInfo file = new FileInfo(loadFile.FileName);

long fileLength = file.Length; //the length of file

//extract the name of the File to be FTPed

string fileName = getFilename(1, loadFile.FileName);

//size of File in MBs

double fileinMB = converToMB(fileLength);

string changedFileName = "";

int parts = 1;

//Checks if the sie of the file chosen in greater than 1MB

//If yes then splits the file with name with an extension .vin

if (fileinMB > 1)

{

parts = splitFile(file, fileinMB, fileName);

}

for (int i = 0; i < parts; i++)

{

if (parts == 1)

{

changedFileName = loadFile.FileName;

}

else

{

changedFileName = "C:\\Test\\" + fileName + "." + i + ".vin";

}

changedFileName = changedFileName.Replace("\\", "/");

while (changedFileName.IndexOf("/") > -1)

{

filePath += changedFileName.Substring(0, changedFileName.IndexOf("/") + 1);

changedFileName = changedFileName.Substring(changedFileName.IndexOf("/") + 1);

}

byte[] fileNameByte = Encoding.ASCII.GetBytes(changedFileName);

//Reads all the data in the file

byte[] fileData = File.ReadAllBytes(filePath + changedFileName);

byte[] clientData = new byte[4 + fileNameByte.Length + fileData.Length];

byte[] fileNameLen = BitConverter.GetBytes(fileNameByte.Length);

fileNameLen.CopyTo(clientData, 0);

fileNameByte.CopyTo(clientData, 4);

fileData.CopyTo(clientData, 4 + fileNameByte.Length);

FTClientCode.SendFile(clientData, changedFileName);

filePath = " ";

}

//deletes the split files after they have been trasnferred throuhg FTP

deleteFiles(loadFile.FileName);

}

// spilts the big file into smaller files of size 1MB or lesser

public static int splitFile(FileInfo f, double fileSize, string fileName)

{

int parts = (int)Math.Ceiling(fileSize / 1);

int eachSize = (int)Math.Ceiling((double)fileSize / parts) * 1024 * 1024;

FileStream inFile = new FileStream(f.ToString(), FileMode.OpenOrCreate, FileAccess.Read);

for (int i = 0; i < parts; i++)

{

string path = "C:\\Test\\" + fileName + "." + i + ".vin";

if (File.Exists(path))

File.Delete(path);

FileStream outFile = new FileStream(path, FileMode.OpenOrCreate, FileAccess.Write);

int data = 0;

byte[] buffer = new byte[eachSize];

if ((data = inFile.Read(buffer, 0, eachSize)) > 0)

{

outFile.Write(buffer, 0, data);

}

outFile.Close();

}

inFile.Close();

return parts;

}

// converts file length into Megabytes

public static double converToMB(long fileLength)

{

return (fileLength / 1024f) / 1024f;

}

public static string getFilename(int i, string fileName)

{

string[] array = fileName.Split('\\');

int length = array.Length;

return array[length - 1];

}

public void deleteFiles(string recvdfileName)

{

string[] filePaths = Directory.GetFiles("C:\\Vinay\\", "*.vin");

string fileName = " ";

int parts = filePaths.Length;

for (int i = 0; i < parts; i++)

{

fileName = getFilename(0, recvdfileName);

if (filePaths[i].IndexOf(fileName) > -1)

{

File.Delete(filePaths[i]);

}

private void timer1_Tick(object sender, EventArgs e)

{

label3.Text = FTClientCode.curMsg;

}

//FILE TRANSFER USING C#.NET SOCKET - CLIENT

class FTClientCode

{

public static string curMsg = "";

public static void SendFile( byte[] clientData, string fileName)

{

try

{

//create a new client socket

Socket clientSock = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);

// get the ipaddress

IPAddress[] ipAddress = Dns.GetHostAddresses("localhost");

// Make IP end point same as Server.

IPEndPoint ipEnd = new IPEndPoint(ipAddress[1], 5656);

curMsg = "Connection to server ...";

clientSock.Connect(ipEnd);

curMsg = "Buffering ...";

curMsg = "File sending..." + fileName;

//add some delay to let the process finish

Thread.Sleep(3000);

//send the data

clientSock.Send(clientData);

//close the client

clientSock.Close();

curMsg = "Disconnecting...";

curMsg = "Files are transferred.";

}

catch (Exception ex)

{

if (ex.Message == "No connection could be made because the target machine actively refused it")

curMsg = "File Sending fail. Because server not running.";

else

curMsg = "File Sending fail." + ex.Message;

}

Some interesting facts about Google

2010-11-20T17:30:00.000-06:00

I was reading some blog and found out some interesting topics about google. So here they are:

1. CREATE UNLIMITED DISPOSABLE EMAIL ADDRESS WITH GMAIL

If you add dots (.) between the letters of your Gmail username, sending a message to the new username will get forwarded to your original email (which is without or with only 1 dot.)

It doesn’t matter how many dots you’ll add between your username, all of the emails sent will go to your original email.

Gmail provides another great explanation:

Gmail doesn’t recognize dots as characters within usernames, you can add or remove the dots from a Gmail address without changing the actual destination address; they’ll all go to your inbox, and only yours. In short:

Abc.def@gmail.com = a.b.c.d.e.f@gmail.com = abcde.f@gmail.com

All these addresses belong to the same person. You can see this if you try to sign in with your username, but adding or removing a dot from it. You’ll still go to your account.

Why is this helpful? Let’s say you want to sign up for a particular newsletter but you’re afraid of spam. Then you can modify your email with the dots so in case you start getting unwanted messages, you can use Gmail filters and send every message your ‘new’ email receives to spam directly.

2. FIND BEATIFUL WALLPAPER IN 10 SECONDS USING GOOGLE IMAGES

Nowadays it’s very easy to find any type of wallpaper using Google images. Here’s what you do:

Go to Advanced Image Search
Click on “Use my desktop size” button. Google will take your desktop size automatically! (Previously you needed to do this manually using the imagesize: operator, plus manually enter the resolution.)
From “Content types” select “photo content.” Even when you search for celebrities, selecting ‘photo content’ instead of ‘faces’ will give you way better results.
Tell Google what type of wallpaper you want by typing some keywords next to “related to all of the words” option.
Many of the results come from wallpaper gallery sites. As you can see, this is so far the best way to browse wallpaper sites and find beautiful wallpapers with little effort.

3. A NEAT TRICK TO FIND ALTERNATIVES TO ALL TYPES OF PRODUCTS USING GOOGLE SEARCH

Let’s say you want to find some iPad alternatives. Sure, you can type “iPad alternatives” into Google search but you’ll get mostly articles from tech sites and not opinions from people just like you.

Solution: Use the “better than [product]” query and you’ll get way better alternatives. If you’re looking for Firefox alternatives, write “better than Firefox”, for Windows, “better than Windows".

4. SEARCH ANY VIDEO LIKE A PRO USING GOOGLE VIDEO

Let’s just say you’re a big fan of MetaCafe.com. You want to learn French and use their internal search engine to find some videos.

Where as in Google Video go to Advanced Search Page. Here you’ll find one option called “Domain: Only return videos from the site or domain.” Let’s just use our example and enter metacafe.com

You’ll notice after you click on “Search videos” that the results are usually:

Way more relevant
On the left, you can filter the videos by duration, quality and date. You can specify for example a date range so Google can return videos posted on MetaCafe from 1st to 30th March 2010. You can also specify to search videos longer than 10 minutes (I haven’t yet seen any video site that offers this option.)

You can do this trick not only with Metacafe but with any other popular video site like Vimeo, DailyMotion or Hulu. All of them have very limited search functionality compared to Google Video.

5. SEE WHAT THE FRENCH HAVE BEEN SEARCHING FOR RECENTLY USING GOOGLE INSIGHTS

You can easily figure out what are the most popular search terms for any country (Google is the #1 search engines in most of the world’s countries) by using one neat tool called Google Insights

Simple Database Connection using C#

2010-11-20T02:45:00.007-06:00

To connect to database using C# is very easy. I am using VS 2008. The database being used is MS SQL Server.

First let me tell the steps involved in connecting to the database

1. Make a note of the connection string. The information needed are

a. Data Source;

b. Initial Catalog;

2. Once you have this information, do not forget to include SqlClient in the "using" directive.

3. Initialize SqlConnection by providing the Datasource and Initial catalog information

4. Open the connection.

5. Read the data using SqlDataReader

6. You can bind the data to a grid view.

7. Close the connection.

Now coming to the coding part.

using System;
using System.Collections;
using System.Configuration;
using System.Data;
using System.Data.SqlClient;

namespace DBConnect
{
   public partial class _Default : System.Web.UI.Page
   {
      protected void Page_Load(object sender, EventArgs e)
   {
   //You can call the function in the button click event.
   DBConnect();
   }

/// <summary>

/// Method to connect to the DB

/// </summary>

/// <returns></returns>

protected void DBConnect()

{

//Connection String.

SqlConnection conn = new SqlConnection("Data Source=Database Source;Initial Catalog=Database;Integrated Security=True");
SqlDataReader dataReader = null;

try

{
//Opening the connection.

conn.Open();

if (conn != null)

     string selectString = "Select * from Employer";
   SqlCommand cmd = conn.CreateCommand();
   cmd.CommandText = selectString;
   dataReader = cmd.ExecuteReader();
   //Checking if the dataReader is empty or not
   if (dataReader.HasRows)
   {
   //Setting the datasource of the grid view to //the SQL Data Reader
   gridView1.DataSource = dataReader;
   //Binding it to the grid view
   gridView1.DataBind();
   }

}

      finally
   {
   //Closing the data reader
   if (dataReader != null)
   {
   dataReader.Close();
   }
   conn.Close();
   }

}
}
}

There you go. Your database is connected and the values are put into the grid view.

My First Blog

2010-11-19T23:55:00.002-06:00

Hi All,
As the heading goes, this is my first ever blog. Not really sure how many posts I will be publishing but then wanted to try out. The idea of writing a blog came to me when I was doing a project of mine.
There were some pretty good blogs which have explained the things to do in detail. It was then that I thought to myself that I will also start writing.
Oh by the way this will be a pure technical blog.