DayDreamerman

Problems Detected

2012-05-05T01:28:00.000+05:30

Due to the severe problems with Google Blogger,now I am leaving this blogger (maybe forever).I will meet on the another blog specified below :

http://daydreamerman.wordpress.com

Till then bye.......

Varnish–Power Booster for Facebook

2012-04-08T02:01:00.002+05:30

One thing most of people does not know is that, most of the Facebook request mostly does not reach the HDD i.e. hard disk.Most of the requests remain in the cache buffer of Facebook (Memcached for Facebook) .

We have seen what basically MemCached in one of the post , now the turn comes is of Varnish, the power booster which picks up every requests in Facebook and sends to the servers,that is it is a reverse proxy.

Above photo shows the best easy diagrammatic representation of how does Varnish work.Varnish is heavily threaded, with each client connection being handled by a separate worker thread. When the configured limit on the number of active worker threads is reached, incoming connections are placed in an overflow queue.Thereby only when this queue has reached its limit then the incoming connections will be rejected.

According to data known, there are following things occurring in Facebook in next 60 seconds :
6,96,000+ Facebook status updates, 5,10,000 comments.

Caching is vital for high-load applications. A typical web application serves a lot of content that costs much more to generate than it costs to cache (including the cost of checking and expiring the cache), so caching can usually improve performance by orders of magnitude.

Varnish Cache is a web application accelerator also known as a caching HTTP reverse proxy. You install it in front of any server that speaks HTTP and configure it to cache the contents. Varnish Cache is too fast. It typically speeds up delivery with a factor of 300 - 1000x, depending on your architecture.

It serves billions of requests every day to Facebook users around the world. Whenever you load photos and profile pictures of your friends, there's a very good chance that Varnish is involved.

A normal proxy redirects the server request through its proxy server, whether via LAN or anonymously and then sends the request to the concerned server. This helps to hide the identity of the original user.
In normal way the cache is more stored either on server.To avoid this a reverse proxy like Squid or Varnish is used.In reverse proxy the compression and load balancing of data is done on the client side.

One of the main feature ( and the BEST feature according to me) in varnish is that it has two modes in it,which makes it enormously useful to network administrators and web programmers to handle requests according to the behavior of request.The following article will be posted about that two modes.Until that time,read this article carefully.

Optimization of News Feed in Facebook

2012-03-18T11:15:00.001+05:30

The Facemash algorithm that I had posted here was just a “trailer” of this actual post for starting understanding Facebook.

Facebook used basically a data structure,called graph.Graph includes two basic things:

1.Nodes with their corresponding value

2.Edges between nodes with their corresponding weight

Whatever you observe in Facebook mainly is always the News Feed, which is the most important feature of Facebook.

So it is always needed to carefully choose which posts to publish on the first page of your News Feed. So the main component of your News Feed and at last of your Facebook page is optimization of News Feed.

Below photo shows formula used by Facebook to optimize the News Feed of Facebook profile.

As shown in above photo, the friends and relationships and post by any one of them are based on three parameters:

1.Weight of edge type

2. Time from which they were friends and how ago the post was posted on Facebook

3. How closer the friends i.e. in same school or just a fan of other friend

Thus,the sum of product of all your friends and “weight” of your relationships with them is what you see in the News Feed of your Facebook profile.

So the better you are friends (it can be known by a simple Facebook app,whose making we will discuss later) the more are the chances that your post will be on the News Feed of other one. Similarly , the more you interact,more is the chance is there that you are featured on the News Feed of your other friend.

Four Your Information(FYI):

Depending on your number of friends and their activity, a click to your News Feed can call up as many as 10,000 stories that account for about 8 megabytes of space. All this occurs in about seven-tenths of a second.

Algorithm for facemash

2012-03-11T22:40:00.001+05:30

You must be remembering this scene from The Social Network movie,when Eduardo Saverin writes an algorithm for making facemash.com.

Actually,the algorithm is a simple algorithm to rank the players of chess,named Elo rating System.According to wiki:

The Elo rating system is a method for calculating the relative skill levels of players in two-player games such as chess.

Algorithm:

According to this algorithm following represents the expected score of for example,Player A

Similarly the expected score for example,Player B is:

Where R_Ais current rating of Player A and R_Bis current rating of player B.

When player A will play the match whatever score he has,it will be compared with the expected score and that will give a new rating for player A. This will be done using the below formula;

Where S_Ais the actual score of player A. K is a constant that has two values for Master players K=16 and for weaker players K will be 32.

Let us take an example of rating girls as done in the Social Network movie. For example we have the following table:

Girl	Current Rating	Result	Score
A	80
B	90	Lost	0
C	30	Won	1
D	50	Draw	0.5
E	40	Won	1

Now as you can see clearly that A was compared with 4 girls and she has scored 2.5 points. If you calculate the expected score using the above formula it will be as follows :

                     2.157(0.485+0.572+0.543+0.557).

Now using the second formula given above we will calculate the new Rating for player A using K = 16 will be

                     R’_A= 80 + 16(2.5-2.157) = 85.2

That's all about the algorithm.Anything more should be reflected in comments.

Build a Server on your computer using Python

2012-03-02T17:57:00.000+05:30

To make a "home-made" server on your computer is a very easy thing,but its tough to make it with satisfying the following requirements :

1.Without the use of Apache,IIS,Nginx,Lighttpd

2.Using open-source

3.With lot of in-built facilities

This seems like dreamy but you have another way of doing it using the world's best open-source scripting-cum-programming language Python it makes it very easy.By the way,Facebook runs mostly on Python.

If you find that you do not know much about Python,no problem.Just write the four lines below,and a "home-made" server would be built.This will help in making your computer an efficient server.

Initially,you must have Python installed.You can download it from following links,copy the link and paste in the browser (prefer to install the version 3.2):

http://python.org/download/

After the installation,do following.

Open a notepad file and type the following code there:

from http.server import HTTPServer, CGIHTTPRequestHandler
port = 8080
httpd = HTTPServer(('', port), CGIHTTPRequestHandler)
print("Starting simple_httpd on port: " + str(httpd.server_port))
httpd.serve_forever()

Save it and remember the location of it.

Thereby,

For Windows user

1.Go to Start -->All Programs-->Python 3.2-->IDLE

2.Press Ctrl+O to open the file saved above.

3.Press F5.

Above window with message "Staring simple_httpd on port 8080" shows success of our work.

For Unix And Mac OS X Users

You need to do two things to prepare your CGI script for execution:

1. Set the executable bit for your CGI using the chmod +x command.
2. Add the following line of code to the very top of your program:
#! /usr/local/bin/python3

Thereby do another thing:

1. Open Terminal (or Shell).
2. Type python <filename of file created above with full path>

That's it,server is running on your *NIX OS.

That's all how a simple "home-made" Python server is made and run,simply with above 4 steps.Send your queries below in the comments.

Apache Cassandra and Tornado -Rocking Facebook infrastructure

2012-02-26T01:04:00.001+05:30

Here is a post about two rock-solid technologies of Facebook which are highly responsible for all the real-time activities.Rather then keeping the suspense within,here's the technologies:

Apache Cassandra

Apache is a well-known name in server domain for its rock-solid reliability and fully-customizable structure.Most of the large companies have a Apache web server by default.

But this variant of Apache,named Apache Cassandra is killer one.It is designed so that its main purpose is what real-time web apps and websites wants : stability and quick response.When you need scalability and high availability without compromising performance,this is the best choice. Cassandra's support for replicating across multiple servers(or group of servers) is best-in-class, providing lower latency for users of website (or apps) and the peace of mind of knowing that the user can survive regional outage,with no single point of failure .

It has a special ColoumnFamily data model,which offers

    1. To save data as column indexes
    2. Log-style updates of data changed in server
    3. Rapid and strong built-in caching
    4. Strong support for materialized(less use of SQL) views(view means an editable-copy of table in database)
.

Tornado

Using a blocking socket, execution will wait (i.e. "block") until the full socket operation has taken place. So, you can process any results/responses in your code immediately after. These are also called synchronous sockets.

A non-blocking socket operation will allow execution to resume immediately and you can handle the server's response with a callback or event. These are called asynchronous sockets.

Why I am explaining this would be clear after reading below.

Tornado is a relatively simple, non-blocking web server framework written in Python. It is designed to handle thousands of simultaneous connections, making it ideal for real-time Web services.

The framework is distinct from most mainstream web server frameworks (and most Python frameworks) because it is non-blocking and very fast. Because it is non-blocking, it can handle thousands of simultaneous real-time connections, which means it is ideal for real-time web services like Facebook.It was mainly created to serve Friend Feed,which is another social sharing service which allows people know what their friends are finding on internet.By the way,Facebook has acquired Friend Feed in 2009.

Thereby Tornado is one of the main web server in Facebook's infrastructure.

Facebook Javascript and Scribe - Technologies in Facebook

2012-02-23T01:54:00.000+05:30

Being enthused by my previous blog post,I have started researching and found out the unknown (or otherwise not understandable) technologies used by Facebook.

Here is a glimpse of some of them :

Facebook Animation (Facebook JavaScript)

It is a customizable library of JavaScript (also called FaceBookJavaScript) which gives developers an easy way to improve their user interface with a line of code or two. All animations are CSS based, so a working knowledge of CSS will really help you out here. It is mostly used to create custom animations using CSS and DOM changes and additions.

While FBJavaScript's Animation library delivers much of the same functionality as other animation libraries, the interface is much different. It's a great improvement over current libraries. Understanding the code doesn't require complex dictionaries of properties, but instead relies on intuitive method chaining.

Scribe

Scribe is a server for aggregating log data streamed in real time from a large number of servers. It is designed to be scalable, extensible without client-side modification, and robust to failure of the network or any specific machine.

In short,it takes your data which is divided into various servers into a single page.
For example,when you log-in and see your profile,all the data is not always (or rather never) stored on a single server. Scribe gets data from all the servers,aggregates it and shows on the client side just as if it is from single server.This shows its great speed and robustness.

It is now used extensively for ticker in Facebook.

Other technologies in successive posts.....

Tools and Technologies that Facebook Use

2012-02-19T19:54:00.000+05:30

Rather then going at a Road Runner speed,I thought it would be better to give off the some of the technical tools and technologies that Facebook uses.

This might be important as it may aware you the per-requisites that you would require to understand the Making of Facebook.

Facebook uses a variety of services, tools, and programming languages to make up its core infrastructure. At the front end, their servers run a LAMP (Linux, Apache, MySQL, and PHP) stack with Memcache.Let’s take a look at exactly what that means.

Following is the most basic technologies that Facebook uses:

Linux & Apache

This part is pretty self-explanatory. Linux is a Unix-like computer operating system kernel. It’s open source, very customizable, and good for security.

Facebook runs the Linux operating system on Apache HTTP Servers. Apache is also free and is the most popular open source web server in use.

MySQL

For the database, Facebook utilizes MySQL because of its speed and reliability. MySQL is used primarily as a key-value store as data is randomly distributed amongst a large set of logical instances. These logical instances are spread out across physical nodes and load balancing is done at the physical node level.

As far as customizations are concerned, Facebook has developed a custom partitioning scheme in which a global ID is assigned to all data. They also have a custom archiving scheme that is based on how frequent and recent data is on a per-user basis.

Most data is distributed randomly.

PHP

Facebook uses PHP because it is a good web programming language with extensive support and an active developer community and it is good for rapid iteration.PHP is a dynamically typed/interpreted scripting language.

Memcache

Memcache is a memory caching system that is used to speed up dynamic database-driven websites (like Facebook) by caching data and objects in RAM to reduce reading time. Memcache is Facebook’s primary form of caching and helps alleviate the database load.
Having a caching system allows Facebook to be as fast as it is at recalling your data. If it doesn’t have to go to the database it will just fetch your data from the cache based on your user ID.

These are the most basic technologies used by Facebook technologies,if you want the detailed or exhaustive list of technologies (obviously with understandable description by me) ,tell in the comments section.

Facebook Homepage to Graph API

2012-02-15T00:52:00.000+05:30

Recently while log-in at facebook,I saw a different picture than other times,as follows:

As pointed above,the part is added is latest thing.Then being curious I went into it and found out data use policy and a lot of interesting tools,as shown below:

It shows "how they use your data" and blah-blah-blah.But the most interesting feature was "InteractiveTools" inside More Resources .Clicking it shows the following photo:

After looking at all the tools,I found the last one:"Graph API",the most important and most useful.But whats under it.....???There are two ways to do it:

1. Obviously by doing all the things I did,and thereby trying to open "Graph API"

2. To read my next blog post about facebook.

Concept behind Facebook

2012-02-15T00:44:00.001+05:30

Facebook divides the social design into three parts:

Community , Conversation , Identity

In simple words,it can be explained as follows:

1.Identity : It means ourselves and how are we represented against others

2.Conversation : It explains how we are communicating with others

3.Community : It refers to people we know and trust

How is Facebook built?

Facebook is built with idea that rather than giving a recipe to eat,let them make it.Means let people make their own community,rather than giving them a ready made community.

Here ends the most boring,but the most important part of Making of Facebook.Now we will understand how does it uses(or exploits....;)...) our data for the purpose.

Facebook Login button faded-the reason behind it

2012-01-26T02:25:00.000+05:30

Recently I observed a rare thing in Facebook,that is my login button has faded on clicking on it.The image is below:


Facebook home-page

But thereafter I remembered a news that blinked in my email , which said about the future attacks of the so-called hacker group Anonymous.Actually the news in a summary way are this :

''Anonymous have announced that they are going to attack on Facebook with some new type of attacks,in near future.''

We know that Facebook is neither a dumbly coded website,nor it has a bad network security professional.So the so-called "new type" of attack would not be any guessed one.

But one flaw that I recognised (and may be even you) is that after entering email-id and password in login,when you click on "Login" button,it responds same like other website i.e. wait for website to load.But actually if you click on it again,then it is sending the GET request to the server again,which is not desirable.If this continues,it might end up in DoS problem for facebook.

To avoid this problem,facebook might have started to disable the login after single-click and fades the button to indiacate that it is being disabled (for a short duration).In this way,they would avoid the "possible" DoS attack by Anonymous.

Hope this works away and avoid the possible attack on our most lovable social networking platform.

Club Registration

2012-01-22T23:08:00.002+05:30

You are definitely genius coming here

2012-01-22T23:07:00.002+05:30

Result in a photo

2012-01-22T23:04:00.001+05:30

Answer to Ping Magic

2011-12-19T21:27:00.000+05:30

The answer to this previous post is Inherited Tag.Below it is explained.

Note : The inherited works only if the website has registered for it. e.g.Facebook and Google has registered for it,others may have not,so they do not get this facility.

Ping Magic

2011-12-17T00:39:00.000+05:30

Yesterday , while going along various sites , I had some problem in network. But the modem was showing that the network was going smoothly. So I used ping command in command prompt , and tried to check it. The network was down.

Then I had a thing in my mind to exploit the command prompt. I started checking various domains in it via ping, like Google and Facebook.

But suddenly when I tried for ww.google.com instead of www.google.com , I found something strange,as follows.

Thereby I tried the same thing with Facebook, by pinging ww.facebook.com,and result was again strange as follows.

I was shocked at this(but afterwards I found the answer).Now it is up to you to give the answer in the comments.The answer would be posted soon...........

DayDreamerman

Problems Detected

Varnish–Power Booster for Facebook

Optimization of News Feed in Facebook

Algorithm for facemash

Algorithm:

Build a Server on your computer using Python

Apache Cassandra and Tornado -Rocking Facebook infrastructure

Apache Cassandra

Facebook Javascript and Scribe - Technologies in Facebook

Facebook Animation (Facebook JavaScript)

Tools and Technologies that Facebook Use

Linux & Apache

MySQL

PHP

Memcache

Facebook Homepage to Graph API

Concept behind Facebook

How is Facebook built?

Facebook Login button faded-the reason behind it

Club Registration

You are definitely genius coming here

Result in a photo

The deduction of your personality based on questionnaire is given in a photo below:

If you like my work,like our Facebook page below:
DayDreamer

Answer to Ping Magic

Ping Magic

DayDreamerman

Problems Detected

Varnish–Power Booster for Facebook

Optimization of News Feed in Facebook

Algorithm for facemash

Algorithm:

Build a Server on your computer using Python

Apache Cassandra and Tornado -Rocking Facebook infrastructure

Apache Cassandra

Facebook Javascript and Scribe - Technologies in Facebook

Facebook Animation (Facebook JavaScript)

Tools and Technologies that Facebook Use

Linux & Apache

MySQL

PHP

Memcache

Facebook Homepage to Graph API

Concept behind Facebook

How is Facebook built?

Facebook Login button faded-the reason behind it

Club Registration

You are definitely genius coming here

Result in a photo

The deduction of your personality based on questionnaire is given in a photo below:

If you like my work,like our Facebook page below: DayDreamer

Answer to Ping Magic

Ping Magic

If you like my work,like our Facebook page below:
DayDreamer