Rule the Web

YouTube Architecture

noreply@blogger.com (Trilok Chand S) — Tue, 11 Jan 2011 21:23:00 +0000

Update 2: YouTube Reaches One Billion Views Per Day. That’s at least 11,574 views per second, 694,444 views per minute, and 41,666,667 views per hour.
Update: YouTube: The Platform. YouTube adds a new rich set of APIs in order to become your video platform leader--all for free. Upload, edit, watch, search, and comment on video from your own site without visiting YouTube. Compose your site internally from APIs because you'll need to expose them later anyway.

YouTube grew incredibly fast, to over 100 million video views per day, with only a handful of people responsible for scaling the site. How did they manage to deliver all that video to all those users? And how have they evolved since being acquired by Google?

Information Sources

Google Video

Platform

Apache
Python
Linux (SuSe)
MySQL
psyco, a dynamic python->C compiler
lighttpd for video instead of Apache

What's Inside?

The Stats

Supports the delivery of over 100 million videos per day.
Founded 2/2005
3/2006 30 million video views/day
7/2006 100 million video views/day
2 sysadmins, 2 scalability software architects
2 feature developers, 2 network engineers, 1 DBA

Recipe for handling rapid growth

 while (true)

{ 

identify_and_fix_bottlenecks();

drink();

sleep();

notice_new_bottleneck();

}

This loop runs many times a day.

Web Servers

NetScalar is used for load balancing and caching static content.
Run Apache with mod_fast_cgi.
Requests are routed for handling by a Python application server.
Application server talks to various databases and other informations sources to get all the data and formats the html page.
Can usually scale web tier by adding more machines.
The Python web code is usually NOT the bottleneck, it spends most of its time blocked on RPCs.
Python allows rapid flexible development and deployment. This is critical given the competition they face.
Usually less than 100 ms page service times.
Use psyco, a dynamic python->C compiler that uses a JIT compiler approach to optimize inner loops.
For high CPU intensive activities like encryption, they use C extensions.
Some pre-generated cached HTML for expensive to render blocks.
Row level caching in the database.
Fully formed Python objects are cached.
Some data are calculated and sent to each application so the values are cached in local memory. This is an underused strategy. The fastest cache is in your application server and it doesn't take much time to send precalculated data to all your servers. Just have an agent that watches for changes, precalculates, and sends.

Video Serving

Costs include bandwidth, hardware, and power consumption.

Each video hosted by a mini-cluster. Each video is served by more than one machine.

Using a a cluster means:
- More disks serving content which means more speed.
- Headroom. If a machine goes down others can take over.
- There are online backups.

Servers use the lighttpd web server for video:
- Apache had too much overhead.
- Uses epoll to wait on multiple fds.
- Switched from single process to multiple process configuration to handle more connections.

Most popular content is moved to a CDN (content delivery network):
- CDNs replicate content in multiple places. There's a better chance of content being closer to the user, with fewer hops, and content will run over a more friendly network.
- CDN machines mostly serve out of memory because the content is so popular there's little thrashing of content into and out of memory.

Less popular content (1-20 views per day) uses YouTube servers in various colo sites.
- There's a long tail effect. A video may have a few plays, but lots of videos are being played. Random disks blocks are being accessed.
- Caching doesn't do a lot of good in this scenario, so spending money on more cache may not make sense. This is a very interesting point. If you have a long tail product caching won't always be your performance savior.
- Tune RAID controller and pay attention to other lower level issues to help.
- Tune memory on each machine so there's not too much and not too little.

Serving Video Key Points

Keep it simple and cheap.
Keep a simple network path. Not too many devices between content and users. Routers, switches, and other appliances may not be able to keep up with so much load.
Use commodity hardware. More expensive hardware gets the more expensive everything else gets too (support contracts). You are also less likely find help on the net.
Use simple common tools. They use most tools build into Linux and layer on top of those.
Handle random seeks well (SATA, tweaks).

Serving Thumbnails

Surprisingly difficult to do efficiently.

There are a like 4 thumbnails for each video so there are a lot more thumbnails than videos.

Thumbnails are hosted on just a few machines.

Saw problems associated with serving a lot of small objects:
- Lots of disk seeks and problems with inode caches and page caches at OS level.
- Ran into per directory file limit. Ext3 in particular. Moved to a more hierarchical structure. Recent improvements in the 2.6 kernel may improve Ext3 large directory handling up to 100 times, yet storing lots of files in a file system is still not a good idea.
- A high number of requests/sec as web pages can display 60 thumbnails on page.
- Under such high loads Apache performed badly.
- Used squid (reverse proxy) in front of Apache. This worked for a while, but as load increased performance eventually decreased. Went from 300 requests/second to 20.
- Tried using lighttpd but with a single threaded it stalled. Run into problems with multiprocesses mode because they would each keep a separate cache.
- With so many images setting up a new machine took over 24 hours.
- Rebooting machine took 6-10 hours for cache to warm up to not go to disk.

To solve all their problems they started using Google's BigTable, a distributed data store:
- Avoids small file problem because it clumps files together.
- Fast, fault tolerant. Assumes its working on a unreliable network.
- Lower latency because it uses a distributed multilevel cache. This cache works across different collocation sites.
- For more information on BigTable take a look at Google Architecture, GoogleTalk Architecture, and BigTable.

Databases

The Early Years
- Use MySQL to store meta data like users, tags, and descriptions.
- Served data off a monolithic RAID 10 Volume with 10 disks.
- Living off credit cards so they leased hardware. When they needed more hardware to handle load it took a few days to order and get delivered.
- They went through a common evolution: single server, went to a single master with multiple read slaves, then partitioned the database, and then settled on a sharding approach.
- Suffered from replica lag. The master is multi-threaded and runs on a large machine so it can handle a lot of work. Slaves are single threaded and usually run on lesser machines and replication is asynchronous, so the slaves can lag significantly behind the master.
- Updates cause cache misses which goes to disk where slow I/O causes slow replication.
- Using a replicating architecture you need to spend a lot of money for incremental bits of write performance.
- One of their solutions was prioritize traffic by splitting the data into two clusters: a video watch pool and a general cluster. The idea is that people want to watch video so that function should get the most resources. The social networking features of YouTube are less important so they can be routed to a less capable cluster.
The later years:
- Went to database partitioning.
- Split into shards with users assigned to different shards.
- Spreads writes and reads.
- Much better cache locality which means less IO.
- Resulted in a 30% hardware reduction.
- Reduced replica lag to 0.
- Can now scale database almost arbitrarily.

Data Center Strategy

Used manage hosting providers at first. Living off credit cards so it was the only way.
Managed hosting can't scale with you. You can't control hardware or make favorable networking agreements.
So they went to a colocation arrangement. Now they can customize everything and negotiate their own contracts.
Use 5 or 6 data centers plus the CDN.
Videos come out of any data center. Not closest match or anything. If a video is popular enough it will move into the CDN.
Video bandwidth dependent, not really latency dependent. Can come from any colo.
For images latency matters, especially when you have 60 images on a page.
Images are replicated to different data centers using BigTable. Code
looks at different metrics to know who is closest.

Lessons Learned

Stall for time. Creative and risky tricks can help you cope in the short term while you work out longer term solutions.
Prioritize. Know what's essential to your service and prioritize your resources and efforts around those priorities.
Pick your battles. Don't be afraid to outsource some essential services. YouTube uses a CDN to distribute their most popular content. Creating their own network would have taken too long and cost too much. You may have similar opportunities in your system. Take a look at Software as a Service for more ideas.
Keep it simple! Simplicity allows you to rearchitect more quickly so you can respond to problems. It's true that nobody really knows what simplicity is, but if you aren't afraid to make changes then that's a good sign simplicity is happening.
Shard. Sharding helps to isolate and constrain storage, CPU, memory, and IO. It's not just about getting more writes performance.
Constant iteration on bottlenecks:
- Software: DB, caching
- OS: disk I/O
- Hardware: memory, RAID
You succeed as a team. Have a good cross discipline team that understands the whole system and what's underneath the system. People who can set up printers, machines, install networks, and so on. With a good team all things are possible.

Bay Area Traffic Report - SigAlert

noreply@blogger.com (Trilok Chand S) — Tue, 02 Jun 2009 18:14:00 +0000

A Sig Alert is defined by the California Highway Patrol as "any traffic incident that will tie up two or more lanes of a freeway for two or more hours, as opposed to a planned event like road construction, which is planned separately." Sig Alerts are issued by the CHP and are posted on their Web site, broadcast on radio and television stations throughout California, and signalled to motorists via electronic message signs on the freeways

http://beta.sigalert.com/Map.asp

Watch TV on Internet Free

noreply@blogger.com (Trilok Chand S) — Sun, 29 Mar 2009 18:25:00 +0000

The best way to watch TV on the Internet, download TVU player and watch them free

http://pages.tvunetworks.com/downloads/player.html#

Echo Topic - Turns Worlds into Customers

noreply@blogger.com (Trilok Chand S) — Sat, 21 Feb 2009 14:37:00 +0000

Advertisers:
Reach your customers and capture their attention when they are most receptive – by placing your message directly into the text of high quality websites. EchoTopic® text links deliver highly relevant online advertising with no complex keywords to manage.

Publishers:

EchoTopic provides publishers with new sources of revenue by delivering in-text advertising campaigns.

http://www.echotopic.com/

'Google Latitude' lets you track friends & family

noreply@blogger.com (Trilok Chand S) — Thu, 05 Feb 2009 13:23:00 +0000

Google has launched a tracking service that lets parents keep an eye on their children - and wives keep tabs on their husbands - round the clock.

The software allows owners of mobile phones or BlackBerry hand-held computers to have their whereabouts followed by family and friends anywhere around the world.

Once the service is activated, the location of a person's phone appears as a blue dot on a map on the screen of whoever is allowed to monitor them.

Google Latitude is available on the following mobile devices wherever Google Maps for mobile v3.0 and above is supported:
- Android-powered devices with Maps v3.0 and above. G1 users in the US will be receiving Maps v3.0 in a system update soon.
- Most color BlackBerry devices
- Most Windows Mobile 5.0 and above devices.

If you don't have the support on your mobile use it by adding a gadget on your iGoogle page. I added and invited few friends to test it.

'Google Ocean' launched as extension of Google Earth

noreply@blogger.com (Trilok Chand S) — Wed, 04 Feb 2009 05:24:00 +0000

Google has launched an extension of Google Earth, which will allow web users to virtually explore the ocean with thousands of images of underwater landscapes.
Google Ocean, which will be included in the newest version of Google Earth, will allow users to swim around underwater volcanoes, watch videos about exotic marine life, read about nearby shipwrecks, contribute photos and watch unseen footage of historic ocean expeditions - all from the comfort of their homes.

Add website URL to Search Engines

noreply@blogger.com (Trilok Chand S) — Fri, 23 Jan 2009 12:30:00 +0000

You can submit your URLs to various search engines as shown below.

	Google
	Yahoo!
	MSN.COM Get 1st Rank
	Carnival of Money
	FREE AD BLOG
	Qango.Com
	WhatUSeek.com
	LinksTraffic
	Tower Search
	HotLaunch
	Search Monster
	Infoseek
	Searchit.Com
	Search Secrets SEO
	NetSearch
	Shoula.com
	The-Search-Site
	Free Submit 20
	WebSquash
	WEB4URL.COM

Online File Storage

noreply@blogger.com (Trilok Chand S) — Wed, 19 Nov 2008 12:17:00 +0000

A file hosting service, online file storage service, or online media center is an Internet hosting service specifically designed to host static content, typically large files that are not web pages. Typically they allow web and FTP access. They can be optimized for serving many users or be optimized for single-user storage. Related services are video sharing, virtual storage and remote backup.

Some of the notable file hosting services are:

Box.net
DropBox
FileFront
MediaFire
MegUpload
RapidShare
SendSpace
SteekR
YouSendIt

I am using Box.net which give 50MB of free space.

Web conferencing - Webinar

noreply@blogger.com (Trilok Chand S) — Mon, 03 Nov 2008 14:38:00 +0000

Web conferencing is used to conduct live meetings or presentations over the Internet. In a web conference, each participant sits at his or her own computer and is connected to other participants via the internet. This can be either a downloaded application on each of the attendees computers or a web-based application where the attendees will simply enter a URL (website address) to enter the conference.

A Webinar, or "Web seminar," is an online event designed to facilitate communication between a small number of presenters and a large remote audience using the Internet. Webinars reach audiences more effectively than regular in-person events because there is no need to plan travel or reserve conference space.

A webinar is a neologism to describe a specific type of web conference. It is typically one-way, from the speaker to the audience with limited audience interaction, such as in a webcast. A webinar can be collaborative and include polling and question & answer sessions to allow full participation between the audience and the presenter. In some cases, the presenter may speak over a standard telephone line, pointing out information being presented on screen and the audience can respond over their own telephones, preferably a speaker phone. There are web conferencing technologies on the market that have incorporated the use of VoIP audio technology, to allow for a truly web-based communication. Webinars may (depending upon the provider) provide hidden or anonymous participant functionality, enabling participants to be unaware of other participants in the same meeting.

sign up for a free trail version at http://www.gotowebinar.com

I got a chance to attend Java classes recently, classes are scheduled 1 hour a day after office hours.

Different modes to access Gmail

noreply@blogger.com (Trilok Chand S) — Thu, 30 Oct 2008 10:49:00 +0000

Gmail Modes
If you can't access Gmail, try some of these URLs:

Safe mode - http://mail.google.com/mail/?labs=0. It disables the experimental features from Gmail Labs, just in case some of them are buggy. You can remove some of the features from Gmail's settings page.

Secure mode - https://mail.google.com/. It encrypts the traffic between your computer and Gmail's servers. Use it from public computers, Wi-Fi networks or to bypass some proxies and web accelerators. There's a Gmail setting that redirects the standard version to the secure mode ("Always use https").

Older version - http://mail.google.com/mail/?ui=1. This version has been replaced in October 2007 by a rearchitectured Gmail, but the old version is a little bit faster.

Basic mode - http://mail.google.com/mail/?ui=html. It's the version that doesn't use JavaScript, so it loads faster and it works well with older browsers. Unfortunately, many Gmail features are missing (contacts autocomplete, chat, spell checker, rich formatting) and each click loads a new page. If you like this version, click on "Set basic HTML as default view" at the top of the page.

Mobile mode - http://mail.google.com/mail/?ui=mobile or http://m.gmail.com. This is a simplified Gmail interface for mobile phones that has even less feature than the basic mode. Use it if no other Gmail mode works for you.

iPhone mode - http://mail.google.com/mail/x/gdlakb-/gp/. A more user-friendly mobile version for iPhone and other mobile phones that use WebKit-based browsers.

iGoogle gadget - http://www.google.com/ig/gmailmax. This is the canvas view for the updated Gmail gadget which can be found in the new iGoogle. Some people found that this interface bypasses most corporate filters that prevent them from accessing Gmail at work.

"No browser checking" mode - http://mail.google.com/mail?nocheckbrowser. If you use a cutting-edge new browser and Gmail serves you the basic HTML mode, try this URL to bypass browser detection.

List of Torrent Sites

noreply@blogger.com (Trilok Chand S) — Wed, 15 Oct 2008 05:19:00 +0000

My favorites are 1 & 2
(Install a torrent client, before you download any torrent files. I use Bittorrent client)

Web Hosting: Who's Popular?

noreply@blogger.com (Trilok Chand S) — Wed, 01 Oct 2008 05:09:00 +0000

Top 10 web hosting providers as on Sept 2008

RANK	SHARED WEB HOST	PRICE	SALES INDEX
1		$4.95
2		$4.95
3		$4.95
4		$3.95
5		$3.95
6		$5.45*
7		$5.94
8		$4.95
9		$8.96*
10		$3.88*

Is your blog on Technorati? Know more about it.

noreply@blogger.com (Trilok Chand S) — Wed, 01 Oct 2008 04:32:00 +0000

Technorati is an Internet search engine for searching blogs, competing with Google and Yahoo. As of June 2008, Technorati indexes 112.8 million blogs and over 250 million pieces of tagged social media. The name Technorati is a blend, pointing to the technological version of literati or intellectuals. Technorati was founded by Dave Sifry and its headquarters are in San Francisco, California.

Register your blogs here and get them indexed.
http://www.technorati.com

WWW Schools (w3schools)

noreply@blogger.com (Trilok Chand S) — Thu, 25 Sep 2008 13:12:00 +0000

I found this website very very useful for learning technologies related to world wide web. At W3Schools you will find all the Web-building tutorials you need, from basic HTML and XHTML to advanced XML, SQL, Database, Multimedia and WAP.

With the online HTML editor you can edit the examples and experiment with the code online very easily.

Try it yourself http://www.w3schools.com/

Web Browser History

noreply@blogger.com (Trilok Chand S) — Thu, 25 Sep 2008 09:12:00 +0000

This is a table of web browsers by year of release of major version, in chronological order, with the approximate number of users in millions.

Internet Explorer 8

noreply@blogger.com (Trilok Chand S) — Thu, 25 Sep 2008 07:07:00 +0000

Internet Explorer 8 public beta 2 was released August 27th 2008.

New features in IE beta 2:

New Tab features allow you to open accidentally closed tabs, and group related tabs together using color codes.

Tab isolation prevents a faulty web site from crashing the whole browser. Only the tab with the faulty page will close.

InPrivate allow you to browse the web without saving any browsing data like; passwords, login data, browsing history, and such.

SmartScreen improves the phishing filter from Internet Explorer 7. It uses a phishing address database, and also recognizes some keywords from the URL (like PayPal in paypal.givemeallyourmoney.com)

Download

Google's Android phone

noreply@blogger.com (Trilok Chand S) — Thu, 18 Sep 2008 08:27:00 +0000

Android is a software platform and operating system for mobile devices based on the Linux operating system developed by Google and later the Open Handset Alliance. It allows developers to write managed code in a Java-like language that utilizes Google-developed Java libraries.

Owners of the new Google-powered mobile phone will be able to unlock the handset by drawing a secret shape on the screen.

The new 'signature unlocking' tool was among the features revealed during a sneak preview in California yesterday.

Other highlights include a built-in compass that will allow people to orientate maps as they use their phone to scout out a restaurant or venue, and a customisable homepage that lets people bookmark their favourite web pages.

Android Demo by Sergey Brin and Steve Horowitz

The True Story of Internet

noreply@blogger.com (Trilok Chand S) — Tue, 16 Sep 2008 06:57:00 +0000

Cool program about "Download:The True Story of Internet" on Discovery Channel.

The other 4 parts of the program can be found below
http://www.youtube.com/watch?v=yGE61LMwaLo
http://www.youtube.com/watch?v=--mHEKXBgbI
http://www.youtube.com/watch?v=l8wi9j-TZws
http://www.youtube.com/watch?v=1Lnlc5e26Xg

Popular Web Aggregators

noreply@blogger.com (Trilok Chand S) — Fri, 05 Sep 2008 14:10:00 +0000

Aggregators automatically check a series of RSS feeds for new items on an ongoing basis, making it is possible to keep track of changes to multiple websites without needing to tediously read and re-read each of the websites yourself. Subscribe to RSS feed using the below aggregators, i have been using Google Reader and iGoogle, trust me they are wonderful.

(Add your feed to My Yahoo!)

(Add your feed to Google Homepage/Google Reader)

(Add your feed to Newsgator)

(Add your feed to Rojo)

(Add your feed to My AOL)

(Add your feed to Netvibes)

(Add your feed to Bloglines)

(Add your feed to The Free Dictionary)

(Add your feed to a Bitty Browser)

(Add your feed to Plusmo)

(Add your feed to NewsAlloy)

(Add your feed to Excite MIX)

(Add your feed to netomat Hub)

(Add your feed to fwicki)

(Add your feed to flurry)

(Add your feed to Webwag)

(Add your feed to Attensa)

(Add your feed to ZapTXT)

(Add your podcast feed to ODEO)

(Add your podcast feed to podnova)

(Add your podcast feed to Podcast Ready)

(Add your podcast feed to Pageflakes)

Google released 'Chrome' browser on Windows

noreply@blogger.com (Trilok Chand S) — Thu, 04 Sep 2008 04:45:00 +0000

Google Chrome is a browser that combines a minimal design with sophisticated technology to make the web faster, safer, and easier. Following are some new features
1. Address Bar - Web search. Web history. Address bar. Suggestions as you type. One unified box serves all your browsing needs
2. New Tab page - Every time you open a new tab, you'll see a visual sampling of your most visited sites, most used search engines, and recently bookmarked pages and closed tabs.
3. Application Shortcuts - Use web apps without opening your browser. Application shortcuts can directly load your favorite online apps.
4. Dynamic Tabs - You can drag tabs out of the browser to create new windows, gather multiple tabs into one window or arrange your tabs however you wish -- quickly and easily.
5. Crash Control - Every tab you're using is run independently in the browser, so if one app crashes it won't take anything else down
6. Incognito Mode - Don't want pages you visit to show up in your web history? Choose incognito mode for private browsing
7. Safe Browsing - Google Chrome warns you if you're about to visit a suspected phishing, malware or otherwise unsafe website.
8. Instant Bookmarks - Want to bookmark a web page? Just click the star icon at the left edge of the address bar and you're done.
9. Import Settings - When you switch to Google Chrome, you can pick up where you left off with all the bookmarks and passwords from your existing browser.
10. Simpler Downloads - No intrusive download manager; you see your download's status at the bottom of your current window.

Install the Beta version
http://www.google.com/chrome

Dodgeball - Mobile Social Software

noreply@blogger.com (Trilok Chand S) — Fri, 25 Jul 2008 06:41:00 +0000

Its a mobile social software available in several cities in the United States.

What does it do? Here is an example
The idea is simple: tell them where you are and they will tell you who and what is around you.

We'll ping your friends with your whereabouts, let you know when friends-of-friends are within 10 blocks, allow you to broadcast content to anyone within 10 blocks of you or blast messages to your groups of friends.

For example, you're having drinks at Ace Bar. Send a text message to 36343 ("dodge") with the message "@Ace Bar" and we'll send out a text message telling all your friends where you are AND send you back a message letting you know if any friends-of-friends are within 10 blocks. If you have a camera phone, we'll even send you their picture.

Visit http://www.dodgeball.com and find your friends and friends of friends. You can find me in NY city :)

List of Bookmark and sharing services

noreply@blogger.com (Trilok Chand S) — Fri, 25 Jul 2008 05:34:00 +0000

I have tried Delicious, Google Bookmarks and found them very good. As these bookmarks are on web you can access them from any where in the world. Delicious lists the top and most popular bookmarks in the internet and you can also have RSS feeds to your favorite bookmarks.
Try it by registering at http://www.delicious.com/

Below are the list of bookmarking and services available on web.

Digg It

noreply@blogger.com (Trilok Chand S) — Thu, 24 Jul 2008 11:40:00 +0000

Digg is a place for people to discover and share content across the web, from the biggest online destinations to the most obscure blog. Digg surfaces the best stuff as voted on by digg users.

How do we do this? Everything on Digg — from news to videos to images to Podcasts — is submitted by our community (that would be you). Once something is submitted, other people see it and Digg what they like best. If your submission rocks and receives enough Diggs, it is promoted to the front page for the millions of our visitors to see.

Click here to see my submissions to digg
http://digg.com/users/s3lokchand

Did you hear Scour?

noreply@blogger.com (Trilok Chand S) — Thu, 24 Jul 2008 11:17:00 +0000

It is the next gen search engine with Google/Yahoo/MSN results and user comments all on one page. Best of all we get paid for using it by earning points with every search, comment and vote. The points are redeemable for Visa gift cards! It's like earning credit card or airline points just for searching! Hit the link below to join for free and we will both get points!

http://scour.com/invite/s3lokchand

Its launched on 14th July 08 and you can find their official blog at
http://blog.scour.com/

Click on the below video to know how to use it

List of Open Social sites

noreply@blogger.com (Trilok Chand S) — Thu, 17 Jul 2008 08:24:00 +0000

Get connected to you friends and colleagues using these open social networking sites.
I have already registered with few of them...

http://www.Engage.com/
http://www.friendster.com/
http://www.hi5.com/
http://www.hyves.net/
http://www.imeem.com/
http://www.LinkedIn.com/
http://www.MySpace.com/
http://www.ning.com/
http://www.orkut.com/
http://www.plaxo.com/
http://www.sixapart.com/
http://www.tianji.com/
http://www.viadeo.com/
http://www.xing.com/
http://en.netlog.com/
http://www.cityin.com/
http://www.oracle.net/

If there are additional open social networking sites please write a comment with those details.

Trilok