Related posts:

1. MongoDB: Migration from Mysql at Wordnik
2. MongoDB : The bit.ly implementation
3. Scalability updates for Feb 18, 2010
4. NoSQL in the Twitter world

Here are some of the big ones I have mentioned on my twitter/buzz feeds.

• Tools: Real-time Relationship Analytics from large scale graph processing using cassandra  [code here]
• Hadoop 0.21 has been released. The one feature I think is really cool is ability to “append” in hdfs. I found the lack of append feature slightly limiting to what I was trying to do last month.
• Short intro to flume : Flume is a distributed log collection service which can collect and write logs to HDFS. If you have ever had problems aggregating logs, you should take a look at this.
• Topsy just upgraded their backend engine and wrote all about it on their blog. Indexing twitter firehose is no small task, and these guys have done an amazing job.
• Its hard to write incrementing or decrementing counters using an eventually-consistent distributed datastore like cassandra. But its not impossible as shown in this example at git.
• Coudkick has been building their business around cassandra. They are now offering their experiences back to the community here and here.
• Riptano has been putting interesting videos online from their cassandra summits.
• Three papers on load balancing.

Related posts:

1. Scalability updates for Feb 18, 2010
2. Scalability Updates for Jan 26th 2010
3. Scaling updates for Feb 10, 2010
4. Scalability links for March 13th 2010
5. Scalability links for Feb 28th 2010

Most continuous integration environments usually do all of their deployments from trunk. Which means every check-in has to be production quality. Digg’s Andrew Bayer gives a good explanation of how they do code reviews and pre-code check-ins before code is merged into trunk.

Site uptime and reliability depends on a comprehensive QA process to protect against unintentional mistakes. And for rapid deployments one has to abandon manual QA processes in favor of 100% automated testing with the goal of getting close to 100% code coverage. Thats hard if the code is not written in a way which can be tested easily.

But, unit and integration tests alone cannot guarantee quality. In addition to testing code which has been implemented in the application, there needs to be tests to look for things which shouldn’t be implemented. For example, it would be nice to have tests to look for non-parameterized SQL calls in parts of code where it shouldn’t exist. If you know there is a wrong way to do something, write a test case for it so that its caught as soon as someone does it.

Some of this would be easy to do if you already follow a test driven development process where you have to write tests before you write code.

The biggest difference between an organization which follows Continuous deployment and one which doesn’t is in how QA is done. QA becomes a shared responsibility where everyone has to contribute. No matter how many tools or guidelines one publishes, if teams using this process don’t believe in it, the quality and availability of website will suffer. Pascal-Louis Perez (from KaChing) used a diagram like the one here to explain how this “culture” is at the heart of continuous deployment.

“Culture” also explains why most of the older organizations who follow a more traditional form of deployment are having a hard time understanding and adapting to this process.

Are you using Continuous deployments in your environment ? What was your biggest hurdle ?

Related posts:

1. Automated, faster, repeatable, scalable deployments
2. Scaling deployments
3. Scalable Tools: Murder: a bittorent based, file transfer tool for faster deployments
4. Monitoring large-scale application clusters

Follow the slides as you watch the video to understand the talk.

TCP and the Lower Bound of Web Performance – John Rauser from Goodfordogs on Vimeo.

Related posts:

1. Web performance Metrics 101
2. Links on scalability, performance and problems

• Links,  PDF and Docs
  • Apache traffic server – HTTP proxy server on the edge
  • Dont let third parties slow you down
  • Keeping tracking of your performance using slow show
  • Mobile web high performance
  • Progressive enhancements: Tools and techniques
  • Removing the human SPOF
  • Scalable internet architectures
  • Data center infrastructure innovation
  • Closure Compiler: Speeding web applications by compiling javascript
  • Ignite: Apache libcloud
• Videos – http://www.youtube.com/user/OreillyMedia

• Slides

Related posts:

1. Velocity 2009 : Conference presentation slides
2. Talks and slides from various web architects
3. New Talks and Slides links from Aug 5 2007
4. Slides from a Cassandra talk at Mountain View
5. Cloud architecture: Notes from an Amazon talk

• Web is slow
• Avg page load time 4.9s
• 44 resources, 7 dns requests, 320kb
• Lot of 3rd party widgets
• digg/facebook/etc
• Measurements of 3rd party widgets
• Digg widget
• 9 HTTP requests, 52 kB
• scripts block the main page from downloading
• stylesheets blocks the main page from rendering in IE
• Adsense takes up  12.8% page load time
• Analytics takes up < 5%   ( move to async widget )
• Doubleclick takes up 11%
• How to make Google AdSense “fast by default”
• Goals / Challenges
• Minimize blocking the publishers page
• Show the ad right where the code is inserted
• Must run in publishers Domain
• Solution (ASWIFT) – Asynchronous Script Written into IFrame Tag
• Make show_ads.js a tiny loader script
• Loader creates a same-domain iframe (using document.write)
• Loads the rest of the show_ads into the iframe by document.write() of a <script> tag
• This loading of iframe is asynchronous.
• Browser specific surprises
• Problems with parallel downloads of same script in IE
• Iframe creation inside <head> in Firefox has a problem
• Requesting headers in Chrome was buggy
• Forward-Back-Reload behavior is buggy (refetching instead of using cache)
• document.domain vs friendly iframes

Related posts:

1. Builtwith.com : Find out what a website’s frontend is built with
2. Urs Holzle from google on “Speed Matters”

• Average web page – 320kb, 44 resources, 7 dns lookups, doesn’t compress 3rd of its content
• Aiming for 100ms page load times for chrome
• Chrome: HTML5, V8 JS engine, DNS prefetching, VP8 codec, opensource, spurs competition
• TCP improvements
  • Fast start (higher initial congestion window)
  • Quick loss recovery (lower retransmit timeouts)
  • Makes Google products 12% faster
  • No handshake delay (app payload in SYN packets)  [ Didn’t know this was possible !!! ]
• DNS improvements
  • Propagate client IP in DNS requests (to allow servers to better map users to the closest servers)
• SSL improvements
  • False start (reduce 1 round trip from handshake)
    • 10% faster (for Android implementation)
  • Snap start (zero round trip handshakes, resumes)
  • OCSP stapling (avoid inline roundtrips)
• HTTP improvements (SPDY):
  • Header compression
  • Stream multiplexing and prioritization
  • Server push/hints
  • 25% faster
• Test done
  • Download the same “top 25” pages via HTTP and SPDY, network simulates a 2Mbps DSL link, 0% packet loss – Number of packets dropped by 40%
  • On low bandwidth links, headers are surprisingly costly. Can add 1 second of latency.
• Public DNS:
  • reduces recursive resolve time by continuously refreshing cache
  • Increases availability through adequate provisioning
• Broadband pilot testing going on
  • Fix the “last mile” complaint
  • Huge increase of 100x
• More developer tools by Google
  • Page speed, speed tracer, closure compiler, Auto spriter
• More awareness about performance
  • Web search ranking – speed impacts ranking
  • http://code.google.com/speed 
  • Browserscope
  • site performance

Related posts:

1. How To Design A Good API and Why it Matters
2. Speeding up 3rd party widgets using ASWIFT
3. Spanner: Google’s next Massive Storage and Computation infrastructure
4. Google app engine review (Java edition)
5. Google App Engine and Social Apps

• Pace of Innovation – Datacenter pace of innovation is increasing.  The high focus on infrastructure innovation is driving down the cost, increasing reliability and reducing resource consumption which ultimate drives down cost.
• Where does the money go ?
  • 54% on servers, 8% on networking, 21% on power distribution, 13% on power, 5% on other infrastructure requirements
  • 34% costs related to power
  • Cost of power is trending up
• Clouds efficiency – server utilization in our industry is around 10 to 15% range
  • Avoid holes in the infrastructure use
  • Break jobs into smaller chunks, queue them where ever possible
• Power distribution – 11 to 12% lost in distribution
  • Rules to minimize power distribution losses
    • Oversell power – setup more servers than power available. 100% of servers never required in a regular datacenter.
    • Avoid voltage conversions
    • Increase efficiency of conversions
    • High voltage as close to load as possible
    • Size voltage regulators to load and use efficient parts
    • High voltage direct current a small potential gain
• Mechanical Systems – One of the biggest saving is in cooling
  • What parts are involved ? – Cooling tower, heat exchanges, pumps, evaporators, compressors, condensers, pumps… and so on.
  • Efficiency of these systems and power required to get this done depends on the difference in the desired temperature and the current room temperature
  • Separate hot and cold isles… insulate them (don’t break the fire codes)
  • Increase the operating temperature of servers
    • Most are between 61 and 84
    • Telco standard is 104F (Game consoles are even higher)
• Temperature
  • Limiting factors to high temp operation
    • Higher fan power trade-off
    • More semiconductor leakage current
    • Possible negative failure rate impact
  • Avoid direct expansion cooling entirely
    • Air side economization 
    • Higher data center temperature
    • Evaporative cooling
  • Requires filtration
    • Particulate and chemical pollution
• Networking gear
  • Current networks are over-subscribed
    • Forces workload placement restrictions
    • Goal: all points in datacenter equidistant.
  • Mainframe model goes commodity
    • Competition at each layer rather than vertical integration
  • Openflow: open S/W platform
    • Distributed control plane to central control

Related posts:

1. Pregel: Google’s other data-processing infrastructure
2. The real concerns about Cloud infrastructure (as it is today)
3. Versioning data in S3 on AWS
4. Spanner: Google’s next Massive Storage and Computation infrastructure
5. Friendfeed using Mysql for Schema-less data

Performance measurement is often the starting point for most web applications and that can’t be done without understanding what goes on between the browser and the server.

Related posts:

1. Links on scalability, performance and problems
2. TCP and the Lower Bound of web performance
3. Speeding up 3rd party widgets using ASWIFT

• Optimization
  • Don’t over optimize. Could take away precious resources away from critical functions. 
  • Don’t scale early. Planning for more than 10 times the load you currently have or are planning to support might be counter-productive in most cases. RDBMS is fine until you really need something which can’t fit on 2 or 3 servers.
  • Optimize performance on single node before you optimize and re-architect a solution for horizontal scalability.
• Tools
  • Tools are what a master craftsman makes… tools don’t make a craftsman a master.
  • Tools can never solve a problem, its correct use does.
  • Master the tools which need to be (could be ) used in production at short notice. Looking for man page for these tools during an outage isn’t ideal.
• Cookies
  • Use cookies to store data wherever possible.
  • Sign them if you are concerned about tampering
  • Encrypt them if you are concerned about users having visibility into it
  • Its cheaper to use user’s browser as a datastore replication node, than build redundant servers
• Datastores
  • NoSQL is not the solution for everything [ example: so long MongoDB ]
  • Ditto RDBMS
  • Ditto everything else
  • Get the requirements, understand the problem and then pick the solution. Instead of the other way around.
• Automation
  • When you find yourself doing something more than 2 times, write scripts to automate it
  • When users report failures before monitoring systems do, write better monitoring tools.
• Revision control
  • Revision control as much as possible.
  • Provides audit trail to help understand what happened before. One can’t remember everything. Excellent place to search during hard to solve production problems.
• Networking
  • Think in packets and not bytes to save load time.
  • There is no point in compressing a CSS file which is 400 bytes since the smallest data IP packet will store is about 1300 bytes (rest of the packet is padded with empty bytes if the data being sent is smaller).
  • In fact compression and decompression will take away precious CPU resources on server and the client.
  • Instead think of embedding short CSS files in HTML to save a few extra packets.
• Caching
  • Static objects
    • Cache all static objects for ever
    • Add random numbers/strings to objects to force a reload of the object.
      • For example instead of requesting “/images/myphoto.jpg” request “/images/myphoto.123245.jpg”
    • Remove the random ID using something like an htaccess rewrite rule
    • Use CDNs where ever possible, but make sure you understand all the objects part of your page before you shove the problem to a CDN. pointless redirects can steal away previous loading time.
• People
  • When you hire someone for operations team, never hire someone who can’t remember a single production issue he/she was caused. People learn the most from mistakes, so recognizing people who have been on the hot seat and have fixed their mistakes.
  • Allow people to take risks in production and watch them how they recover from it. Taking risk is part of adapting to new ideas, and letting them fail helps them understand how to improve.

Related posts:

1. Disaster Recovery: Impressive RPO and RTO objectives set by Google Apps Operations
2. Thoughts on scalability
3. Automated, faster, repeatable, scalable deployments
4. Scalable logging using Syslog
5. Heroku platform for scalable web applications