Most continuous integration environments usually do all of their deployments from trunk. Which means every check-in has to be production quality. Digg’s Andrew Bayer gives a good explanation of how they do code reviews and pre-code check-ins before code is merged into trunk.

Site uptime and reliability depends on a comprehensive QA process to protect against unintentional mistakes. And for rapid deployments one has to abandon manual QA processes in favor of 100% automated testing with the goal of getting close to 100% code coverage. Thats hard if the code is not written in a way which can be tested easily.

But, unit and integration tests alone cannot guarantee quality. In addition to testing code which has been implemented in the application, there needs to be tests to look for things which shouldn’t be implemented. For example, it would be nice to have tests to look for non-parameterized SQL calls in parts of code where it shouldn’t exist. If you know there is a wrong way to do something, write a test case for it so that its caught as soon as someone does it.

Some of this would be easy to do if you already follow a test driven development process where you have to write tests before you write code.

The biggest difference between an organization which follows Continuous deployment and one which doesn’t is in how QA is done. QA becomes a shared responsibility where everyone has to contribute. No matter how many tools or guidelines one publishes, if teams using this process don’t believe in it, the quality and availability of website will suffer. Pascal-Louis Perez (from KaChing) used a diagram like the one here to explain how this “culture” is at the heart of continuous deployment.

“Culture” also explains why most of the older organizations who follow a more traditional form of deployment are having a hard time understanding and adapting to this process.

Are you using Continuous deployments in your environment ? What was your biggest hurdle ?

Related posts:

1. Automated, faster, repeatable, scalable deployments
2. Scaling deployments
3. Scalable Tools: Murder: a bittorent based, file transfer tool for faster deployments
4. Monitoring large-scale application clusters

Follow the slides as you watch the video to understand the talk.

TCP and the Lower Bound of Web Performance – John Rauser from Goodfordogs on Vimeo.

Related posts:

1. Web performance Metrics 101
2. Links on scalability, performance and problems

• Links,  PDF and Docs
  • Apache traffic server – HTTP proxy server on the edge
  • Dont let third parties slow you down
  • Keeping tracking of your performance using slow show
  • Mobile web high performance
  • Progressive enhancements: Tools and techniques
  • Removing the human SPOF
  • Scalable internet architectures
  • Data center infrastructure innovation
  • Closure Compiler: Speeding web applications by compiling javascript
  • Ignite: Apache libcloud
• Videos – http://www.youtube.com/user/OreillyMedia

• Slides

Related posts:

1. Velocity 2009 : Conference presentation slides
2. Talks and slides from various web architects
3. New Talks and Slides links from Aug 5 2007
4. Slides from a Cassandra talk at Mountain View
5. Cloud architecture: Notes from an Amazon talk

• Web is slow
• Avg page load time 4.9s
• 44 resources, 7 dns requests, 320kb
• Lot of 3rd party widgets
• digg/facebook/etc
• Measurements of 3rd party widgets
• Digg widget
• 9 HTTP requests, 52 kB
• scripts block the main page from downloading
• stylesheets blocks the main page from rendering in IE
• Adsense takes up  12.8% page load time
• Analytics takes up < 5%   ( move to async widget )
• Doubleclick takes up 11%
• How to make Google AdSense “fast by default”
• Goals / Challenges
• Minimize blocking the publishers page
• Show the ad right where the code is inserted
• Must run in publishers Domain
• Solution (ASWIFT) – Asynchronous Script Written into IFrame Tag
• Make show_ads.js a tiny loader script
• Loader creates a same-domain iframe (using document.write)
• Loads the rest of the show_ads into the iframe by document.write() of a <script> tag
• This loading of iframe is asynchronous.
• Browser specific surprises
• Problems with parallel downloads of same script in IE
• Iframe creation inside <head> in Firefox has a problem
• Requesting headers in Chrome was buggy
• Forward-Back-Reload behavior is buggy (refetching instead of using cache)
• document.domain vs friendly iframes

Related posts:

1. Builtwith.com : Find out what a website’s frontend is built with
2. Urs Holzle from google on “Speed Matters”

• Average web page – 320kb, 44 resources, 7 dns lookups, doesn’t compress 3rd of its content
• Aiming for 100ms page load times for chrome
• Chrome: HTML5, V8 JS engine, DNS prefetching, VP8 codec, opensource, spurs competition
• TCP improvements
  • Fast start (higher initial congestion window)
  • Quick loss recovery (lower retransmit timeouts)
  • Makes Google products 12% faster
  • No handshake delay (app payload in SYN packets)  [ Didn’t know this was possible !!! ]
• DNS improvements
  • Propagate client IP in DNS requests (to allow servers to better map users to the closest servers)
• SSL improvements
  • False start (reduce 1 round trip from handshake)
    • 10% faster (for Android implementation)
  • Snap start (zero round trip handshakes, resumes)
  • OCSP stapling (avoid inline roundtrips)
• HTTP improvements (SPDY):
  • Header compression
  • Stream multiplexing and prioritization
  • Server push/hints
  • 25% faster
• Test done
  • Download the same “top 25” pages via HTTP and SPDY, network simulates a 2Mbps DSL link, 0% packet loss – Number of packets dropped by 40%
  • On low bandwidth links, headers are surprisingly costly. Can add 1 second of latency.
• Public DNS:
  • reduces recursive resolve time by continuously refreshing cache
  • Increases availability through adequate provisioning
• Broadband pilot testing going on
  • Fix the “last mile” complaint
  • Huge increase of 100x
• More developer tools by Google
  • Page speed, speed tracer, closure compiler, Auto spriter
• More awareness about performance
  • Web search ranking – speed impacts ranking
  • http://code.google.com/speed 
  • Browserscope
  • site performance

Related posts:

1. How To Design A Good API and Why it Matters
2. Speeding up 3rd party widgets using ASWIFT
3. Spanner: Google’s next Massive Storage and Computation infrastructure
4. Google app engine review (Java edition)
5. Google App Engine and Social Apps

• Pace of Innovation – Datacenter pace of innovation is increasing.  The high focus on infrastructure innovation is driving down the cost, increasing reliability and reducing resource consumption which ultimate drives down cost.
• Where does the money go ?
  • 54% on servers, 8% on networking, 21% on power distribution, 13% on power, 5% on other infrastructure requirements
  • 34% costs related to power
  • Cost of power is trending up
• Clouds efficiency – server utilization in our industry is around 10 to 15% range
  • Avoid holes in the infrastructure use
  • Break jobs into smaller chunks, queue them where ever possible
• Power distribution – 11 to 12% lost in distribution
  • Rules to minimize power distribution losses
    • Oversell power – setup more servers than power available. 100% of servers never required in a regular datacenter.
    • Avoid voltage conversions
    • Increase efficiency of conversions
    • High voltage as close to load as possible
    • Size voltage regulators to load and use efficient parts
    • High voltage direct current a small potential gain
• Mechanical Systems – One of the biggest saving is in cooling
  • What parts are involved ? – Cooling tower, heat exchanges, pumps, evaporators, compressors, condensers, pumps… and so on.
  • Efficiency of these systems and power required to get this done depends on the difference in the desired temperature and the current room temperature
  • Separate hot and cold isles… insulate them (don’t break the fire codes)
  • Increase the operating temperature of servers
    • Most are between 61 and 84
    • Telco standard is 104F (Game consoles are even higher)
• Temperature
  • Limiting factors to high temp operation
    • Higher fan power trade-off
    • More semiconductor leakage current
    • Possible negative failure rate impact
  • Avoid direct expansion cooling entirely
    • Air side economization 
    • Higher data center temperature
    • Evaporative cooling
  • Requires filtration
    • Particulate and chemical pollution
• Networking gear
  • Current networks are over-subscribed
    • Forces workload placement restrictions
    • Goal: all points in datacenter equidistant.
  • Mainframe model goes commodity
    • Competition at each layer rather than vertical integration
  • Openflow: open S/W platform
    • Distributed control plane to central control

Related posts:

1. Pregel: Google’s other data-processing infrastructure
2. The real concerns about Cloud infrastructure (as it is today)
3. Versioning data in S3 on AWS
4. Spanner: Google’s next Massive Storage and Computation infrastructure
5. Friendfeed using Mysql for Schema-less data

Performance measurement is often the starting point for most web applications and that can’t be done without understanding what goes on between the browser and the server.

Related posts:

1. Links on scalability, performance and problems
2. TCP and the Lower Bound of web performance
3. Speeding up 3rd party widgets using ASWIFT

• Optimization
  • Don’t over optimize. Could take away precious resources away from critical functions. 
  • Don’t scale early. Planning for more than 10 times the load you currently have or are planning to support might be counter-productive in most cases. RDBMS is fine until you really need something which can’t fit on 2 or 3 servers.
  • Optimize performance on single node before you optimize and re-architect a solution for horizontal scalability.
• Tools
  • Tools are what a master craftsman makes… tools don’t make a craftsman a master.
  • Tools can never solve a problem, its correct use does.
  • Master the tools which need to be (could be ) used in production at short notice. Looking for man page for these tools during an outage isn’t ideal.
• Cookies
  • Use cookies to store data wherever possible.
  • Sign them if you are concerned about tampering
  • Encrypt them if you are concerned about users having visibility into it
  • Its cheaper to use user’s browser as a datastore replication node, than build redundant servers
• Datastores
  • NoSQL is not the solution for everything [ example: so long MongoDB ]
  • Ditto RDBMS
  • Ditto everything else
  • Get the requirements, understand the problem and then pick the solution. Instead of the other way around.
• Automation
  • When you find yourself doing something more than 2 times, write scripts to automate it
  • When users report failures before monitoring systems do, write better monitoring tools.
• Revision control
  • Revision control as much as possible.
  • Provides audit trail to help understand what happened before. One can’t remember everything. Excellent place to search during hard to solve production problems.
• Networking
  • Think in packets and not bytes to save load time.
  • There is no point in compressing a CSS file which is 400 bytes since the smallest data IP packet will store is about 1300 bytes (rest of the packet is padded with empty bytes if the data being sent is smaller).
  • In fact compression and decompression will take away precious CPU resources on server and the client.
  • Instead think of embedding short CSS files in HTML to save a few extra packets.
• Caching
  • Static objects
    • Cache all static objects for ever
    • Add random numbers/strings to objects to force a reload of the object.
      • For example instead of requesting “/images/myphoto.jpg” request “/images/myphoto.123245.jpg”
    • Remove the random ID using something like an htaccess rewrite rule
    • Use CDNs where ever possible, but make sure you understand all the objects part of your page before you shove the problem to a CDN. pointless redirects can steal away previous loading time.
• People
  • When you hire someone for operations team, never hire someone who can’t remember a single production issue he/she was caused. People learn the most from mistakes, so recognizing people who have been on the hot seat and have fixed their mistakes.
  • Allow people to take risks in production and watch them how they recover from it. Taking risk is part of adapting to new ideas, and letting them fail helps them understand how to improve.

Related posts:

1. Disaster Recovery: Impressive RPO and RTO objectives set by Google Apps Operations
2. Thoughts on scalability
3. Automated, faster, repeatable, scalable deployments
4. Scalable logging using Syslog
5. Heroku platform for scalable web applications

Few interesting statistics listed

• Facebook serves 570 billion page views per month (according to Google Ad Planner).
• There are more photos on Facebook than all other photo sites combined (including sites like Flickr).
• More than 3 billion photos are uploaded every month.
• Facebook’s systems serve 1.2 million photos per second. This doesn’t include the images served by Facebook’s CDN.
• More than 25 billion pieces of content (status updates, comments, etc) are shared every month.
• Facebook has more than 30,000 servers (and this number is from last year!)

I’m not sure facebook is really the “largest site” based on servers alone, but its definitely the largest based on unique users in US.

Related posts:

1. Facebook internals
2. Scalability Stories (15th Sept) Mysql Proxy, Cluster Fire System, Facebook apps and Twitter
3. Hive @Facebook
4. Scaling technorati – 100 million blogs indexed everyday
5. Monitoring Cloud health

Whats not mentioned in the slide was Gary’s reference to the number of key changes in 0.7 version of Cassandra. He thinks beta would be out in a month and that it will address a lot of issues which is currently keeping a lot of Cassandra users away. Few interesting points

• 0.5, 0.6 use the same version of SSTABLE (to store data on disk), but 0.7 changes that. This will require some kind of migration if 0.7 doesn’t support reading old versions of SSTABLE.
• until now, one needs 50% disk space available (free) to do compaction operation. This might improve with 0.7
• 0.7 would probably have more support for avro (instead of thrift). He wonders why thrift hasn’t caught on
• Vector clocks coming..
• altering keyspace and column families is not possible on a live system today… might change with future version
• Compression is being thought about…

He strongly urged users to use client libraries which abstract out the internals of Cassandra’s internal workings. It was convincing enough for me to investigate a move from cassandra’s java lib, to “hector” for my java application.

Related posts:

1. Talk on “database scalability”
2. Cassandra as a communication medium – A service Registry and Discovery tool
3. Cassandra : inverted index
4. Cloud architecture: Notes from an Amazon talk
5. Cassandra for service registry/discovery service