Among the dozens of take-aways that I took from the event, the one I’m taking about here is Amazon’s viewpoint on Regions and Availability Zones (AZs).  For those not familiar, AZs are the equivalent to at least one completely independent data center.  Regions are groups of at least 3 AZs.

During James Hamilton’s talk on Wednesday he went out of his way to explain that Amazon does not ever plan on distributing products like DynamoDB across multiple regions.  Hamilton explained that using multiple AZs provides all the redundancy you need to keep your services running indefinitely.    He further explained that until the speed of light improves (never), this will always be the case for Amazon.  Cross region response times are in the 70-80ms range, unacceptable for most transactional applications.  Amazon designs their availability zones to be between 2-10 miles apart, which provides about 2-3ms response times, reasonable for transitional applications (almost as fast as SSD latency).

This was somewhat of an eye opener for me.  He went on to explain that designing systems to be redundant across broad geographies is just a bad idea.  Broad geography based active / passive architectures rarely to work as planned.  Why?  Because you test them too infrequently.  If you’re always running active / active you know your redundancy is working every day.   If you’re bought into this point, then you’ll want an active / active architecture, which inherently is difficult and risky to do when you’re dealing with high latency.   So to eliminate the latency problem, you locate your second or third data center in the same town, linked by dedicated fiber.

This seems logical, but I would really love the data to back this up.  For example, hurricane Sandy’s effect surely was broader than a 2-10 mile radius.  Or how about a tornado that is half a mile wide and travels 20 miles?  While it seems that probability is extremely low, it seems reasonably more probable than some sort of event effecting both LA and New York.  Perhaps the old theory of having a data center on each coast is just not effective taking into account complexity risk?  Amazon thinks so, but do you?

I was recently asked by a few students what they can be doing to get ahead before graduating.  The conversation caused me to go back an dust off an article I published in 2006 regarding my personal experiences leveraging opportunities during college.  This is a revised version of the original post:

Starting Something

In the fall of 1999, I was about 2 weeks into my freshman year at SCSU when I stumbled upon the university’s student web system, a way for students to publish their own websites. Intrigued by the ability to create my own site, I started to explore.  After contemplating the possibilities I decided upon creating some sort of snowboarding related site, a passion of mine at the time.  After becoming hooked on my new found ability to publish to the masses, I quickly found myself spending just about every waking moment outside classes working on the project.  This all eventually led to creation of BoardingPlanet.com , an exploratory project turned startup business, which in retrospect turned out to be one of the more influential experiences in my life.

I was pretty green to start, only knowing HTML and some JavaScript from a AOL member page I had created for my band in high school.  I knew there was more out there but didn’t have much direction.  My declared major at the time was Electrical Engineering, so I had a “C for Engineers” class where I picked up the C language.  I found some resources on the web regarding PERL which is pretty close syntactically with C, so I gave it try.  The first dynamic version of BoardingPlanet included some messy PERL scripts for sending form data to an email address.  The next evolution of BP came when I picked up on the PHP/MySQL stack.  I still remember to this day being introduced to MySQL by a friend who at the time was running a similar site focused on the MN skateboarder community.  I had ridiculously been writing data to flat files.  Needless to say, the discovery of databases was an epiphany moment.  PHP was also awesome.  It was great language for people getting started in web application development as it was freely available and had an extremely strong user community.

Project Turned Passion

At some point BoardingPlanet had become more than just a side project, it was a passion.  Features like a member system (think Facebook for snowboarders), film reviews, interviews and streaming radio consumed just about every free minute of my time, which ended up being something like 9PM – 4AM daily.  I probably re-factored every feature in the site about once a month as I learned the ropes of application design and best practices.  As I stumbled upon new things like the PHP GD library, I would dream up features to exploit it.  My friends at the time thought I was crazy spending so much time on the site.  SCSU's culture at the time was completely inverse to what I was doing (and probably still is).  Most of my friends looked at school as means to a degree which would ultimately get you a job at huge company like General Mills, Cargill or Best Buy.  They really didn't get it, but I didn't let it bother me too much.

It wasn’t just a nerd fest either.  As the site grew, I spent a good amount of time on editorial content, marketing and business development.  On the marketing end, I had built up relationships with other snowboarding sites which drove a significant amount of traffic to the site.  The byproduct of those relationships was also a solid amount of organic search listings.  Other grassroots marketing initiatives (some ideas good, some bad) included printing up 10,000 stickers to give away for free, posters at local ski resorts and flyers in local board shops.  Still today one loan BP sticker exists on the “No parking sign” on the south side of Ritchie Auditorium at SCSU.  Each year I did snowboarding film reviews, mostly as an excuse to get videos for free.  I’d write up emails to the production or distribution companies requesting copies for review, most of them gladly complied.  I also scored other random stuff for the purpose of reviewing, including skateboards, snowskates and even a snow making machine (that was fun).

Sometime in 2000 I received an offer from a company to acquire the site which was quite the memorable moment.  After the excitement settled down I reached out the universities free legal advice group for their perspective, which was a great learning experience.  While I ended up declining the offer, it was a remarkable feeling to have outside validation that what I was building was actually worth something in the market.

Sometime around the start of my senior year of college I started to diverge from BoardingPlanet, spending more time on freelance web work, running the E-Business Club on campus and working on other projects.  After 2 or 3 hosting moves, BoardingPlanet.com was eventually lost in the shuffle.  Today the site is gone, but the experience of building it was life changing.

Moral of the story

College is one of the most unique opportunities in your lifetime, take advantage of it.  There is a lot going on inside of the classroom, but there is even more outside of that.  Exploring the world of software development, business, or whatever your major is, outside of academia is key to hitting the ground running when you complete your degree.

Students may think they don’t have a lot of free time for this type of thing in college with classes, homework and social life, but the reality is once you hit the workforce, get married and have kids your time is even more scarce.  Use the resources that college gives you: time, professors, other students, clubs and organizations.  These opportunities are harder to come by once you are graduated.  So go ahead, start something your passionate about while you’re in school and come out with more than just a degree.

After reflecting upon my experience, here are some tips for future startup weekend goers:

1. Pitch.  - I wasn't planning on pitching, but seized the moment after I realized it was a big opportunity to have 80+ people to evaluate my idea.

2.  MVP - Define your Minimum Viable Product and then go do it.  Focusing in on something that demonstrates the primary value of your app is key.  You don't have enough time to let everything into the product.

3. Ud.com - use this tool to find product or company names that are available across social networks and domain names.  Very useful for brainstorming and doing quick availability checks.

4.  Wordpress Templates - We didn't use WordPress, but in retrospect it may have been a better option.  If you're just looking to put up a smoke screen site, WordPress is money.  Here is a collection of LeanStartup templates:  http://torgronsund.wordpress.com/2010/09/16/7-wordpress-themes-for-launching-your-minimum-viable-product/

5.  Seek out feedback - There are mentors who run around during the weekend giving feedback - use them.   There were even a few groups (CartBrain and RockYourBlock) that really stepped it up and went outside to solicit actual customer feedback.

6.  Use 3G Hotspot. - At events like this WiFi is a dog.  Thanks to our designer / developer Benjamin, we had a phone in hot spot mode which worked really well.

7. Bring a mouse.

It's pretty simple.  Let's break it down:

In a typical test you run the following:

• A - Control
• B - Treatment

You stop the test once you hit your predetermined sample size you spent hours calculating.

Using the Convergence Method, you run the following:

• A1 - Control #1
• A2 - Control #2
• B - Your treatment

By monitoring the convergence in performance of the two exact same controls you're able to understand when you've hit your minimum sample size.  In other words, when the cumulative performance of A1 and A2 come within your required variance the overall results of your test are statistically valid.   An expected variance might be .5% or whatever you choose.

Here's what it looks like:

As you can see above, the the cumulative conversion variance between A1 and A2 decreases over time and eventually converges into a relatively steady variance of approximately +-.1% around hour 13.  It's at this point in time that we consider the overall results of the test valid.  It's also important to note that your measure must be cumulatively computed over time, not at discrete time intervals.

In addition to monitoring the convergence of the controls, you'll also want to consider the length of the test based on potential variability in performance over a given time period.  In this test I chose 24 hours.  You may want to run it longer if you feel day of the week would affect the performance of your control and treatment at independent levels.  Don't get too worried about this though.  Odds are that most time of day and day of week variances will affect your control and treatments equally.

A lot of material out there regarding calculating sample sizes assumes you need calculate it before you run the test.  In the offline world this is usually true.  You need to know how many mailers to print or survey candidates to find.  With internet marketing tests you generally have access to real-time performance measures, which is what enables this method to work.

So put down that stats book and calculator and use convergence in your next test.

MVP is the concept of completing the minimal amount of work in order to prove two things:

1. People want your product
2. There's a business model

You're trying to learn whether or not your assumptions about the world needing your product are true.  The focus is learning.   Not profitability, scalability or defensibility.  Those things come once you've proven that early indicators or measurements support you have a market and model.  The disciple of differentiating features between your market/model and other objectives is where the true value of this process lies.  The process forces you to think objectively about how each feature is going to help you better understand your product's viability in order of importance.  Very often this process is skipped leading to wasted effort on features that do nothing to prove or disprove your assumptions.

For example, you decide to include the ability for users to "remember my login".  You end up spending 1 day of development and testing to complete this functionality.  However, after the first release  to market you realize that users are simply not signing up for your product at the rate you assumed.  You then move into your second iteration focusing in on the sign up process.  In retrospect the one day spent developing the “remember my login” functionality did nothing to better understand the market or model.  It was a great feature, users loved it, but that didn't matter.  Users didn't even like the idea of the product enough to sign up to use the great feature.

So you’re probably thinking, ok fine, eliminate that feature, but what if the signup rates were spectacular and user retention was horrible due to the missing feature?  I'd say that's an excellent problem to have.  You've learned your priority #1 assumption is holding true, now it’s time to focus on retention or whatever your secondary assumptions might be.

Ok, fine.  But what if those individuals I converted into users never come back?  They might not.  That’s ok.  Odds are with your limited test marketing budget you still have the majority of the world’s population left convert into a user.

But what if I receive bad press from those initial users?  Any press at all is a good thing.  Its feedback.  Receiving no feedback is worse.  You don't have a clue why users hate your product.  Great example of this is Twitter.  Twitter received a massive amount of bad press regarding the stability of their platform in their early days and even still today.  I didn't really matter.  Users still used their product because they loved it.  They had the MVP nailed.

This concept can be extremely difficult to do, but the results are worth it.  Removing all kinds of stuff you really think is cool is painful.  It may even completely change the makeup of the product.  But that’s the point.  The increased level of pain up front pays dividends in the long run by getting your product to market faster allowing you to learn and iterate towards a product people love.