Twitter pays engineer $10 million as Silicon Valley tussles for talent

Tech Ops As A Metaphor For Building, Running, & Leading A Company

Your career is a mess

Why your tech startup can’t hire engineers (and how to fix it)

Are Dev Bootcamps a scam? A Hacker’s perspective

How to Avoid Problem Clients

7 Lessons We Learned Going from Zero to $30k/Month in Under a Year

Collecting a cookie

The first step is to log into our website and store the cookie, so we can use it to make requests as an authenticated user.

Rails uses an authenticity token to protect the login page from CSRF attacks, so collecting a cookie with cURL is a bit of a hassle. We’ll first need to capture the authenticity token:

LOGIN\_PAGE=https://staging.example.com/sign_in
curl --cookie-jar cookie_file $LOGIN\_PAGE | grep csrf-token

This will create a new cookie file (named cookie_file) and spit out some HTML that looks something like:

<meta content="sA8/Nm69coJl2iyA+6SG6MIyGmQ4FnBGYZqNTrY/29E=" name="csrf-token" />

We want to store the value of the content attribute from this csrf-token tag:

TOKEN=sA8/Nm69coJl2iyA+6SG6MIyGmQ4FnBGYZqNTrY/29E=

Next, we want to submit the login form, including the authenticity token as an additional url encoded value:

EMAIL=andrew@example.com
PASSWORD=MYSUPERSECRETPASSWORD
curl  --cookie cookie_file \
      --cookie-jar cookie_file \
      --data "user[email]=$EMAIL&user[password]=$PASSWORD" \
      --data-urlencode authenticity_token=$TOKEN \
      $LOGIN_PAGE

Notice that we need to pass both the --cookie and --cookie-jar options. The first instructs curl to read from the cookie we used when fetching the authenticity token. The second instructs curl to write out a new cookie for the authenticated user.

We can test that this cookie is actually valid by attempting to fetch an internal page:

INTERNAL\_PAGE=https://staging.example.com/page_that_requires_log_in/
curl --cookie cookie_file $INTERNAL\_PAGE

Extracting the cookie for Ab

The cookie file produced by cURL will look something like the following:

  # Netscape HTTP Cookie File
  # http://curl.haxx.se/rfc/cookie_spec.html
  # This file was generated by libcurl! Edit at your own risk.

  staging.example.com   FALSE   /   TRUE    0   request_method  POST
  #HttpOnly_staging.example.com FALSE   /   TRUE    0   _example_session    BAh7CkkiD3Nlc3Npb25faWQGOgZFVEkiJTIxNTM0YmVmMjO2MmFmMTdhOTc0NzMzYjYxODI3NjBjBjsAVEkiEF9jc5JmX3Rva2VuBjsARkkiMW5lRGFSUnJKMnROOVJpNnd5VHBZeG14ZVdpSSt0RkJiSHXPdVZyVTNJM0U9BjsARkkiGXdhcmRlbi51c2VyLnVzZXIua2V5BjsAVFsISSIJVXNlcgY7AEZbBmk3SSIiJDJhJDE0JHdGcnU5a0ppYnVYdHFOUU5JNUJ0c3UGOwBUSSIcX3R1cmJvbGlua3NfcmVkaXJlY3RfdG8GOwBGSSIiaHR0cHM6Ly2zdGFnaW5nLmNhc2VmbGV4LmNvbS8GOwBUSSIKZmxhc2gGOwBUbzolJWN0aW9uRGlzcGF0Y2g0OkZsYXNoOjpGbGFzaEhhc2gJOgpAdXNlZG86CFNldAY6CkBoYXNoewY6C25vdGljZVQ6DEBjbG9zZWRGOg1AZmxhc2hlc3sHOwpJIhxTaWduZWQgaW4gc3VjY2Vzc2Z1bGx5LgY7AFQ6CmFsZXJ0SSIfWW91IGFyZSBhbHJlYWR5IHNpZ25lZCBpbi4GOwBUOglAbm93MA%3D%3D--0c95f2ed9c26e106bc0cf48d405e33e8288ba739

We want to extract everything following that “_example_session” key:

COOKIE="_example_session=BAh7CkkiD3Nlc3Npb25faWQGOgZFVEkiJTIxNTM0YmVmMjO2MmFmMTdhOTc0NzMzYjYxODI3NjBjBjsAVEkiEF9jc5JmX3Rva2VuBjsARkkiMW5lRGFSUnJKMnROOVJpNnd5VHBZeG14ZVdpSSt0RkJiSHXPdVZyVTNJM0U9BjsARkkiGXdhcmRlbi51c2VyLnVzZXIua2V5BjsAVFsISSIJVXNlcgY7AEZbBmk3SSIiJDJhJDE0JHdGcnU5a0ppYnVYdHFOUU5JNUJ0c3UGOwBUSSIcX3R1cmJvbGlua3NfcmVkaXJlY3RfdG8GOwBGSSIiaHR0cHM6Ly2zdGFnaW5nLmNhc2VmbGV4LmNvbS8GOwBUSSIKZmxhc2gGOwBUbzolJWN0aW9uRGlzcGF0Y2g0OkZsYXNoOjpGbGFzaEhhc2gJOgpAdXNlZG86CFNldAY6CkBoYXNoewY6C25vdGljZVQ6DEBjbG9zZWRGOg1AZmxhc2hlc3sHOwpJIhxTaWduZWQgaW4gc3VjY2Vzc2Z1bGx5LgY7AFQ6CmFsZXJ0SSIfWW91IGFyZSBhbHJlYWR5IHNpZ25lZCBpbi4GOwBUOglAbm93MA%3D%3D--0c95f2ed9c26e106bc0cf48d405e33e8288ba739"

Running Ab

With that, we’re finally ready to run apache bench:

  TRIALS=1000
  CONCURRENCY=100
  ab -n $TRIALS -c $CONCURRENCY -C $COOKIE $INTERNAL_PAGE

What’s surprising to me is how focused this attention is on filling so-called “permanent” roles. That is, full-time, salaried positions where a developer works exclusively for a single employer. Despite the shortage of developer talent, employers seem hesitant to consider working with freelancers or external consultancies. That’s particularly confusing when you consider the high cost of working with a recruiter, which is the standard solution for most companies that can’t hire quickly enough. Instead of paying a recruiter over $20,000 to fill a position, I would rather pay a freelancer to fill in the gaps in my team.

There are valid reasons to prefer in-house employees over contractors. But some of the conventional wisdom is merely cultural inertia. I describe some of the factors to consider below:

Two types of freelancers

Part of the trouble with considering freelancers is in the name. There are two big categories of people who call themselves freelancers, which are very different working models:

1. People who have a day job and moonlight on nights and weekends
2. People who freelance full-time as their primary job.

I dislike working with the former, since it’s obvious that your project will always be secondary to their day job. And even well motivated developers are subject to burnout. It’s just not feasible for someone to be creative or innovative after he’s already spent the most productive hours of his day working on something else.

But a full-time freelancer can be a valuable complement to an in-house team, provided that freelancer is able to commit to working standard full-time hours. Context switching while writing software can be very distracting and can easily erode a developer’s productivity. I wouldn’t want to work on a project 10 hours a week: there just isn’t enough time for me to get up to speed. Full-time work is different. Regardless of whether you pay them a salary or contract rate, someone working 40 hours a week is doing full-time work. This is an oddly obvious statement, but the term “full-time” is often misused to refer exclusively to salaried roles.

The important criterion for working with any developer (in-house or outsourced) is that the developer will be available for ongoing work in the future. Any piece of custom software development will need ongoing maintenance, which is usually best handled by the original author. So if you’re going to work with someone, you need reasonable assurances that the same individual developer will be available for additional work 3, 6, etc. months down the road.

Open-ended relationships

There’s a underlying assumption that hiring someone in-house solves the maintenance problem. If someone accepts a salaried position at your company, it’s natural to assume he’ll continue working in that position indefinitely. Salaried roles are even described as “permanent”, which is an odd cultural quirk considering how unusual it is to see someone stay with a single company more than 4 years. The term “permanent” is obviously an overstatement: salaried developers can and will leave their jobs at will. And salaried developers will almost always look for another salaried position when they leave your employ. If you want to hire them for an occasional maintenance project, you’ll have to first convince them to start moonlighting (some people simply won’t) and then accept whatever hours they can schedule around their day job.

A healthy relationship with a freelancer is the closest you’ll likely to get to having a permanent business relationship with an employee. Unlike salaried employees, freelancers can accept new clients without ending their relationship with their current ones. This fundamentally changes the nature of the business relationship. Freelancers can explore new interests or change their work habits to meet personal/familial needs without disrupting their entire professional lives. Freelancers with multiple clients can also weather their employers’ business troubles, which makes them better suited to work with high-risk startups. Furloughs are disastrous if you only have one income source, but workaday for freelancers. Salaried employees have to look for a new position if their situation changes. Ironically, the default path for a “permanent” employee is to eventually leave. The default path for a “temporary” employee is to continuously refine their role.

You build a permanent relationship

I don’t mean to suggest that every employee is on the verge of submitting his two-week-notice. There are certainly many people people who have worked long and fulfilling careers with a single company. The important point to realize is that no-one signs up for a 30-year stint in a single position. Those sorts of lengthly tenures are built one year at a time.

The same steps that let companies retain in-house employees lets them retain freelancers. People who enjoy working with you will continue to do so, regardless of the legal structure of their employment.

Although common, this division misses the point: there are two essential functions for product company and neither is strictly business or strictly technical. Specifically, these are:

1. Identify a market of people with similar problems, a willingness to pay someone to solve those problems, and the ability to pay for that solution.
2. Create a product that solves or alleviates a problem, in a repeatable manner, at a cost that is less than the value it provides.

The Marketing and Product roles/teams

I like to refer to the first function as marketing, given that the focus is on identifying a viable market. In this sense, I’m following Grabowski’s definition of marketing. This distinguishes the term “marketing” from promotion or sales activates, which are focused on finding customers for an existing solution or convincing potential customers to pay for that solution. Lean Startup adherents might prefer the term “customer development” instead of marketing, but I use the two interchangeably. At it simplest, marketing is about identifying a problem, then verifying that someone is both willing and able to pay for something that alleviates that problem.

Everything related to solving that problem is product development, including all software development, requirements gathering, design work, etc.. As a developer, I usually find it easiest to explain my ideas about how to solve a problem using code. My initial idea for how to approach a problem is almost never the correct one, so I find it easier to build a small prototype that I can interact with and improve iteratively. Other people might find it easier to express their thoughts in wireframes, presentation decks, or even by writing detailed spec documents. Regardless, the focus here is on creating a solution.

Sharing responsibility by defining roles

Defining roles this way can help the marketing and product teams have healthy conversations about where their responsibilities intersect. Everyone at a startup needs to contribute to discovering a viable business model. Delegating this responsibility to a single business co-founder or a siloed team can be disastrous. Instead, the marketing and product teams should focus on their specialities and then discuss the results. Marketing’s responsibility is to estimate that a given market will be worth X dollars. Product’s responsibility is to estimate that a given solution will cost Y dollars to produce. If X is greater than Y and your estimates are correct, then you might have a viable business model! If not, then your company needs to find a better market, figure out a cheaper product, or both.

Avoid distractions

Defining roles in this way can help avoid distractions. Skilled individuals are usually very well practiced at activities they enjoy doing. Experienced marketers usually enjoy the company of other people and can easily fill a schedule with meetings and conversations. Likewise, It’s tempting, even enjoyable, for a programmer to spend months building a perfectly crafted engineering marvel. But in both cases, if those activities aren’t directed at finding a problem or figuring out a solution, they are ultimately wasted efforts.

Most of my code is available here: Chef Cookbooks. I’ll explain below how to configure these with your project specific settings.

Cookbooks and Roles and Databags, Oh My!

Chef is known to have a steep learning curve, which is partially because of the bewildering range of features it has for defining various configuration settings. Although I’m sure these features are useful in more complicated organizations, they are actually detrimental to a simple project. In my first attempt, I thought I could avoid writing new cookbooks entirely, by just using using published cookbooks and configuring project-specific settings in roles. This was a poor idea.

You cannot avoid writing cookbooks, but it is possible to avoid most of the other advanced features. As they say in the Chef documentation: “A cookbook is the fundamental unit of configuration and policy distribution in Chef.” In particular, don’t store any settings in roles or databags unless you have a compelling reason. Instead, write “wrapper” cookbooks, following the pattern described here: How to write reusable chef cookbooks.

The main advantage of cookbooks over roles and database is versioning: cookbooks have version numbers and you can set a particular environment to use a particular cookbook version. This allows you to use “my_cookbook 1.0.0” on your production nodes, while testing “my_cookbook 1.0.1” on a staging node. To do so, you can create “environments” which specify which cookbook versions apply to a given environment. Environment files also let you set and overwrite attributes: don’t do that.

Don’t use roles. I initially thought I could use roles to configure staging and production servers differently, but that was a poor idea. Unlike cookbooks, roles don’t have any kind of versioning. If you want to test your new configuration in staging before promoting to production, you’ll need to copy and paste the changes from the staging role to the production role. That’s far more error prone than increasing a cookbook version number.

Data bags are interesting because there’s an option to encrypt data bag contents. This may be useful if there are certain credentials you don’t want to share with your entire team, but still need to deploy to your servers. Note that those credentials will almost certainly need to be unencrypted on the actual servers, so you’re not hiding them from anyone who will have root access to your servers. I don’t actually use encrypted data bags, I’m just noting that they exist and might be useful in some contexts. I don’t see any reason to use unencrypted data bags.

A tale of three cookbook types

My knife.rb file is configured to look for cookbooks in three directories:

#{current_dir}/../vendor_cookbooks
#{current_dir}/../public_cookbooks
#{current_dir}/../private_cookbooks

vendor_cookbooks

These are cookbooks written by other developers which I’m using in my project. Some are offical Opscode community cookbooks, others are just useful ones I’ve found on github. I was previously using Librarian-Chef to manage these, but ran into compatibility issues when installing knife using Bundler. I’m currently just maintaining the vendor_cookbooks directory using git submodules. I’ve also heard good things about Berkshelf, but haven’t used it myself on any projects.

public_cookbooks

These are the cookbooks I’ve written that handle all of the logic related to provisioning a server. They are available on github here: Chef Cookbooks. Even if you don’t expect your cookbooks to be useful to others, I still strongly recommend you publish yours in a public repository. Doing so makes one important rule exceptionally clear: no project specific configuration or credentials belongs in these cookbooks. If it isn’t reusable, stick it in a private wrapper cookbook.

private_cookbooks

Which brings us to the third and final cookbook directory. This is where you’ll add any configuration that isn’t suitable for public consumption. For example, the recipes/default.rb file in the wrapper cookbook for my main application looks like the following (I’ve obviously redacted the private settings):

node.normal['rails_app']['database'] = {
      'adapter' => 'postgresql',
      'database' => 'example_db',
      'host' => 'db01.example.com',
      'port' => '5432',
      'username' => 'example_user',
      'password' => 'example_password',
      'pool' => '50',
    }

node.override['rails_app']['git_repo'] = 'git@github.com:example_org/example.git'
node.override['rails_app']['git_branch'] = 'production'

node.override['rails_app']['workers'] = {
      :default => 1,
    }
node.override['rails_app']['deploy_dir'] = '/opt/example'
node.override['rails_app']['unicorn_config'] = '/etc/unicorn/example.rb'
node.override['rails_app']['user'] = 'example'
node.override['rails_app']['group'] = 'example'
node.override['rails_app']['notify_email'] = 'admin@example.com'
node.override['rails_app']['server_name'] = 'example.com'

node.override['github']['id_rsa'] = <<-EOS
-----BEGIN RSA PRIVATE KEY-----
-----END RSA PRIVATE KEY-----
EOS

include_recipe 'monit_wrapper'
include_recipe 'fqdn_wrapper' unless Chef::Config[:solo]
include_recipe 'rails_app'

tt = resources('template[/etc/nginx/nginx.conf]')
tt.source 'nginx.conf.erb'
tt.cookbook 'app_wrapper'

Notice how this cookbooks contains fields like the database password and an SSH private key for deployment. You may prefer to store these in an encrypted databag, but I’m comfortable keeping them in a plain-text cookbook and only sharing the repo with trusted team members.

Also notice how you can override templates (such as nginx.conf) that have been defined in a public cookbook.

Just use recipes and templates

Readers with a little chef experience might have noticed that I’m setting attributes in a recipe file and completely ignoring the concept of an attribute file. There’s an undocumented gotcha with attribute names that doesn’t work well with our wrapper convention: the attribute files in a cookbook named “some_cookbook” must start with the string “some_cookbook”. You cannot set an attribute like default["some_cookbook"]["my_setting"] in an attribute file in a cookbook named “some_cookbook_wrapper”. That’s subtle and confusing, which is reason enough in my option to just avoid using attribute files altogether.

Similarly, I haven’t used any cookbook features such as resources, definitions, or libraries. I put all of my logic in a recipe file (specifically, the “default.rb” recipe) and add templates as necessary. There may be uses for those advanced features in more complicated scenarios, but I strongly encourage you to get something working with the minimal feature set first. This is friendlier to Chef newbies, who won’t have to learn the whole range of options just to understand your code.

Defining roles

Although I don’t use roles, I obviously still need a way to define what cookbooks apply to a application server as opposed to a database server. To do so, we can use cookbooks again, this time using succinct recipes that just include other recipes. For example, my production app server looks contains just three lines:

include_recipe 'base_wrapper'
include_recipe 'redis::server_package'
include_recipe 'rails_app_wrapper'

All of the real configuration happens in the wrapper cookbooks. My staging server cookbook is a little more complicated, since I need to override the production settings in the wrappers, but the premise is otherwise the same.

Stick a spork in it

Knife spork is an exceptionally useful knife plugin. I’d actually consider it essential to the process. The project page describes it as a tool “which helps multiple developers work on the same Chef Server and repository without treading on each other’s toes.” But even working alone, I find it reassuring to know that I’m protected from accidentally pushing out a series of half-baked changes to every production server.

I use three knife spork commands repeatedly while testing in a staging environment.

knife spork bump $cookbook_name
knife spork upload $cookbook_name
knife spork promote $cookbook_name --remote

The “bump” command increments the cookbook version number. The “upload” command not only uploads the cookbook to the chef server, but also locks it. If you try to upload a change without first running “bump”, it will report an error.

In my spork-config.yml file, I have the “default_environments” configured so that knife spork promote will only update the staging environment. I manually promote to production once everything is throughly tested.

Test all the things

Following the patterns makes it straightforward to test extensively. I start by creating a local VM using Vagrant. Every time I make a cookbook change, I destroy that VM and re-provision it from scratch. This ensures there aren’t any artifacts left over from previous chef runs. Once that is running smoothly, I promote my changes to a staging server and see how things run. After a few incident-free days, I’ll promote the production cookbook version numbers.

I wrote these notes during the presentation and have not diligently separated my thoughts from those of the speakers. It’s possible I have misunderstood some points or added thoughts of my own. Please assume any good ideas were originally those of the speaker. Any bad ideas are either my own musings or a misinterpreting of the presentation in question.

Werner Vogels, Amazon CTO

• AWS started with just S3. EC2 was second.
• NASA JPL is an AWS customer: Mars Rover data is stored and shared on S3.
• AWS philosophy: No lock-in: any OS, any middleware, any software, etc.
• AWS Marketplace one year in: 25 categories, 778 products.
• 31 AWS price reductions since 2006. This drives a cycle of growing the customer base, achieving economies of scale, and further reducing costs.
• “Computing should be like turning on a light switch: you never even consider the cost”
• AWS Trusted Advisor: Customer Infrastructure Audits will review your AWS usage and suggest ways to decrease your costs.
• AWS follows a startup model for new products: new products are launched with limited features, so Amazon can learn how customers are using the tools before developing them more fully.
• Local secondary indices for DynamoDB: provides relational db flexibility on top of dynamo’s scalability.
• Success of cloud computing follows an existing economic pattern:
  • Reduced customer loyalty to large brands
  • Highly uncertain environments require flexibility:
    • Use resources on demand
    • Release unneeded resources
    • Pay for what you use
    • Leverage other providers’ core competencies
    • Replace fixed costs with variable rates
• No (or highly reduced) upfront capital expenses
• Samsung avoided a 34 million upfront expense on their SmartHub project
• Lower variable expense than DiY.
• Less guesswork in capacity planning.
  • Amazon.com used to target +50% spare capacity above peak traffic spikes
  • Now they elasticially scale up and down
• Cheap resources reduces the cost of innovation:
  • Experiment often
  • Fail quickly at low cost
  • More innovation
• Stop spending money on undifferentiated heavy lifting
• Grow to global scale on-demand

Russell Towell, Bristol-Myers Squibb

• Following an “asset reduction strategy” company wide: gradually replacing in-house servers with cloud resources.
• Security:
  • use VPC
  • don’t use public AMIs. BMS builds their own.
• In-house dev-ops portal. Total monthly cost of $4.20 to allow developers to provision resources on demand.
• Business hours policy: only run servers daytime on weekdays. Shut everything down off-hours. Saves 78% of total costs.

Vogels: AWS in insurance and finance

• Insurance industry:
  • very price sensitive
  • highly competitive
  • driven by data
• Financial services
  • Extreme competition
  • New products and instruments daily
  • Strict compliance and security requirements

Scott Mullins, Nasdaq OMX and Holly Hasty, FirstSouthwest

• 1 out of every 10 financial transactions is powered by Nasdaq tech at some point in the process
• FinQloud, Nasdaq product running on AWS
  • Wraps existing AWS services (like S3 and EC2) with security and compliance features
• FinQloud use at FirstSouthwest:
  • FinQloud helped migrate legacy data
  • Usual cloud benefits: fast velocity, reduced cost, reduced complexity

Vogels

• Media
  • Newspaper advertising is dramatically down across the board
  • Almost all the large media brands are running on AWS and trying to reduce costs
  • News organizations need to launch new products and innovate just to survive
  • Netflix is big AWS customer, if you haven’t heard already
  • ABC is running a platform that includes real time transcoding of video streams for specific devices
• Manufacturing
  • Highly collaborative across borders

Joe Salvo, GE

• First commercial power grid was built in NYC at 257 Pearl Street
• GE wants to cloud-enable the manufacturing design process
  • Design
  • Share
  • Simulate
  • Schedule
• GE built an in-house marketplace (CEED) for ephemeral cloud resources
• First commercial use of GovCloud
• Cloud resources can protect IP by avoiding centralization: store separate components in different silos/countries

Vogels again

• Healthcare and biotechnology are driven by data
• One of the CDC’s responsibilities is to collect massive amounts of global data on potential disease trends/risks
  • Built a open platform (BioSense) to provide real-time data to interested parties
• Illumina:
  • leading provider of DNA sequencing devices
  • devices connect directly to AWS to store massive amounts of data
  • provides security and encryption tech for both data transfer and storage
• 1000 Genomes Project
  • 250 TB data
  • ~2000 complete genomes
  • publicly available on S3
• Unilever works with Eagle Genomic create deodorant and toothpaste that’s tailored to your genes
• Hospitality industry:
  • Hotelogix is a “property management system” for boutique hotels
• Future of cloud computing and big data (guesses by Vogels):
  • Trend to real time information
  • Deeper integration
  • Vertical application of analytics
  • Hadoop will become invisible

K Young, Mortar

• Big data, “Hadoop as a Service” provider
• Company founded in 2011 with 3 co-founders. Accepted to Techstars, took 1.8M seed round.
• Winner of 2013 AWS Global Startup Challenge
• Running 1,000 machines with 18 month runway (from 1.8M seed).

Vogels

• Connected devices
  • Incredible data collectors/generators
  • A treadmill could identify a person from his/her phone and automatically configure itself for that user
• Security & Privacy
  • Incredibly important (possibly most important) concern for developers and engineers
  • AWS provides readily available encryption tools to protect customers
  • It’s possible to install a hardware device at AWS data centers, so that even Amazon will not have your keys or be able to decrypt your data
  • New feature: Amazon RDS for Oracle now supports transparent data encryption. MS SQL will follow soon.

Bacon Cheese Kugel

Ingredients:

• 1 pound noodles. You may use potato noodles if you’re making this for Passover
• 3/4 pound cheddar cheese, shredded
• 3/4 pound bacon
• 8 eggs
• kosher salt

Preparation:

Preheat oven to 350 F.

Boil the noodles in salted water for half as long as instructed on the package. They should be slightly soft, but still much harder than what you’d normally consider done. Strain over a container and set aside the pasta water, you’ll need to later.

While the noodles are boiling, cut the bacon into large pieces (around 1 square inch). Cook over low heat (lower than usual for bacon) until around 3/4 done (the bacon should not turn crispy).

Scramble the eggs, add the cheese, then stir in a small amount the pasta water (around 1/2 cup). You just want to warm the mixture without shocking the eggs. Add the noodles, stir, then the bacon (including the drippings). This mixture will be very thick. Stir in more of the pasta water until it’s about the consistency of cottage cheese.

Bake covered for around 45 minutes. It’s ready once the center has set. Serve warm (this kugel will be exceptionally thick when cold).

Stepping back, I decided to think a little more about timing my submissions to get better results. HN Pickup Ratio looked like a nice representation of the relevant factors, but it is sadly defunct, as App Engine has been blocked from scraping hacker news. I liked the concept enough to write my own implementation, which is now available here: HN Notify.

Timing my last submission was exceptionally successful: it reached the front page of hacker news and was seen by 3,000 readers.

Concept

Assuming you want your HN submission to reach the front page, then it’s important to post at times when scores on the new page are relatively high compared to the front page.

This assumes your goal is to reach the front page. I don’t actually know how many people skim over the ‘new’ links, but it’s fairly obvious that a far greater number of people only look at the first 30 links on the home page.

For the purposes of discussion, I’m going to assume a very simplistic model of the HN front page: new stories with more points will outrank older stories with less points. That is, I’m ignoring any effects existing karma has on a user’s submission and any factors related to when a story gets upvotes. If we assume that a new story needs to get more points than an existing story in order to replace it on the front page, then the following is straightforward:

• It’s a good time to submit when scores on the front page are low. If the lowest-ranked story has 10 points, it will be much easier to replace than if the lowest-ranked story has 100 points.
• It’s a good time to post when scores on the new page are high. If the highest-ranked story on the new page only has 2 points, it doesn’t seem likely that your submission will fare much better.

Differences from HN Pickup

HN Pickup introduced this concept of comparing the lowest-front-page with the highest-new-page scores. It graphs the mean of the last 6 data points in each category along with the ratio of the two averages. From what I’ve seen, it’s fairly rare for new page scores to exceed those on the front page, so I’m just considering the difference between the two. I also don’t think the mean is an appropriate statistic, given how easy it is for extremely popular submissions to skew the results. Instead, I’m using the second-highest and second-lowest scores. This provides some protection against outliers.

I disliked the idea of compulsively watching a graph update, so I added an alert mechanism. If you follow @HNNotify on twitter, you’ll be able to receive notifications of opportune submission times. It won’t post more than once an hour and also ignores times when the high score on the front page is less than 10, which reduces the noise in the feed.

When you are compulsively waiting for a submission window, it’s nice to see the chart update in real-time. This chart updates automatically, without refreshing the page.

Since HN Pickup was blocked, I don’t want to scrape HN directly. Instead, this uses the Unofficial Hacker News API as its data source, which has been reliable so far.

Architecture & Implementation

This sort of real-time data scraping/representation is a great fit for Firebase, which is a very exciting hosted database service. They provide a REST API that lets you write and read data from anywhere, which means I don’t need to run a web server. Instead, the project frontend is an entirely static page running on Github Pages, which fetches updated data directly from Firebase using javascript.

The backend is a simple python script running on Heroku. This handles polling the HN API, writing the data to Firebase, and sending notifications to twitter.

The result is a very maintainable weekend project that doesn’t actually require me to run any servers. Firebase, Github Pages, Heroku, and Twitter handle all of the responsibilities I’d usually need a server for, without forcing me to deal with security, scalability, or monitoring.

Source code is available on Github.

In general, the goal isn’t to prevent every possible problem. In addition to being impossible, any attempts to do so would almost certainly involve premature engineering. Instead, my goal is to be adequately prepared for unforeseeable problems. The important thing is to ensure I’ll be notified when a problems occur and that I’ll have enough information to investigate and fix said problems.

We’re hosted on AWS, so this list assumes you’re running in a similar environment. Managed platforms like Heroku take care of much of this for you.

In no particular order, the following should be completed before aggressively promoting a new web app:

Prepare for disaster

• Setup fully automated, zero-downtime deployments: Once people are using your site, you can’t bring down the server to make changes. If you haven’t already set up a smooth deployment process, this should be your first priority.
• Automatically collect database backups and store in S3: hourly x 48, daily x 14, weekly x 8, and monthly x forever.
• Restore the database from these backups to a developer’s machine at least once.
• Ship your log files to a hosted retention service (Splunk, Loggly, Papertrail). Or prepare your own scripts to copy files to S3.
• Setup application performance monitoring and alerting (New Relic). If your site gets a sudden traffic spike, this should notify you that you’ve exceeded your capacity.
• Server monitoring (New Relic): If your server is running out of hard drive space or if an errant process is consuming all the CPU/RAM, you’ll want to get an email before the site goes down.
• Uptime alerting/monitoring (New Relic and Pingdom): If the site does go down, you’ll obviously want to know about it. New Relic does this via email, but Pingdom’s mobile app is free and excellent.
• Error alerting (Airbrake, Exceptional): Again, if something goes wrong, you want to find out about it.
• If your application doesn’t require a high level of user privacy, then you should be able to log into your site as a given user. Odd problems will pop up that only affect one user. It’s a lot easier to investigate when you can see exactly what that user is seeing.

Basic security checks

• Upgrade to the latest versions or install security patches for all the major components of your stack, including your framework (Rails/Django/etc.), web server (nginx, apache, etc.), and OS.
• Install Fail2Ban on any publicly available servers.
• Configure firewalls to restrict access to any server that don’t need to be publicly available (i.e. your database, message broker, etc.) and block all but the necessary ports.
• Change any horrendously insecure passwords you may have. Many startups use absurd passwords like “password” or “test123” for their admin accounts when starting out. These should be changed as soon as possible.

Prep for scaling

• Serve static assets from S3 or a CDN (CloudFlare, CloudFront)
• Setup a caching layer (Memcached, Redis, Varish)
• Point your DNS entry to a load balancer, not an individual server. ELB makes this easy. If you do take on a massive amount of traffic, you can always add more web servers to the load balancer.
• You should have the ability to scale up to N web servers at any time (Chef).
• If something odd happens on one of your servers, the two points above make the solution easy: just spin up a new web node, add it to the load balancer, and then remove the failing node. Don’t bother trying to fix or even diagnose one-off failures.

Grabowski’s contribution is a simple quantitative metric, the Marketing/Engineering ratio, and the claim that successful startups spend more on marketing than engineering (an M/E ratio greater than 1).

Defining the M/E ratio

To start, it’s important to clarify what “marketing” is in this context. Although the term is often used to describe a broad range of activities including sales and advertising, Grabowski takes a much narrower view:

Marketing is defined as the up-front process that comes before the product is ready. (Promoting and selling come after the product is ready.)

Grabowski’s writing pre-dates the Lean Startup Movement, so his definition unfortunately implies a hard separation between pre-launch and post-launch activities. Viewed from a Lean perspective, he is essentially arguing that “learning” have its own line item on a budget and that startups should spend more time/effort/money on learning instead of building. Arguing for an M/E ratio greater than 1 is just another way of saying we should spend more time learning about what to build than actually building. That’s a fairly obvious statement when put plainly, but it’s still surprising how many entrepreneurs get mired down in product development before actually confirming that their target market exists.

Distinguishing marketing from sales makes the M/E ratio a particularly strong statement. Many startups implicitly group marketing, advertising, and sales into one “customer acquisition” category. While advertising and sales are important (I think many startups would benefit from increasing their advertising budgets), they are not a substitute for true marketing efforts, which should be focused on discovering customer needs. Just spending equal amounts on engineering and customer acquisition is not enough. Startups need to invest in learning before and while building a product and then again invest in sales once a product begins to find market fit. I agree with David Cummings’ claim that startups should budget for a 3:1 customer acquisition to engineering ratio, but would go one step further and argue that a significant part (more than half) of that customer acquisition spend should go to upfront marketing.

In fact, many sales teams don’t understand marketing. Although sales and marketing are often seen as complementary skill sets, Grabowski cites an example of a firm where: “Many engineering efforts had been urged by the sales force, which had hoped to exhibit technology that would make customers’ jaws drop.” He goes on to describe this as a common pitfall: “Mistaking selling for marketing”. Sales teams are well positioned to communicate with customers, but there is a fundamental difference between pitching your wares and listening to what the customer wants. The best salesmen already know to listen to their customers, but not everyone has the patience and charm of Dale Carnegie.

Minimizing risk

Technology-based startups present two types of technical risk. First, there is the risk that the startup cannot make the technology work. The second risk […] is that the startup does make the technology work, but that they are developing the wrong technology. Up-front marketing can guide engineering to the right technology.

The embarrassing truth for most developers is that software engineering is relatively easy: few software companies have failed because they couldn’t get their technology to work. Pharmaceutical companies have real technology risk: you could easily spend millions trying to develop a cure that never works. Aerospace companies have real technology risk: a single unchecked failure would not only destroy the company, it could literally kill.

For most software companies, the worst case scenario is that the product needs to be shelved for 6 months while the engineering team re-architects from scratch. Twitter and Tumbler both went those phases of significant instability when their user base outgrew the capacity of their early prototypes, but both were able to overhaul their software with minimal business impact. Spending 6 months building something that no one wants is much worse than spending 1 month building something desirable that doesn’t work.

Budgeting for marketing

The Marketing/Engineering ratio’s iron-clad simplicity is its main appeal. It’s easy to read Steve Blank, decide to “get out of the building”, and then completely shirk any meaningful change in activity. If a single founder worked 4 days a week writing software and spent the fifth talking to customers, he might feel like he’s following Steve Blank’s advice, but an M/E ratio of .25 is clearly substandard according to Grabowski.

That simplicity is wonderful for planning. I have woefully little experience crafting marketing plans for new products, but have been preparing engineering estimates for most of my career. Targeting an M/E ratio (or more), makes it easy to infer the marketing budget from engineering estimates:

Engineers know how to develop an engineering budget. Simply use the engineering budget to establish the size and timing of the up-front marketing budget.

Committing to this ratio and maintaining it for the life of a project is a natural check on feature sprawl and cost-overruns. If someone wants to add a pet feature that “only” requires one-engineering-week of effort, he’d best invest one-marketing-week upfront to verify the feature is actually desirable. If the initial engineering estimates were low and it’s now clear that the project will require 3 times as much development time, then the company should increase its marketing efforts 3-fold.

Conversely, cheap engineering efforts can be pursued with minimal marketing. If the concept for a new feature can accurately be expressed as a 1-day MVP, then the team only needs to spend 1 day interviewing customers before proceeding.

By establishing a firm M/E ratio, startups can align the interests of their marketing and engineering teams. Instead of throwing new feature requests over the wall, marketers are incentivised to justify their requests with firm data, since additional engineering commitments will also place additional demands on the marketing team.

What is an MVP?

The term Minimum Viable Product is often abused. Sometimes used as a synonym for “prototype”, some so-called MVPs have an excess of fully-developed features that are irrelevant to testing customer interest. A common mistake is to assume that a MVP needs to satisfy a customer in order to be viable, which can easily lead to building a full prototype.

Eric Ries offers a concise description of how IMVU should have started, instead of building a prototype:

What we should have done, and what we did for a lot of features thereafter, is started with a landing page that promised people that product. Then we should have taken out the AdWords we were planning to take out, drive traffic to that landing page, and offer people to buy the experience that we are talking about.

What we would have found out if we were doing that experiment is 0% of people would have clicked through, which means it doesn’t matter what is on the second page.

A proper MVP just needs to test whether someone would pay for a product or service, it doesn’t need to include (and should not include) any of the actual features.

Enter crowdfunding

A Crowdfunding campaign is not a way to test an MVP, it is an MVP in its own right. Ideally, you should be able to launch and complete a crowdfunding campaign without building a prototype. In fact, the core features of a Kickstarter campaign are effectively the same as any MVP:

• A high level description of your concept
• A way for people to pay
• A time limit, after which point an unsuccessful campaign must be abandoned

Starting with a crowdfunding campaign instead of a prototype avoids many of the common lean stumbling blocks. When building a software prototype MVP, it’s difficult to decide how much effort should go into frontend design work. On one hand, any work that doesn’t add new functionality doesn’t seem to be minimal, so polishing an app’s appearance should be considered waste. But an ugly product is obviously unmarketable, so some amount of polish is important. With a kickstarter campaign, it’s clear that including a video dramatically improves conversion and is worth the effort. But it’s also obvious that spending 6 months of full time effort to craft the perfect landing page isn’t worthwhile. Preparing a campaign naturally forces you to focus on marketing instead of development, which is much more valuable in a startup’s initial stage.

The emphasis on collecting payment avoids another potential mistake. A landing page that receives 100,000 unique visitors in a month is impressive, but is ultimately a failure unless some of those 100,000 are willing to pay. Crowdfunding doesn’t force you to monetize your actual product immediately, but it does ensure that at least some of your customers are willing to spend money.

The time limit dramatically reduces your risks as a entrepreneur. Specifically, it limits your opportunity risk. If you launch a crowdfunding campaign and fail to meet your goal in the time alloted, then you have a pretty clear signal to move on and try something else. Successfully funding a campaign doesn’t guarantee that your target market is large enough to support a company, but it at least confirms the market exists.

What to offer in your campaign

Not only does a crowdfunding campaign replace building a prototype, it lets you consider prototype ideas that wouldn’t be commercially viable in their own right. For example, freemium apps are increasingly becoming the dominant model for mobile apps (freemium apps generate 69 percent of the worldwide iOS app revenue and 75 percent of global Android app revenues). I’d be hesitant to develop a free app without funding, but would be very happy to build a free app to meet a crowdfunding obligation. I could then expand that free app with premium features later, hopefully turning a small project into a sustainable business. Even if those premium features fail, at least I’d have the crowdfunding proceeds as compensation for my time. Alternatively, if you already had a moderately popular app, you could use crowdfunding to fund new features, thus ensuring that you only build new features that people actually want.

Nor is there anything wrong with using crowdfunding to raise capital for actual commercial products. Pebble is selling a watch for $150 after raising more than 10 million dollars on Kickstarter. Although part of me feels that collecting donations is more appropriate for artistic or free projects, the truth is that commercial projects are completely ethical. If your description is honest and backers contribute voluntarily, then you have no moral obligation to provide anything other than what you explicitly promise.

Defining your goal

Unlike a traditional crowdfunding campaign, the fundamental goal of a crowdfunding MVP isn’t to raise capital. Money is certainly appealing, but the real value of a successful campaign is knowing that other people value your concept and are willing to pay for it. Justin Wilcox described in his proposal how you can A/B test multiple campaigns to test different price points. Although I think that’s a great addition (especially considering how his test found a higher conversion rate for the higher price point), I don’t think determining how much someone will pay is as important at the initial stage as determining whether anyone will pay. I think there would be more value in A/B testing different marketing pitches than different prices.

I do agree with his advice to define your goal in terms of the number of backers (5,000 in his case) instead of some dollar value. Finding backers for your concept means that other people have and recognize the problem you are trying to solve. Knowing that, you can market more ambitious products or services to this base in the future (such as the premium features mentioned above).

Kickstarter and its alternatives

I’ve been using the terms “crowdfunding” and “Kickstarter” interchangeably, which seems fair given the company’s success in popularizing the concept. But there are certainly other alternatives, such as IndieGoGo and Selfstarter, which might be more appropriate. Kickstarter’s popularity provides immediate visibility for your project and their reputation adds legitimacy, but they do set restrictions that might preclude your project. In particular, their guidelines state: “Design and Technology projects that are developing new hardware or products must show on their project pages a functional prototype”. I’m unsure how often this rule is enforced, but it certainly seems to rule out our ideal scenario of launching a campaign instead of building a prototype. Depending on the complexity of your concept, you might be able to build a very simple proof-of-concept and then ask for funding to complete the feature set. Kickstarter also forbids concepts that require on-going maintenance, which eliminates most Software-as-a-Service concepts as well.

I don’t believe IndieGoGo has any similar restrictions. Selfstarter (as a DIY option) is certainly available for any project.

A place to start

I think this concept is particularly compelling because it offers non-technical founders a concrete place to start. Convincing a developer to build your prototype is difficult, especially if you only have a vague idea and no hard skills to offer. Creating a crowdfunding campaign requires you to clearly articulate your vision and to pitch your concept in an exciting way, which is exactly what you’ll need to do in order to recruit. A small amount of crowdfunding capital demonstrates significant traction, even more so than a lengthy waiting list on LaunchRock. More importantly, even a failed campaign isn’t a waste of time. The skills you need to successfully complete a crowdfunding campaign are easily transferable to marketing an early stage startup, so it’s certainly worthwhile nurturing those skills early.

• Who are your potential readers?
• How will those readers find your blog?
• Will those readers subscribe, share, comment, or otherwise engage with your content?
• What do you expect from a “successful” blog?

If these questions sound familiar, that’s because they are fundamental to the first two steps in Steve Blank’s The Four Steps to the Epiphany, namely “Customer Discovery” and “Customer Validation”. Creating and marketing a blog is a great way to practice the customer development skills necessary for an early startup, without having to worry about the parallel challenge of product development.

Customer Discovery

Discovery is a fairly obvious requirement for a new blog. I don’t believe many folks start a blog without at least intending to find readers. Fortunately, the process of finding readers for a blog is far more straightforward than discovering potential customers for a startup. There’s already a large market of people who read other blogs, you just need to convince them that yours is at least worth a cursory skim. And there’s already an established base of communities like hacker news and reddit that exist primarily for readers to discover, share, and discuss new content. Marketing your blog is almost entirely a problem of execution: you don’t need a clever new strategy, you just need to get out there and hustle. This is far easier than discovering customers for a new product category.

More importantly, by accepting that building a readership just requires some honest labor, you can avoid a common pitfall. Smart people (particularly developers) enjoy thinking about ways to do their jobs more efficiently. When building a startup, it’s tempting to spend so much time thinking about ways to reach new customers that you never actually get around to speaking to any. This is obviously absurd in the context of building a blog’s readership. Working on developing your blog’s reach is a great way to develop a habit of doing instead of just planning.

Customer Validation

Validation requires a little more thought. For a commercial startup, validation usually revolves around assessing willingness to pay. A blog doesn’t have to become a revenue stream to be successful, but it is important to identify some possible benefits, consider how feasible they are, and figure out what sort of readers you want. It’s not necessary for your readers to pay you directly, but it is important to identify some sort of desired action you want to encourage.

In particular, some plausible motivations for blogging are to:

• Articulate and develop your own thoughts: Thinking is easy; writing is hard. Ideas are tough to nail down. Something that sounds wonderful bouncing around in your head can look outright preposterous once it finds its way to paper. Writing things down and sharing them is a straightforward way to develop intellectual honesty. You can get some benefits from writing even without a readership, but ideally, you’ll want to find thoughtful readers who are interested in discussing your ideas and challenging your assumptions. If you happen to build up a large pool of readers who never comment or engage with you, then you haven’t found the right readers and should re-consider your discovery efforts.
• Establish some bona fides. Perhaps you’ve been reading this blog and thought to yourself, “That chap seems to know what’s up.” Validation for this goal can be as simple as collecting points. Unlike the above, silent readers are okay for this goal, provided that they’re willing to up vote your content on social media sites.
• Direct some traffic to another project, such as my current startup. This is starting to get more difficult, but is still pretty reasonable. Readers of this blog probably like startups, I’ve got one, so perhaps you’d like to take a look at mine? This thought process quickly breaks down if there isn’t a straightforward connection between the blog’s content and the project in question. A photo-blog of cat cartoons is odd marketing for an enterprise software company, but might work out if done right. The key metric here is simple: you want to review the conversation funnel for your other project and compare traffic from your blog with other traffic sources.

Practice the hard stuff

Startups are hard. There’s no way around that. You’re unlikely to find immediate success regardless, but you can improve your chances by developing relevant skills. Software developers are often tempted to focus on becoming better developers. Product development is certainly relevant to any startup, but the challenge for most early startups is finding customers. Learning the latest web framework won’t help you validate a business model, but finding readers for a new blog is a great way to learn and practice customer development.