The best architectures, requirements, and designs emerge from self-organizing teams

Although Cloudspace works mostly with startups (which are small almost by definition), this principle is worth commenting on as a general rule. As startups grow in size, this rule becomes increasingly relevant.

The people who make the best decisions are the ones who are closest to the causes and effects of that decision. If you’re a part of a team, working in the thick of it, you’ll make a better decision than someone removed from the situation. You can see this principle applied in our rule that we have engineers estimating work — they’re closest to the code so they make better decisions on how long it will take to change things. This also applies to product changes — community managers can help make informed decisions because they are close to the users who are experiencing pain points.

Believing in this principle implies that if you’re not on the team designing a system or a process, you should avoid imposing rules or demands whenever possible. You have to sell people on your ideas, and convince them that what you’re proposing is good. If you can’t convince them, then you need to consider that you might have been wrong, or change the way you’re explaining it.

An example of this is at Apple. Apple’s OS is composed of a number of components, each built by a separate team. The OS project managers don’t dictate any project-wide processes. From version control to code standards to testing — there are no externally imposed rules on teams. As long as they deliver good results, they can work in any manner they want. If a project manager wants a team to change their process, it’s up to them to convince the teams one at a time. They don’t impose mandates because mandates don’t work.

This principle isn’t to say that teams can’t take any input from the outside — obviously external forces (like the client and users) shape the direction of the project. They provide the broad strokes, and the team fills in the details. Strategy vs Execution.

The point is to focus on the true requirements, not how they are done. It’s helpful to think about things in terms of “I don’t care as long as SIMPLE_OUTCOME_HERE”. For example, one of our requirements is that the process for deploying/redeploying be quickly reproducible. This usually means automated deployments, runbooks, and notes. But, the teams are able to use whatever solution that fits the projects needs. Usually these are the same across many of our projects, but there are exceptions, and self-organizing teams handle them well. In this case, I don’t care as long as code can get pushed to the server quickly and reliably.

This ties back into the issue of trust. Communication makes sure that the teams organize correctly, but there has to be trust, and the confidence of understanding that there are many ways to solve the same problem.

There’s a quote from General Patton that I like. Patton was a man whose ability to get results from a team speaks for itself. He said “Never tell people how to do things. Tell them what to do and they will surprise you with their ingenuity.”

Sometimes you just have to do it live! Sometimes a fix is so crucial that you head directly to the production server, edit the code, save the file, and restart the server. It’s not an ideal situation but it happens. Most of the time, the fix is pretty small and you can just go to your local repository after things are fixed on the live server and remake the changes to your code base and commit them. When the fix is more complex, you may want to commit the code on your production server and create a patch to apply your changes to the code in the repository. If your repository is in git, here are some instructions on how to use ‘git format-patch‘ and ‘git am‘ to bring everything back in sync.

Patching can be useful for more than just hot fixes.  Sometimes you don’t need to work on a live server, you just want to and by “live” I mean on an actual server just like the production box instead of your local machine. This can be very useful for anything from starting a new project to working on v2 of an enormous live system. Rather than mess with a local copy of the code (and trying to get some advanced technology working on your mac) or trying to configure a virtual machine (or several) to let you interact with a duplicate environment of what your production server will be, you can just build the server, deploy to it once and edit in place. You don’t have to worry about differences between when your development and production environments when you are working on an exact copy of the production system. You can just make changes to the code and restart the application to see them. Of course you’ll want to commit your work regularly and make sure to get it into the repository. You can do this by creating patches with git as well.

Can you follow these instructions?

Go to your local repository, make sure you are on your master branch, ‘git pull’ to make sure you have all recent changes and view ‘git log’ to find the ID of your latest commit. Save this value for later. Hopefully all of these commits have already been deployed to production, let’s check. Go to the production server and go into your project’s current folder and view ‘git log’. If your production server has all of the same commits then you are good to go. It is okay if the production server has more commits because you’ve already committed the new code on the production server, as long as they are all after the commits that are in your local repository. If you have extra commits on the master branch of your local repository these instructions will not work exactly as printed, take some time to read up on git am and git format-patch though and you can figure it out. (Try keeping your work off the master branch and only merge your working branches back into the master when you are ready to deploy.)

Patching Instructions

Step 1) Commit Code on Production Box First, go ahead and commit the changes you made to the code on the deploy branch on the production server if you haven’t already. You can commit it all at once, or split it into multiple commits if that makes more sense for the changes you have made. For the sake of this demo we’ll say that you put everything into one commit with the commit message “my fix” which is a horrible commit message but works great for example purposes.

git commit -m "my fix"

Step 2) Create Patches On the production server go into your application’s current directory. Using the ID of the last commit you have in your local repository run the command to create the patches.

git format-patch LAST_LOCAL_COMMIT_ID
(*replacing the LAST_LOCAL_COMMIT_ID with the value saved before
 -- should look like 48aa02be08fd5f2ed641b50f5611b968ff2699a7)

This tells git to create a patch file for each commit that happened since the specified commit. With our one commit, this would create one file named ‘0001-my-fix.patch’

Step 3) Get Patches Now download the patches to your local computer with your protocol of choice (scp, sftp, etc) and put them in the root directory of your local repository.

Step 4) Run Patches Go into your local repository and run the patches

git am -s 0001-my-fix.patch
(*replacing 0001-my-fix.patch with the correct file name)

Path files tend to be named starting with ordered numbers (0001, 0002, etc) so if you made multiple commits on the production server, you may need to run multiple patch files. You can run them each individually or you can run them all at once like so

git am -s 0*
(*assuming no other files in the directory start with a 0 [zero])

Step 5) Push Patched Commits to Remote Branch

Now the commits have been applied to your local repository and you need to push them to the remote.

git push origin master

Ta-da! Now the repository is back in sync with the code actually running on the server.

88% of a sample of Gen Y-ers voiced support for a law that would require websites to delete captured information. Sixty-two percent of them wanted the right to know everything a website knows about them.

From Do You Trust Your Social Network? at HBR

Let’s say you need to test for a value that you store in the database but you can not find on any page in your application. Perhaps you needed to have setup several users and test that one particular user has a valid internal GUID set on it.  You could just select the user into a local variable by a known label to test it.

Then /^the test user should have a valid guid$/ do
  user = User.find_by_login("test_user")
  user.guid.should =~ /\w{8}\-\w{4}\-\w{4}\-\w{4}\-\w{12}/
end

This works pretty well for a time, but it closely couples the test to the actual login used to create the user.  The step can’t be used to test a user with any other login value and you may find the need to change your test user’s login when writing a new test.  It’s never fun to have to rewrite a bunch of your tests to for something minor and you don’t want to clutter your step definitions with tiny variations on the same test. The example step above also has to pull the user object from the database every time. Extra find methods running  all over your test suite can defintely slow it down.

If you assign an instance variable to an object in the “Given” step when you create the object, then you can test that object’s values in a “Then” step later on.

Given /^a user "(.+)" with password "(.+)"$/ do |login, password|
  @test_user = User.create(:login => login
                           , :password => password
                           , :password_confirmation => password)
end

You can test the values of the instance variable in a later step.

Then /^the test user should have a valid guid$/ do
  @test_user.guid.should =~ /\w{8}\-\w{4}\-\w{4}\-\w{4}\-\w{12}/
end

Now you could give your test user any login value that you want.  Smart usage of instance variables can add flexibility to the steps in your test suite and cut down the time it takes your tests to run.

It’s not like people have stopped innovating. It just goes by a different name. From DNA startups, web startups, electric car startups, alternative energy startups — startups are the new R&D. It’s a more focused, market-oriented R&D, but ultimately dollars are being spent to build and discover the next big thing. And with every startup win comes a bunch of new rich startup people who want to funnel money into the next round of ventures — continuing the cycle.

Part of the reason that startups innovate — and big companies don’t — is the idea of sustaining change versus disruptive change (The Innovators Dilemma). Big companies don’t do it well, small companies do — and the smallest company of all is a startup.

So as a car lover, it’s nice to see General Motors (the hippest of all the conglomerates) dropping $100-million into a venture capital fund. R&D is back! GM is too big to fail innovate, so they’re pushing money into a small organization that can get behind relatively smaller wins, prove the new idea/market, and then get scaled up with the financial resources that GM can provide.

Just like the web startup world, where small groups try out new ideas, then get big money to take the idea to the next level. Remember the old adage — “scaling problems get funded”.

(And yes, Edan. I agree — R&D is startups are badass.)

<link rel="next" href="page2.html">

With HTML5, that’s all you’ve got to do to preload a page. Hat tip to Conner over at keyboardy for pointing this out. It’s like cheating on behalf of your users, to get them faster page loads. Of course, this is what “web accelerators” have done for years, so it’s good of the W3C to add this feature to the spec.

He mentions a few example use cases to get the creative juices flowing, and more importantly, points out that this technique can dramatically change your server stats. So, if you’re using a package to analyze web server logs, you’ll need to account for pages being preloaded. Right now only Firefox supports this next/preload feature, and it sends a custom header with the request. I imagine other browser makers will too. Still — use of this feature could mean all old and unmaintained stats packages are useless — aren’t you glad you use open source and can edit the package to account for the new header?

One thing that he didn’t mention is the implications for your server. Let’s say every page has a single “next” page referenced — just one and no more. If you’ve got a bounce rate of 50%, you’ll be serving a lot of pages that no one will ever see. Static sites don’t have to be concerned — serving flat files is a no-brainer. But, if you’ve got a dynamic site, lots of uncached data, and a service that’s routinely around 60-70% capacity on a normal day, you could overload a web or DB server. Put more than one prefetch instruction per page, and the load goes up even more. Ops teams take note — a bad template or front-end decision could easily double your traffic overnight, without the upside of gaining additional users. This potential for crashing means we could be talking about the single most dangerous HTML tag  — worse than <blink> and <marquee>.

Edit: mcotton, a Hacker News commenter pointed out that a nice solution is to configure your server to ignore requests with the preload header if they are above a certain load.

Source - http://awe.sm/53rPt

We have been using versions of Agile Development at Cloudspace for most of our 13 year history — even before it had a formal name. However, we didn’t implement pair programming, or extreme programming (XP) as it is sometimes called, at the company until more recently. One of the driving factors for moving to pair programming was the rapid growth of our business working with tech startups. In order to serve these clients, we needed to grow the team rapidly. As we started discussing options, implementing pair programming came up. Pairing junior developers with more senior developer on the team to mentor them through on the job training sounded like a good idea. In fact, we thought this was so much of a good move that we carved out specific team roles for Jr. Developers and Pair Leads.

Guess what? It didn’t work.

The “Pair Leads” got frustrated trying to bring the junior guys, good developers with Comp-Sci degrees and industry experience, up to speed on our system of development. And the junior guys weren’t learning half as fast as we hoped. In fact, they weren’t learning half as fast as they did with the old non-pair programming system. We tried a few things to improve the process overall, but nothing seemed to work. Unfortunately, compared to computer systems, debugging people issues are very hard. But sometimes, you guys get lucky.

In a conversation last week, Steven Covey’s book The 7 Habits of Highly Effective People came up. I can’t remember how or why this came up, but I’m glad it did. The person I was talking with said that they tried to read the book but gave up because it felt dated. I hadn’t read the book since business school, but could still remember quite a bit of subject matter. I started spouting them off a few of the habits (as best I remembered them):

Sharpen the saw.

Seek first to understand, then to be understood.

Then I said:

In order to be interdependent, you must first be independent.

I said it and I just froze. That was it! We were trying to throw junior guys who were not first independent working in the Cloudspace system into a pairing relationship which requires two highly effective developers to work interdependently. We totally blew it and now I knew why.

When used properly with skilled engineers you can’t beat pair programming for solving hard problems; creating high quality code; and, as Covey would said, making 1 + 1 = 3. But it is absolutely the wrong way to get junior developers trained.

Hindsight being 20/20, I now see the two types of pairing as 180 degrees from each other. When highly effective developers pair program the conversation is around system performance, best practices, tech trends — discussions of craft which often break into passionate arguments about the best way to achieve an objective. When one of the developers is not yet up-to-speed, the discussion is completely different and easy to spot. This type of pairing primarily involves discussion of syntax, improving skills, following processes & procedures, “let me do that for you to save time.” Basically, anything but craft is discussed. Yes, pair programming helps keep the developers focused and catch errors at every skill level. But I no longer believe that this is worth using pairing as a form of training.

So where do we go from here? At Cloudspace we are now re-tooling our training (we have some very cool stuff in the works on this front), ongoing developer development, and career paths. New recruits will go through training and then work independently on simple projects that really shouldn’t need pairs — all while receiving a high level of feedback; especially in the form of code reviews. They will only graduate to pair programming after they have demonstrated a basic level of mastery of their craft. (One member of the team has suggested that we bring developers into a dark room with tiki torches and read their fate from slips of paper written by senior developers Survivor style; but we aren’t there quite yet.)

Overall, we think that Cloudspace will be much better for making the change. And, as always, we wanted to share our thinking and screw-ups as publicly as possible so that everyone else can learn from our mistakes. Hope this one helps.

“Don’t worry about it. Start. Begin,” he replied.

He was right. I had made a classic mistake and in doing so I was ignoring our work processes with my own pet project idea.  I had ‘gone huge’ with the idea from the very beginning. I was planning user accounts, graphs, how to handle variant covers (a recent favorite variant for Green Lantern 51), and complex relationships for keeping a single master library of existing comic issues that would be separate from each user’s collection and reading log. Don’t get me wrong, it is good to be passionate about your idea and to know what challenges may be in your future but do not let that get in the way of getting your project going. Start small. Do the simplest thing possible. Constantly adapt your plans to your application’s needs. Be Agile.

That day I stopped mulling about things like how to order numbers that sometimes had letters on the end (see Batman 700U) and took my project agile. I started by building the simplest thing possible. I got going with no users, just an entry form with blank fields for comic title, issue number and date. Since then I’ve added a pop-up date selector, notes fields, the ability to enter ranges at one time, different collection views and a few other features. It has been almost a month but I’ve only worked on my application for 15-20 hours, at least 10 of them in the last couple of days. The ComicReads application is usable. It rocks! You could clone the ComicReads github repo, rake create & migrate your database, run ‘script/server’, and be ready to log your own collection but I don’t expect you to do so.

The most exciting part is that I have stopped logging my comic reading on notecards this past weekend and am now exclusively using my new application on my virtual machine. I haven’t added users yet, switched to mysql or solved several sticky issues— but I’m on my way. By starting with the features I needed— the ability to log the simplest and most essential parts of my comic book reading— I’ve made my project more exciting. Now that I have a working version, I find myself trying to work on it more and more. One day I will add user support, grab a URL, get a better design, and throw the system up for public use. Not this week though, I’ve got too much else to do.

Go ahead and get started. Begin! Throw out the users, forget connecting your application to twitter or facebook accounts, don’t worry about which database you are going to need, just go for it. Start by solving the problems that make your application interesting.

Like MBL, we use external javascript that is referenced from the blog. If our site ever goes down, even for 2 minutes, browsers try to load the javascript and pause for 30 seconds. Publishers then get upset that we are blocking their blog and take us off. How did you get around this with MBL? How did you display real-time data (when user visits their picture immediately shows up), but decouple from the database (which I assume was necessary)?

Since we have been asked this question more times than I can count, I figured it was time to post a quick response on the blog so anyone who is interested could benefit…

Keeping the javascript from slowing sites down was absolutely the biggest technical challenge on MyBlogLog.  Once we sold MyBlogLog to Yahoo! we designed a massive, redundant architecture across multiple colos and countries to solve the performance issues.  That said, there are a number of things that anyone can do on the cheap to improve performance.  Off the top of my head, you could do any/all of the following:

1. Move to an ajax/asynchronous load of the javascript widget.
2. Move the main javascript to a CDN where you shouldn’t have slowdowns.
3. Have the CDN javascript call your servers to populate data, but gracefully timeout if it doesn’t get a fast enough response.
4. Move all member data into Memcached or similar memory caching layer on your servers.
5. Make sure all javascript tags are set to defer load — though browsers don’t always implement this correctly.
6. Use AmazonAWS or similar to setup multiple colos near users on the cheap.
7. Move the tracking script lower in the page so it doesn’t slow things down.

Finally, and most importantly, the system doesn’t need to *be* real-time.  It just needs to *act* like it is real-time. We had a line in the javascript on MyBlogLog that automatically showed the current user visiting the site at the top of the list.  No database calls were needed.  Even better, since the user didn’t have a clue who had visited the site just before them, we could cache the list of recent visitors every minute — longer for sites with low traffic — and show it below the current user.  This had a significant improvement in the end-user experience while costing us nothing.