I think that Roozbeh has a bit too much time on his hands…

The most efficient and effective method of
conveying information to and within a development
team is face-to-face conversation.

I’ve previously mentioned we have an IRC channel. That’s due to practical considerations. When we need to frequently and quickly share information among a group of people who deal with each other often throughout the day (and are generally on the same page to begin with), IRC will work. Notice how many qualifiers I put on that sentence

It would be unsuitable for multiple teams to physically interrupt one another’s thoughts numerous times per day. IRC has another advantage because it is a “pull” channel. When I finish a feature or need a mental break, I can switch to IRC for a moment, and address any questions. When we need to have an extended talk, we walk over to that person’s room or jump on Skype/iChat video. Nothing beats seeing a face. Babies are known to stare at faces within days of birth, and our brains are hardwires to see faces and extract information from them constantly. Look at how well-known and recognized the “face on Mars” has become.

When we’re talking client-engineer communications, the whole goal is to get people who have different mental states to be thinking the same way. Engineers need to understand the product and what drives business value, and the product owner needs to understand the technical requirements to make the system work. To do that requires more bandwidth than short text messages. This is another facet of businesspeople working with engineers on a daily basis. You need frequent, high-bandwidth communication.

At the start of a new project, I always prefer an in-person meeting. There’s nothing that beats shaking someone’s hand, getting a meal with them, and learning about their project. It gets a great sense of who they are, where they’re coming from, and it helps build the trust that is mentioned in Principle #5 (”A Team Of Motivated Individuals”). Once we get going, video chat usually replaces in-person meetings, since our clients are spread across the country. It’s not reasonable to physically travel to weekly planning meetings, and we’ve adjusted our process to accommodate this need.

That’s all there is to this principle. Communication from businesspeople to a team, and among team members is vital, and the best way to communicate is face-to-face. If you’d like to discuss it more, I’d prefer to go grab a coffee or lunch with you, but I’ll settle for a comment on this blog post. Next post, I’ll be talking about progress in a project, and how it gets measured.

If there was a loop, it would be a small one (less than 5 records total). Our code that read the loop would run forever if it were to encounter a loop (and eventually overflow the stack). Since it would be a very small loop, and it was not reasonably possible to prevent a loop from happening, we had to handle it in the reader. Our reader was recursive, so we passed in an array to the recursive call, and it would constantly add the ID being looked at. At each recursive step, we’d check if we had already seen the ID we were about to look at, and if we found a loop, we exited accordingly.

def reader(record, previous_ids = [])
  raise "loop" if previous_ids.include?(record.id)
  ...
  return reader(next_record, previous_ids + [record.id])
end

We wanted to write a test to ensure that our code would never follow a loop. The only way to do this was to write a test that created a loop of records, and run the function on the loop. If the loop-blocking code worked, it exited instantly. But if the code were to not work, our test would run for a very long time before we overflow the stack. This is essentially testing the halting problem, a famous computer science problem with no real solution, and no chance of ever being solved.

The most straightforward solution available was this:

it "doesn't loop forever" do
  # setup test and create loop
  puts "This is a test of the halting problem. If you see this message for more than a few seconds, it means the test has failed."
  # run test on the loop
end

If you want to put this in a CI system, a more heavy-duty way of doing this would be to start two processes: one that runs the possible infinite loop, and one that raises an exception after N seconds if the first process hasn’t completed.

Filed under: hey it works

In your main deploy.rb, add an

after "deploy:symlink", :update_config_file
after "deploy:symlink", :update_database_yml

And just throw as many of these in as you need. This will call these tasks no matter which stage you are deploying to (and as an added benefit, if you forget to define a method for a stage, it will error and alert you). Next step is to define an “update_config_file” or “update_database_yml” file for each stage. In your /config/deploy/ path, you’ll have a file for each stage that you deploy to. Go in there and add something like this:

PRODUCTION_DB_INTERNAL_IP = "1.2.3.4"

desc "Update database.yml with production information"
task :update_database_yml do
  replacements = { "DATABASE_IP" => PRODUCTION_DB_INTERNAL_IP }

  replacements.each_pair do |pattern, sub|
    run "sed -i 's/#{pattern}/#{sub}/' /path/to/deploy/current/config/database.yml"
  end
end

Just define as many of these tasks as you need, and change your config files to contain whatever values you need them to. Clean and simple, happens 100% automatically with each deploy.

Build projects around motivated individuals.
Give them the environment and support they need,
and trust them to get the job done.

This Agile principle is such good business advice, I could approach it from many other non-Agile directions. It almost transcends business and applies to all human endeavors.

Starting with the motivation issue. Larry Bossidy, one of the world’s top CEOs, says that when hiring, a person with absolute determination to succeed will always do better than someone with a high IQ and elite education who is cruising.

Approaching this from a different direction, I recently read a football coaching book, and one of NCAA coaches who contributed to the book said that given the choice between a player with solid technique and a player who is 100% motivated, he will pick the motivated player 9 out of 10 times. This advice is coming from a top performer in an industry that is incredibly well-scrutinized with clear outcomes and performance metrics. This gives me good reason to trust it.

So, speaking of trust, let’s move to that. I think that motivation can be universally agreed on as valuable, and if you don’t believe me, you’re probably not motivated enough to post a comment and debate it.

Trust is tricky. A favorite Warren Buffet quote of mine is that “it takes 20 years to build a reputation and five minutes to ruin it.” How can you quickly decide if a team is trustworthy? Of course we’ve all got markers for trust, but I think that a track record of similar and successful projects is a good place to start developing trust. If you can get access to the person who filled the “product guy” or client roles, ask them how they felt the team handled it. Did they feel that the team made intelligent decisions, but kept the client in the loop? Most importantly, did the team provide good value for the dollar spent?

Another thing that can help identify trustworthy people is talking directly to them. Are their answers clear and specific, or vague and filled with jargon? Do they make an effort to understand the client’s business and perspective, or do they insist on a particular solution no matter what the context?

Since trust is something that naturally has to build over time, don’t wait for 100% trust. If you’re always waiting for the perfect conditions, you’ll never get anything done. Jump in and give them a shot. This is one place where Cloudspace takes advantage of fast delivery and iterative development. Our process is based off single weeks of work. Of course, we’d love to work with you for several months, but if you have 10 weeks of work, start off for a week or two and see if you’re happy. Our clients are free to leave at any point, so this makes for an environment where we consistently perform and deliver. It also makes for an incredibly simple contract.

The last parts of this principle relate to the environment and support. I think this primarily involves internal teams, although a smooth client relationship is always helped when the client trusts the technology decisions and doesn’t second guess how things are constructed. There’s a balance between understanding the infrastructure and guiding the project, versus micromanaging and dictating. If you were going for surgery, you wouldn’t tell the doctor to use a #5 scalpel because you read an article on WebMD. You’d ask the doctor what is going on, expect clear and intelligent answers, then give the doctor space to work.

With respect to internal teams: Cloudspace helps create a healthy environment for our engineers by treating them both as engineers whose skills need to be kept current, and as human beings who have higher needs (think Maslow’s hierarchy). Because we treat our people well, our clients get great results from them.

To keep technical skills sharp and communication healthy, we have daily standups that last about 15 minutes where everyone gets a turn to share what they did since yesterday’s standup, and anything new they learned. We have weekly “hack sessions” where we review a new technology, or spend time cross-training (teaching things that are immediately relevant to our projects). We constantly talk to people and make sure they are happy with the progress of their project, and to identify things that may limit progress. We’re all in a private IRC channel so that we can instantly tap collective team knowledge, or jump to a public channel and tap the knowledge of the community.

Also, good milk comes from happy cows (I know this; I saw them gazing out at the ocean at Point Reyes a few weeks ago in California before eating some of the amazing cheese their milk produced at Cowgirl Creamery). We have team outings where we’ll go out for beers, bowling, laser tag, go-karting, or a movie. We have a comfortable dress code — sandals and shorts are common in our Florida summers — but we still respect our clients enough to put on a collar to greet them. Since we pair program our Agile projects and have to communicate, we don’t wear headphones, but it’s typical to trade off on choosing music to put on the speakers (I personally love Pandora, Grooveshark, and The Hype Machine).

All of these things add up to a lean and effective team that has what it needs to execute client strategy and deliver great code. We trust our engineers to do a great job — we don’t micromanage them. Our turnover is virtually zero and we’re proud of it. That’s the value of a good environment and healthy support.

Business people and developers must work
together daily throughout the project.

In the post on agile principle number two, I gave the example of a Secret Santa management system, as an example of how being able to accommodate changing requirements can benefit a project. As a refresher, here’s the key story points of the project:

• A user should be able to sign up
• A user should be able to log in
• A user should be able to get a recipient assigned
• A user should be able to add notes to their recipient
• There should be an “Add Amazon product” link that searches on Amazon
• There should be a “Buy on Amazon” button

I’m going to reuse this example to show the value of business people working closely with developers. You can refer to the original blog post if you like. I’m going to be counting the number of product owner-engineering interactions in the Agile scenario:

1. Meeting with the dev team to decide that sign up and login functionality are essential
2. Being notified of the sign up and login completion/deployment
3. Verifying sign up and login
4. Being notified of the recipient selection completion/deployment
5. Verifying recipient selection
6. Being notified of the notes completion/deployment
7. Verifying notes
8. Having developers stop work

That’s 8 different product manager-engineer interactions, and this is a tiny hypothetical product — we only did 4 story points! This doesn’t include questions or clarifications that inevitably come up in all projects while the developers are at work. You show me a client that thinks it’s okay for each of these interactions to wait for the weekly (or longer) review and I’ll show you a client who is overpaying for bored developers.

So what if you’re concerned that these interactions suck up time? Given that other than planning meetings (which occur in waterfall too) the interactions generally last 30 seconds for a quick question or notification, or 5-15 minutes for verification. I’ve never met a client who was too busy for this. Also these frequent interactions give you frequent input. That lets you guide your project — think about driving a car that you can only steer every 30 seconds. It’s worth it.

This is actually one area where Cloudspace adds it’s own flavor to Agile. Pure Agile processes mean an onsite customer, working as a product manager full time. We realize that when working with startups, they don’t always have the time or resources to devote a full person to being available immediately. So we give our clients flexibility. We find out why the software is valuable and what the business goals are so that we can make good decisions if we reach a point where the product manager is unavailable but we need to move forward. Of course, sometimes the good decision is to hold off on one ticket and be productive on another. Clients are always consulted on key decisions. We’re also good at batching these interactions into the daily meetings — a 15 minute meeting or email where we update the client on the day.

From the developer side of things, having easy access to answers is great. There’s nothing worse than trying to guess how an entire feature should work, and then having to rewrite the whole feature due to a difference in perspectives. Remember: happy engineers are productive engineers.

Speaking of happy engineers and productive engineers, in my next post we’ll be talking about the 5th principle of Agile: trusting a team of motivated engineers.

P.S. If you are one of those rocking motivated engineers, Cloudspace is hiring!

Tammy Tappan, PMP
Systems Administrator,
United for Families

“Tim, this is simple. Why are you writing about this?”

select count(*) from messages;

And it is simple. If this were MyISAM. See, MyISAM always stores the number of rows on the table header. So, whenever we ask “how many rows are there?”, it can just grab the count and return it. Not InnoDB.

In InnoDB (for internal reasons), the number of rows has to be counted. Every single time. One of the tables we were counting was over 1.2 million rows. On a small EC2 instance with no other queries/major processes, this takes 1 minute and 20 seconds. This is unacceptable.

We looked at another command that will estimate the number of rows, but according to MySQL, it can be off by up to 50%

show table status;

Being off by up to 50% is unacceptable, even if it is fast. So we kept looking. Most of the recommendations were “make a table that stores the count, and add to/subtract from it every time you do a write/delete from the actual table”. Also not helpful. We’ve got over a million rows, and this is a prototype that’s only been online for a few weeks. I refuse to double the number of writes to our DB.

John points out that since our table is so huge, we’ve already got indices on it. Why not make the query use one of them?

mysql> select count(*) from messages use index (index_messages_on_remote_created_at);
+----------+
| count(*) |
+----------+
|  1276831 |
+----------+
1 row in set (3.19 sec)

Win. The trick is to hint it at a specific index. So, if you’re getting poor response times from InnoDB count, it means you’ve got lots of rows that have to be counted one at a time. And since you’ve got a lot of rows, you have at least one index. Just pick an index that will never contain a NULL value, and tell MySQL to ‘use index (index_name)’. Also, thanks to Wallace, who commented on the MySQL Performance Blog