Highways Agency

The same forces are at play within your team. Everyone is doing his best to optimize their work. When optimizing, you take all the information you have about the current situation and try to make the best decision based on that information. The more information you have, the better your decisions.
For maximizing information flow, it’s a good idea to make your team sit together. Sitting together fosters passive information gathering. This passive information consists of both the colleagues’ moods and a feeling for how things are progressing. Having information about moods and progress creates situation awareness. Situation awareness enables you to make well informed decisions.

Unfortunately, situation awareness is very fragile. As soon as people are physically separated (i.e. by continents, cities, or even floors and walls), they begin losing those passive information sources. Suddenly, they must actively search for information. But you can only search for information you already know exists – in short, you miss everything you’re not aware of.

Even the information you know about is hard to actively collect if you’re separated from the team. You need to decide to do it, you need to activate whatever communication channel you’re using (phone, chat, email, walking over, …) which consumes energy. If you’re already short for time, you might not be able to do this. Which means you lose even the information you know is out there waiting to be consumed.

Walls kill situation awareness and this leads to severe loss of information for the entire project team.

Situation awareness enables spontaneous collaboration.

• If you spot a problem, you can interrupt (stop the line) and deal with it
• If a team mate needs help, she can just ask for it (no barriers like chat, email, or even getting up from the desk)
• If a discussion nearby sparks your interest because you think you can learn something or add value, you just open your mouth

But you’ll miss out on these opportunities if your team setup does not foster situation awareness. Make sure you’re able to spot the trouble of the mother getting off that train and that nothing prevents you from jumping in and lending a hand.

Other posts:

• Stop. Reflect. Adapt. The 3 Steps to Stop Writing Bad Code
• Stop Being So Busy Putting Out Fires That There’s No Time For Fire Prevention!

Greg Walters

AWS and unfuddle

Setup a t1.micro instance with aws.amazon.com for your blog. Remember, if you don’t have an AWS account yet, this micro instance is free for 12 months! We get ~20KPIs per month so the micro instance is just enough to cover this (with significant tuning).

Sign up with unfuddle.com for 200MB worth of free, private git repositories.

Install & configure puppet standalone

$ apt-get install puppetmaster puppet

$ vi /etc/hosts
127.0.0.1 master.successfulengineering.com
...

$ vi /etc/puppet/puppet.conf
...
[agent]
server = master.successfulengineering.com

Setup your git repo

unfuddle has a nice help document for this, but the gist is basically to add the base puppet installation directory on your instance to unfuddle’s git repo:

$ cd /etc/puppet
$ git remote add unfuddle git@successfulengineering.unfuddle.com:successfulengineering/puppetmaster.git
$ git config remote.unfuddle.push refs/heads/master:refs/heads/master
$ git add *
$ git commit -am "initial commit"
$ git push unfuddle master

Now, you can clone this to your local dev env and start creating manifests! After you push your local changes to git, do a git pull on the remote instance. To apply manifest updates while building up your site, use the --test option (otherwise puppet will start in daemon mode).

$ puppet agent --test

Congratulations! Now you’ve got a running puppet standalone environment backed with a git repository. Time for the real work.

Base environment (L)

Here’s what our nodes.pp file looks like:

node default {
  include setenv
  include ntp
  include users
 
  include mysql
  include apache
  include php
 
  include blogs
}

The setenv module gets my default packages installed (htop, unzip, wget, git-core, vim, fail2ban), sets up vi as the default editor, and installs a default locale at /etc/default/locale as:

LANG="en_US.UTF-8"
LC_ALL="en_US.UTF8"

NTP should be obvious, and the users module ensures Matthias and my users’ exist with the appropriate SSH keys and admin group roles.

Base applications (AMP)

Now that our base Linux environment is setup, let’s get to the ‘AMP’ section.
We setup our base MySQL server with the following manifest:

class mysql {
  package { "mysql-server":
    ensure => present,
  }
  service { "mysql":
    ensure => running,
    enable => true,
    hasstatus => true,
    require => Package["mysql-server"], 
  }
  file { "/etc/mysql/my.cnf":
    ensure => present,
    content => template("mysql/my.cnf.erb"),
    notify => Service["mysql"],
    require => Package["mysql-server"],
  }
  exec { "set mysql root password":
    path => "/usr/bin",
    unless => "mysql -uroot -p${root_mysql_password}",
    command => "mysqladmin -u root password ${root_mysql_password}",
    require => Service['mysql'],
  }
}

The $root_mysql_password variable is declared in our site.pp file.

Apache is a bit more complex:

class apache {
  include apache::install
  include apache::conf
  include apache::sites
  include apache::mods
}

For sites and mods, I define a custom method to allow us to easily enable/disable new sites and required modules. Here’s a snippet from the apache::mods class:

  define mods_stats ( $ensure = 'present') {
    case $ensure {
      'present' : {
        exec { "/usr/sbin/a2enmod $name":
          unless => "/bin/sh -c '[ -L /etc/apache2/mods-enabled/$name.load ] \\
            && [ /etc/apache2/mods-enabled/${name}.load -ef /etc/apache2/mods-available/${name}.load ]'",
            notify => Exec["force-reload-apache"],
 	    require => Package["apache"],
        }
      }
      'absent': {
        exec { "/usr/sbin/a2dismod $name":
          onlyif => "/bin/sh -c '[ -L /etc/apache2/mods-enabled/${name}.load ] \\
            && [ /etc/apache2/mods-enabled/${name}.load -ef /etc/apache2/mods-available/${name}.load ]'",
            notify => Exec["force-reload-apache"],
            require => Package["apache"],
        }
      }
      default: { err ( "Unknown ensure value: '$ensure'" ) }
    }
  }
  apache::mods::mods_stats { 
    "rewrite" : ensure => present, 
    notify => Service["apache"], 
  }
  apache::mods::mods_stats { 
    "deflate" : ensure => present, 
    notify => Service["apache"], 
  }

Although the definition of mods_stats may seem a bit lengthy, it pays itself back pretty quickly when you can easily enable/disable any mods throughout your manifests.

The PHP module is more of the same, using puppet templates to control the memory allocation for apc.ini and php.ini. Email me for more details.

WordPress sites

So we have our default LAMP stack setup and initialized with our passwords, modules, and memory constraints. Let’s get down to business – namely, the blogs module.

class blogs {
  include blogs::agileweboperations
  include blogs::successfulengineering
}

Let’s have a look at this blog’s manifest:

class blogs::agileweboperations {
  # Apache website
  file { "/etc/apache2/sites-available/agileweboperations.com":
      ensure => present,
      owner   => root,
      group   => root,
      mode    => 644,
      source => "puppet:///modules/blogs/etc/apache2/sites-available/agileweboperations.com.conf",
      require => Package["apache"],
      notify => Service["apache"],
  }
  apache::sites::site { "agileweboperations.com" : ensure => present }  
 
  exec { "setup awo db":
    path => "/usr/bin",
    unless => "mysql -uroot -p${root_mysql_password} ${awo_mysql_db}",
    command => "mysql -uroot -p${root_mysql_password} -e \"\
CREATE DATABASE ${awo_mysql_db} DEFAULT CHARACTER SET UTF8;\"",
    require => Exec["set mysql root password"],
  }  
 
  exec { "setup awo user":
    path => "/usr/bin",
    unless => "mysql -u${awo_mysql_user} -p${awo_mysql_pass}",
    command => "mysql -uroot -p${root_mysql_password} -e \"\
CREATE USER '${awo_mysql_user}'@'localhost' IDENTIFIED BY '${awo_mysql_pass}'; \
GRANT ALL PRIVILEGES ON ${awo_mysql_db}.* TO '${awo_mysql_user}'@'localhost' WITH GRANT OPTION;\"",
    require => Exec["setup awo db"],
  }
}

This puts all the pieces of the puzzle together: our custom apache configuration, enabling the site with an easy one-liner, and bootstrapping the WordPress database and user (only if they don’t exist).

We don’t do a WordPress install per-se. We’ve been running this site for almost four years, and we have it in a git repository (conveniently hosted next to the puppet repo with unfuddle). All the migrations we’ve performed in the past have shown us the “famous 5-minute installation” can’t beat the ease of a git clone.

Crafting and finding the above manifests on the Internet took me about 12 hours total work effort, but I have a few years of puppet experience under my belt as well. Hopefully, this guide gives you some ideas and pointers about how to get control of your WordPress environment with puppet. Let me know if you have any questions!

Other posts:

• Configuration Management: Introduction to Puppet
• Puppet vs. Capistrano – a short comparison
• Migrate Your WordPress Blog to a Bitnami EC2 Instance

Is your growth being stifled by your current company? Time to find a place where you can assume new responsibilities and pickup new technical skills.

If you have the feeling that you cannot grow in your current job, it’s time to move on. Staying put not only means getting sidelined and frustrated, but even worse, your technical skills will become crusty and outdated. Put the fun back into working and learning in an environment which challenges you and values your contribution.

The Environment

At gutefrage.net, you’ll find a challenging and open environment which puts you back in the driver’s seat of your own growth. You’ll build and run web applications scaling to millions of users while earning real money!

The Tools

We at gutefrage.net believe in the value of experience, and know that a seasoned Web Application Engineer can thrive in any language or web framework. We’re currently looking for support in growing our Ruby On Rails based portals. If you’re fluent in any modern web application framework (like Zend, Django, or even Ruby On Rails), Test Driven Development is second nature, and you know your way around a Linux box (maybe even using configuration management tools, preferably Chef), we should talk.

The Location

gutefrage.net is located in the center of Munich, Germany and belongs to the well established Georg von Holtzbrinck publishing group, which is active in more than 80 countries.

Contact Us!

If you want to help expand our well established and fast growing web sites www.pointoo.de or www.helpster.de, please contact us right away. Either send me a DM @mmarschall or use the offical route by mailing to jobs@gutefrage.net. Of course, we’ll keep your inquiry confidential – no one should get hurt.

I’m looking forward to hearing from you!

See, what we build:
www.pointoo.de
www.helpster.de

Find out what your future colleagues are writing about:
Blogs: www.agileweboperations.com and www.nistu.de
Follow us on Twitter: @mmarschall, @nistude

Other posts:

• How to Deploy Ruby on Rails With The Opscode Chef Application Cookbook
• Aegis: Role-based Permissions for your Ruby on Rails application
• Setting up a test database on a ruby on rails continuous integration server using SQL instead of schema.rb

For nearly four years Dan and I have shared our experiences and ideas about agile development and DevOps.

We would like to ask you, our dear readers, how we could help you to become even more agile and have more fun doing your job.

Please help us to understand your needs better by taking our short survey at: http://www.surveymonkey.com/s/MKYQPL5.

We’d love to hear from you!

Dan & Matthias

Other posts:

• About Agile Web Development & Operations
• Agile Web Operations: What do YOU want it to be?
• 1st Birthday: Best of Agile Web Operations

That tiny, little detail is a collection of inefficiencies you’ve introduced over time. On a dedicated, big iron server, it doesn’t really hurt if a database query takes 0.2 sec. And you didn’t care that the very same query took 2s on your local development box, because you knew it would be “fast enough” (™) on the live site. Well, you’re right…right up to the point where your application has to scale, when you’ve got enough users on the system that even your “big iron” begins to cough and sputter.

Scale-up is not an option if you’re already using top-of-the line hardware

If you’re already running the biggest server available, it’s really hard to scale that box up. So, your remaining options are:

• Scale-out to other servers
• Optimize your code

Both options are tedious – especially if your site is under heavy fire. It would be so nice to just upgrade to a bigger server and buy yourself a couple of weeks to optimize your app and prepare for scaling.

Avoiding over-capacity

One way to avoid crashing into such a road-block is to run your application on smaller hardware from day one. Even if you’ve got big iron available, it’s a good idea to put your web app inside a virtual machine, limiting its resources. By doing this, you’ll get early warnings of scalability issues. If you write a query which takes too long, you’ll notice on the same day. If your application gets a load spike, you have some breathing room to scale it up while you’re preparing the scale-out architecture. Working with constrained resources is a good thing because it forces you to do a better job right from the start!

No related posts.

drewgstephens

The daily routine of the morning stand-up is so ingrained, I go through the above liturgy without conscious thought. For me, the stand-up provides a focused center for the team, our morning huddle. We look each other in the eyes, see how we’re feeling – we hear in each others’ voices strong commitment or uncertain hesitation. We lean on each other for support and promise ourselves that at the end of the day we will be one step closer to the goal.

I recently spiced up our daily by introducing a medicine ball as a speaking token. This token does several things:

• while holding it, you go through the 3 responses quicker (the thing weighs over 12 lbs!)
• you know how heavy that ball is, so you let your fellow devs’ report their status w/o interrupting
• it’s a ball – it’s fun!

I got the idea after reading an excellent Wall Street Journal article on the subject “More Meetings Are Stand-Up Jobs”. Check it out – you’ll find several other ideas on how to break up the monotony of your daily.

Remember that the daily stand-up isn’t just another meeting. It’s your team huddle. It’s your commitment to getting a step closer to your team’s goal.

No related posts.

jurvetson

To be able to self-organize and perform optimally, a few basics need to be in place. With the proper foundation, the whole team can organize themselves and perform better.

Clear Direction

The most fundamental requirement for a team to be able to self-organize is a clear direction. Only if everyone within the team knows exactly what the overall goals are can they make the right trade-off decisions. Typically, the managers know much more about the strategy of the company and the current goals than any team member. In a command and control pattern this is assumed to be OK as all decisions are made by the manager. But, in a self-organized team, the team members are supposed to take decisions themselves. They can only make good decisions if they know enough about the current goals as well as the underlying strategy. Management has to make sure that the team is in the know. This requires trust and openness, two things which do not come easily to some managers.

Visual Management

The second success factor for self-organized teams is visual management. If every individual team member is supposed to take good decisions all the time, they need to know more than just the goals and strategy. They need to know the current situation as well. Setup lots of visual controls for the current status: task boards, build indicators, performance graphs, … everything possible. It’s a must that everyone can always see the real-time status.

Ownership

You need a team of individuals who care. That’s very simple. If your people don’t care about their work, you’re in trouble anyways. But, if you let these same people take over critical decision making, your troubles will multiply. If you give control to your team, they need to be able and willing to take up ownership. Initially, this is pretty hard; almost all problems seem to be out of the team’s hands and they raise them as impediments. But, over time, they learn to take ownership for more and more of these problems. This speeds things up dramatically and makes the team even more successful.

Moving from micro-managed to self-organizing teams will lead to both higher motivation and performance. In order to succeed, you must set a clear direction, make the current status visible, and build ownership directly into your team.

What are your experiences in making teams self-organized? Let us know in the comments below!

Other posts:

• Empower Your Team – You Won’t Regret It
• How Digital Agile Management Tools Make You Blind (And How A Physical Kanban Board Can Help You See Again)
• A Luxury Problem: How Emerging Iterations Eat Team Commitment For Breakfast

After DevOps Borat and BroOps it’s time for The DevOps Dudes!

Click through to see it full size

Other posts:

• DevOps is NOT a Job Description
• DevOps: Why Silos Suck And How To Break Them
• Do you have a DevOps Culture?

This is a guest post by Prasad Chaudhari, freelance java consultant. He was appointed as a project manager for the project mentioned below and played a role of ScrumMaster.

mattwilde66

Locally, we have six scrum teams: five software development teams, and one IT operations team. Each team has a sprint backlog, scrum board and burn-down chart. Automated tests and continuous integration are part of the daily business.

Our company has a universal understanding and acceptance of the Lean and Agile Value Chain, beginning with rough user concepts and ending with customer delivery. Without this basic building block, as the theory of constraints suggests, there will either be many unfinished stories in progress (i.e., because operations cannot make regular releases), or developers will starve for stories (because the business is unable to prioritize).

Last year, we decided to grow our software development capability by adding another scrum team in India. After a few iterations, reflections and process adaptations, the remote team became productive. We’ve been successfully running agile offshore for over a year now, and I’d like to share our key learnings.

Can offshore teams really be agile?

If it’s important to have a true understanding of agile and scrum on-site, it is absolutely critical to ensure the offshore team members have internalized the concepts of agile software development. Just knowing what scrum artifacts are, doesn’t necessarily mean that the team understands agile methodology. Every team member must comprehend the purpose behind each artifact. During the initial sprints, we explained at-lengths the purpose of each artifact and encouraged feedback sessions before each meeting to spot misunderstandings early.

The offshore team’s organization structure is the next thing to check. Usually, offshore service providers are structured hierarchically. During project resource staffing, we ensured that the organizational roles would not conflict with the project roles. Our employees are placed within one of three levels of organizational hierarchy. After being hired, new colleagues are briefed about the importance of their scrum role within a project, and how it does not affect their career progression within the company.

During every scrum meeting, we encourage all team members to speak openly about the project rather than only few member speaking for everybody. This frees up the more senior resources for more important architecture work and improves productivity.
Finally, the entire team cannot deliver quality if QA isn’t properly integrated. Testers must be empowered to ensure the delivery as per the definition of done. At the same time, they need to know that their job is not only to report and retest bugs, but deliver quality software together with the team.

Do offshore teams need extra scrum artifacts?

We use the standard scrum artifacts. Initially, we omitted the story preview meeting due to the scarcity of time available for POs, but we soon had to hold it in order to get better prepared stories for the “planning meeting 2″. We observed a very positive impact of stakeholder review on the productivity and morale of the team. Especially for long running (10 months) migration projects like ours, where the offshore team is not co-located with the business, the stakeholder review is a powerful artifact. Stakeholder interest for this meeting can be generated by asking them for what they would like to see/hear, adopting the presentation to their interests, or by encouraging them to click around the software.

Finally, if multiples teams are working on a shared code base, a scrum of scrums is very useful. Everyday, a member from each team visits this common stand up with representatives from the other teams. Each participant explains what their team is doing and if the team’s current activity might block other teams from working.

The Scrum Master Location Dilemma: On-site or offshore?

Simply put, the ScrumMaster should be located where the most impediments are. For us, most of the project impediments were at on-site. As our infrastructure and processes have been tuned over years to suit our local needs, they did not work well with the special requirements of working with an offshore team. Consider the remote availability of important support tools like Jira, our wiki, test databases, and maven repositories.

As I mentioned before, around 70% of development was done on-site and everybody was working on the same code base so it was critical to integrate all development activities. In our case, the ScrumMaster was on-site and his main task was to ensure that all external impediments like unavailable code repositories, broken builds, and lost database connections were addressed and resolved as soon as possible.

However, as the offshore team grew to 9 developers and 2 testers, the ScrumMaster needed to do this job in two places at once. To solve this dilemma, we created a hybrid system for fulfilling ScrumMaster responsibilities with a shared task list. Assigning these tasks to the offshore team, we made sure that the responsibilities are distinct and did not conflict with each other. This sharing of responsibilities will be expanded upon in the next section.

Offshore Planning Meetings slow and painful?

At project kickoff, “planning meeting 2″ was an incredible test of patience for the on-site participant. A new team, poor story quality and a complex domain made it difficult for offshore team members to make any sure estimates. The root cause of the problem was lack of well-groomed product backlog and story estimates in hours. Additionally, the offshore company management had set KPIs for the team that it should finish all stories to which the team commits during the planning meeting. This measure, taken to improve the productivity of the team, was surely in the back developers’ heads while estimating stories. We solved this problem in multiple ways. At first, by enforcing story preview meetings and making sure the stories are well defined and small. Although it helped considerably, meeting times were still not reasonable.

Initially, we did planning meeting 2 immediately after the planning meeting 1, but later we introduced a gap of few hours between these two meetings. In this gap, the offshore team was given a chance to research, brainstorm and come up with the design. This reduced the time of planning meeting-2, and everybody in the offshore team had a chance to actively take part in the discussion.
We split the offshore team (9 developer and 2 testers) in half and appointed a team lead. To avoid the trap of hierarchy, we called them “story responsible” and rotated this role.

Now, planning meeting 1 is done with the entire team, but planning meeting 2 (the detail planning) is done with two teams. This has removed the wasteful pain of an entire offshore team trying (and failing) to plan all stories. With a team lead sharing the ScrumMaster responsibilities, we have also achieved linear scalability. The two team leads take over the moderation of meetings, retrospectives and solving offshore impediments.

Are you communicating well?

As we all know, communication is key to any process. Making use of the predefined, scrum artifacts ensures that communication takes place, but it does not ensure the quality of communication. Quality of communication is directly proportional to the quality of interpretation by the receivers. Ambiguous communication is differently interpreted by the individual project team members. We continuously improved our quality of communication by trying different communication channels and feedback loops.

After months of experimenting, we settled upon these three communication channels: Skype for video, telephone line for voice, and “live meeting” for desktop sharing. Having an uninterrupted voice connection was vital; especially when 11 people are all taking part in a meeting. Hence, we decided to use a land-line for voice communication instead of Skype or Google talk.

Although telephone calls are more effective than emails, certain documents (like guidelines) need to be written down. Most of the on-site communication was implicit, based on situational awareness and simple co-location – definitely not enough for an offshore team to start implementation.

Therefore, we discussed the requirements together with the offshore team and asked them to write the final specifications. The on-site team reviewed these during development and after completion to suggest improvements. To provide a strict framework for the offshore team, we wrote the guidelines for clean-code, pre-commit checklists, and operational wiki notes.

Creating Situational Awareness

One of the powerful tools scrum provides is the storyboard. Of course, for distributed teams, use of common physical board is not possible, so we use Jira with the excellent Greenhopper plug-in. As all 6 teams are working on the same code base, we use Teamcity for continuous integration and keep the build green by using pre-commit hooks. Individual developer commits are staged on the integration server, compiled with the latest revision of code, and, finally, run against all automated tests. Only after passing all of these quality gates, is the commit pushed to the source code repository for the rest of the team.

To summarize, adopting agile software development processes was instrumental to the success of this project. Developing working software in each sprint provided us, as well as company management, with confidence in the early phase of the project, and retrospectives gave valuable input for fine-tuning the process.

Having a fully functional PO role would have definitely helped in further improving the team’s productivity. But, we successfully achieved linear scalability by splitting offshore team into two and letting both team leads share some of the ScrumMaster responsibilities. For the next project, we are working on a more well-groomed product backlog, and better story point estimations.

Looking back, I believe the most important learning was continually improving every aspect of our development. After all, being agile means ‘readiness for change’, doesn’t it?

Other posts:

• Waterfall, SCRUM and Lean Software Development simulation as teaching platform
• How “Good to Great” applies to agile software development
• State of Development: Annual Address on How We Ship Software

Wonderlane

The DevOps hype produces some strange effects. Not only do tool vendors try to jump on the DevOps band wagon by declaring their products “DevOps inside” or listing DevOps as a feature, but companies start to look for a “DevOp” in their job ads. Don’t be misled! Here’s what DevOps is really about:

DevOps Is About Culture

The fundamental basis for successful DevOps is a culture of trust and a feeling of fellowship. Everything starts with how people perceive each other: Is it an “us vs them” culture or is it a “we”-culture? I don’t see any job description in here.

DevOps Is About Automation

Let’s look into automation. A lot of the advantages DevOps promises lies in the right use of automation tools. Automation removes variance from your processes and minimizes human error. While you’ll definitely need people with automation skills and experiences, this is not enough. Automation only – without the right culture – will not provide the benefits you hoped for.

DevOps Is About Measuring

Then let’s examine measuring. While measuring is a critical, mandatory practice for improving processes, it’s not really a job description. Every employee should formulate hypotheses, run experiments, and validate or scrap her ideas. No magic sauce here either.

DevOps Is About Sharing

What about sharing? I’m sorry to say that while the DevOps movement is largely driven by sharing ideas, problems, and tools, this isn’t really a good job description. It is an attitude more than a task.

If you run through the points, which really define DevOps you’ll see for yourself how strange it is to try and hire a “DevOp”. DevOps is not a job – DevOps is Culture, Automation, Measurement, and Sharing (CAMS).

Other posts:

• Do you have a DevOps Culture?
• 2011: Time to Escape the DevOps Echo Chamber
• DevOps: Why Silos Suck And How To Break Them