Getting to the real problem

Rather than simply label the developers as incompetent, to be successful you need to understand a bit about the culture to get an understanding of where this problem could be coming from.  Nearly 1/3rd of the world’s poor live in India, with estimates ranging around 645 million people.  That poverty has led to a culture of making the most from everything they have.  Very little is thrown away, and what is thrown out is then picked through and reused again.  That spirit of ingenuity is what has allowed India to start pulling itself up to become a major influence in Asia and the rest of the world.  There are literally dozens of examples of this that you see every day.  From motorcycles turned into trucks for hauling goods to cow manure used as cooking fuel, everything can be used for something. 

But that ingenuity comes with a downside.  When resources are scarce, the mindset is generally to take the shortest solution to any problem.  One of my favorite examples of this mindset shows up in Indian construction.  All over India beautiful new skyscrapers of steel and glass are going up, most of those do so with construction workers crawling on scaffolding made of bamboo sticks lashed together.  Why?  Because bamboo sticks are all that they need.

For your developers, the same mindset leads to software that functionally works but is of poor quality.  Your requirements outline the functionality required but if you don’t make the quality requirements as equally clear then you leave it to them to decide the path they take and more times than not, that path will include bamboo scaffolding and smell of cow poop.

How to to get past “Good Enough”

The key to ensuring the quality of the software delivery is to create processes that enforce proper practices.  Don’t leave quality decisions to the teams until they show that they understand the level of quality you expect as a standard.

1. Make nonfunctional details part of the specification.  This includes unit tests, code coverage, documentation, etc.   Code is not accepted until these items are also met.
2. Use modern development automation to help find issues before testing.  These include continuous integration, automated builds, and automated regression testing.
3. Provide training for all technologies that you expect the teams to be proficient in.  Don’t assume they already know anything more than basic syntax.
4. Code review EVERYTHING!!
5. Find leaders within your teams who understand the quality required and appoint them as mentors to the teams.

Sometimes a cigar is just a cigar

While my interpretation of the cultural mindset of “Good Enough” might explain poor quality standards, this doesn’t mean that there aren’t some bad apples out there.  Not every developer has the drive and discipline required to produce quality software.  Your best bet for those individuals is the same as bad developers in any country.  Minimize their ability to drop bombs into your code base until you can eventually get them off the team and out of your hair.

<Disclaimer> Please see my earlier post for a basic disclaimer and explanation of the intent of these posts.</Disclaimer>

Setting the stage

In order to understand the viewpoints I’m going to express, it’s worth taking a couple of paragraphs to describe where I started this journey.  I have more than 20 years of experience developing software.  In that time, I’ve held a number of different roles and have worked in every conceivable type of team and industry.  I’ve had projects where I was the only developer and I’ve worked on projects that had literally dozens.  As the industry has evolved, a major paradigm shift occurred as companies looked to commoditize software development and inevitably went searching for cheaper resources.  India and their large systems integrators answered that call and so for the past decade or so, you would be hard pressed to find a significant development team without a native born Indian on it.  In my recent positions I have been a manager of both onshore and offshore native Indian resources, as well as American, English, French, and various other countries.  But the largest majority has been from India.

Here in the USA, there’s been no small amount of controversy over this paradigm shift.  Whether your personal views fall on the side of expanding global markets or nationalist job protection, there is ample evidence available both for and against your views.  In my personal history I’ve been booted from a project and replaced by three offshore resources due to a corporate mandate to move half of all development offshore.  I’ve also staffed projects through my consulting company using offshore resources that were being abused by their management and who refused to let me help for fear of being sent back to India.  And I’ve met more than a few happy, successful Indian natives who after working in the USA long enough to get their greencard eventually went back home to India to be with their family.  But no matter what the situation, the undeniable fact is that the software industry has tied itself inextricably to India.  So as a manager it only makes sense to try to understand software development in this new model, hence these articles.

Disclaimer

While it is useful as a tool to speak of the cultural, work, and management aspects of outsourced software development (specifically in India) in sweeping generalities, please know that I do so with the complete understanding that these descriptions won’t describe every person in/from India.  Each person is an individual and I have had the great pleasure of working with some of the best talent in India.  So as those friends read these posts please remember, I’m not necessarily describing them.  On the other hand, when taken in large numbers the generalizations I will describe are accurate in my perception, if not intent or reality.  I welcome any of my colleagues to correct me if I’ve misinterpreted something in these articles.

I am running 3 24” hi-def monitors (1920×1200) using 2 ATI Radeon 6850 cards.  More than ample video power for gaming, video watching, and maybe if I’m lucky running WPF.  Recently though I’ve been getting a popup that says that “Windows has detected that my computer is running slowly”…  Like Hell!!!  Shortly after that, Aero dies and I get a message that says Windows has switched to the Basic theme and all of my wonderful transparency effects are gone.  Clearly this isn’t a horsepower issue.  After doing some Googling around, it seems that both IE9 and Firefox 4 are causing Windows to freak out when running with Hardware Acceleration (which they do by default) in extended monitors (i.e. not my primary but one of the other two).  There seems to be some finger pointing going on between ATI drivers, Windows, and the browsers and so I haven’t been able to find any clear answer to why this is happening and how to fix it.

In the meantime, I’ve switched to Chrome which has hardware acceleration OFF by default and that doesn’t seem to cause the problem.

After installing all of the drivers, Windows updates, etc. I went about configuring the machine for .NET development.  After installing Visual Studio 2010 with WPF (.NET 4.0) I ran into a problem.  Every time I run Visual Studio it works for a few moments, then everything freezes, my screen goes blank, and then it comes back with a toaster popup saying “Display driver has stopped responding and has recovered”.  A few more moments go by, and the same thing happens again.

After scouring the web, I finally found that WPF (which is what Visual Studio 2010 is written in) is apparently having issues with the latest drivers for the ATI 5XXXX series boards.  I’ve checked the forums at AMD but don’t see anything at all mentioning this problem.

To work around the issue, you can turn off WPF hardware acceleration.  To do this, you need to set the following registry key

[HKEY_CURRENT_USER\Software\Microsoft\Avalon.Graphics]
“DisableHWAcceleration”=dword:00000001

If you’re unsure how to do this, here is a zip file you can download with .reg files you can double click on to disable or enable hardware acceleration.

I’m not entirely certain what the ramifications are of disabling hardware acceleration other than the obvious performance reduction, however with my quad core/8G RAM setup I haven’t seen any noticeable difference.

I especially want to thank my teams.  Together we’ve gone from hours spent toiling day and night in a run down office in west London to hours spent toiling day and night in Bangalore.  I know it hasn’t always been fun, but it’s never been boring.  And mostly it’s been a great honor for me to work with all of you and to call you “my teams”.  I hope that you have all learned much but I want to remind you of a few things you’ve heard from me before so that you remember them after I’m gone.

1. You’re better than you think you are.  Make big promises, push yourself.  You never know what you can do until you remove your own limits. 
2. Always do the right thing.  It’s one thing to know what the right thing is, but entirely another to have the strength not to bow to pressure and actually do it.  Don’t let other people push you into doing things you know aren’t right for our systems.  Remember, it always costs less to do it right the first time.
3. Keep learning.  You’ve chosen an exciting industry to work in but it requires that you constantly learn new things.  Don’t let what you do today be the same things you did yesterday.  Your code/tests/designs should improve with every iteration, every release.

Thank you again for all your hard work and best of luck in the future. 

Paul Ballard

Fortunately some film makers got together and put the experience into a movie called Outsourced.  While the movie exaggerates a bit on the conditions in most of the modern cities in India (for example most professionals work in very nice glass and steel buildings, not concrete bunkers), there were so many absolutely dead-on depictions of everyday life that Susan and I laughed our asses off.  Here are a few examples that we have personally experienced.

<spoiler alert>

• Riding on the trains – I tried to do a trip in the train but the seats were so close together that the person in front of me, who was facing me would have had to interlock his legs with mine to have enough room.
• Getting mobbed by taxi drivers – This happens at every station or airport.
• Cows (and other livestock) in the streets – Absolutely happens every single day.  We’re beginning to be on a first name basis with several. 
• Nice homes next to slums – This happens fairly often but is usually beautiful office buildings next to shanty towns.
• Digestive Distress – The minute the guy ate the snow cone, Susan and I looked at each other and said “He’s going to regret that!”.  While eating in India is easy enough if you’re careful (and the food is GREAT), eating off vendors in the street is usually a bad idea for the western stomach.
• “Unclean” Left hand – we heard about this before we got here but it’s not a big issue in major cities.  Squat toilets (like the one in the movie) are still found occasionally though and many toilets don’t have any toilet paper in them.  Susan has learned to carry some in her purse whenever we leave the house.
• Sugar in coffee/tea – Indians love sweets and all things sweet.  They dump huge amounts of sugar in their coffee and tea.  In fact a common drink from vendors is fresh pure cane sugar juice.  I’ve heard that India has the highest rate of diabetes per capita of any nation in the world.
• McDonnel’s – Okay they have McDonald’s all over India but don’t think you’re going to get your burger fix there.  They don’t serve beef in any McDonald’s that I’ve been to in India.  Instead they have chicken patties and veggie burgers.  They even have the Big Maharajah, which is a chicken patty equivalent to the Big Mac.  Fortunately for me, they also have the Filet-O-Fish and that is what I usually get.  The fries I’m happy to say are the same all over the world.  They also don’t server beef in Pizza Hut, Taco Bell, Papa-Johns, Dominos or any of the other western chains that are all over India now.

Of course those are just some of the interesting and funny things to point out, there were many more subtle things that only somebody who lives here would notice.  It’s these little things that make living in India such a fantastic experience and something I know will be with me the rest of my life.

• Helpfulness – throughout the movie “Toad” is never far from somebody willing to help.  From teaching him how to jump onto the train to helping him find that special room for the night, there’s a cultural desire to help people here that is amazing.  A cynic would chalk this up to their desire to make a quick buck off of what they see as a wealthy  foreigner but there’s more at work here than that.  I’ve seen our local CFO go to what to my American eyes seemed like fantastic lengths to make sure that everybody at a dinner meeting got exactly what they wanted.  It’s a trait in him that I admire and wish came as naturally to me as it seemed to with him.
• “No Problem”, i.e. anything is possible – In the end of the movie, when they move the call center to the roof, I’m sure that seems completely made up.  The reality is, solutions like that occur every day here in India.  This is a country full of people who are used to living with so little that they can solve problems using anything, even a VW hubcap.  No problem. 
• Hard work – Indians are by nature hard workers.  I’m not just talking about the manual laborers or farmers, but even the professionals including my great team of developers.  Long hours and strange shifts (to match their western clients working hours) are the norm here. 
• Holi – The festival of color shown in the movie is very real.  I unfortunately missed it this year as I was in London, but Susan got tagged by her friend Heta.  It’s not as widely celebrated here as it is in Maharashtra (where the movie is set), but it’s still a lot of fun.  I saw the aftermath of the buses and sidewalks when I got back. 

In addition to the funny story telling there are also a lot of excellent insights into the reality of outsourcing and the global markets. 

• Outsourced Jobs – While we like to think we’re the best in the world at everything, the harsh fact is that there are a lot of people in the world willing to do the same job for less money.  The only way to protect yourself from this is not to let yourself fall into commodity work.  Stand out from the crowd, make yourself more marketable.  Even the Indian jobs can be outsourced to the next cheaper country, which happens all the time.
• Made in America – We want to buy American, but we want to pay Chinese/Indian.  American workers cost more money which in turn drives up the cost of products.  We can’t keep pushing companies to provide cheap products and not expect them to find cheaper ways to make them, including outsourcing.  If we don’t like it we can always pay the extra $220 for the American made Bald Eagle statue.  But instinct (and Walmart) says that we’re probably not going to do that.
• “You need to learn about India” – India is the second most populated country in the world and THE most populated English speaking country.  Its combination of low cost labor and access to natural resources means that its economy will continue to grow and become a bigger player in the world markets.  It is absurd to think that any major company can compete without a strategy for Asia which includes India.  1.6 billion people cannot be ignored.  Learning about Indian culture now can only be beneficial both for business and personally.

So if you’re reading this and want to get a better view of what life in India is like for an American, I highly suggest you watch Outsourced.  Oh, and along with all the insight it’s actually very funny, and a bit romantic (in an Indian sort of way).

When the aquarium guy came to set up the tank for me he dumped what looked to be about 14 random fish into the water.  They range in size from about 3 inches long to my big Split Tail Barb (The Boss) which is about 8 inches long and a Tiger Oscar (aka Big Oscar or The Bully) that is just a bit smaller.  My first thought was not to bother even naming them as they’d probably all die from being dropped into the tank the first day it was turned on.  But fate smiled on them and with only two exceptions all of the fish have survived okay so far.

However, all is not bubbles and mermaids in aquarium land tonight.  I came home and happened to notice that one of my smaller fish, a shy nerdy one who tends to keep to himself (I call him Mr. ButtUgly, because… well he’s butt ugly), has become the victim of mob violence.  The poor guy was laying in the sand at the bottom of the tank, looking like he was gasping for air (or water?).  To my horror (to say nothing of his), the other fish had EATEN HIS FINS!  His dorsal fin and tail fins are gone! 

At first I wasn’t sure what had happened.  I’ve never had an aquarium so I thought maybe he shed his fins or something.  And while his gasping looked worrisome I recently had another fish (Little Oscar) survive the same sort of thing for nearly 2 weeks after he got so excited about feeding time he leapt right out of the tank and flopped around on the carpet for a while.  So I turned the lights back off and went about my business.  But after just a few minutes I noticed the other fish were circling him menacingly and then biting at his tail while he laid there defenseless and gasping. 

You’ll be happy to know that I immediately intervened on his behalf and pulled him out of harm’s way.  He’s now resting quietly in a private bucket where I will keep a close eye on him for the next few days and see if he can manage to grow back his missing appendages.  I’m hoping I can nurse him back to health and then feed him such that he gets big and bulky so he can go back into the tank and get some well deserved scaly justice!  In the meantime, if you believe in a higher power (Jesus, Buddha, or L. Ron Hubbard – any one will do), pray for little ButtUgly as he tries to work through the trauma of life in… The Aquarium Of Doom! 

One of the most frustrating parts of being from the US and living in India is that Internet sites you’ve trusted for years will use your IP address to look up your physical location and then do any number of things to show that they don’t trust you.  These often mindless restrictions range from the mildly annoying, like MSDN asking me to sign into the site in Hindi, to the severely frustrating tendency to limit access to media and other site content all-together (Yes Hulu, I’m talking about you, bastards!  I’m an American citizen and I have a God-given, constitutionally guaranteed right to watch Family Guy).

While this might seem like a small nuisance to most readers (assuming I ever get any), this is actually just more evidence of the fact that the Internet is NOT just one huge global network of anonymous freedom.  As Google’s recent fight with China showed, it’s actually a conglomeration of loosely joined networks each with its own rules and limitations dictated by governments, corporations, and other high minded bodies.

I believe in the future we’re going to start seeing these walls become more visible, with clearly defined boundaries for data storage, media access, and even email.  This will have a huge effect on things like offshore development, news reporting (blogs or otherwise), SaaS applications, etc. etc.  And for those of you that think these boundaries are restricted to foreign countries with oppressive government policies, think again.  If you’re reading this blog post from darkest Africa, deeply red China, or the sunny shores of Florida you can count on the fact that some government body somewhere can shut off access to it at any instant.

Receiving the MVP award is a great honor, and one that has to be earned each year.  I was very proud of having received it for the past four years and appreciated Microsoft recognizing the support I gave the community.  But this past year I’ve been buried with work in my new job and just didn’t have the time to speak and write as much as I should have (as the lack of blog posts here shows).  So this year I didn’t ask to be renewed, preferring instead to step down and make room for some other more deserving folks like Rob Marshall and Chris Hay.

I want to thank my US MVP Lead Ed Hickey for supporting me all those years as well as Akim Boukhelif for supporting me on this side of the big pond.  Best of luck to all of the new MVPs and maybe when my work life settles down I’ll once again be worthy of the MVP title.