Water Hammer:  a pressure surge or wave resulting when a fluid (usually a liquid but sometimes also a gas) in motion is forced to stop or change direction suddenly (momentum change). Water hammer commonly occurs when a valve is closed suddenly at an end of a pipeline system, and a pressure wave propagates in the pipe.

They say everything you need to know you learned in Kindergarten.  I still managed to learn a little in later years, including my graduate years studying water.  But the concept is the same — the world is full of patterns, and systems begin to look alike regardless of the discipline.

Which brings me to my point: driving change in an organization has analogies to driving change in a fluid system.  The people dynamics have corollaries in fluid dynamics.  People have momentum and fluids have momentum.

When you close a valve in a water system, it is easy to congratulate yourself for your success.  No water is passing!  I have changed the system!  Problem solved.  But depending on the system, seconds or minutes or hours later you might be dead.  Explosions, ruptures, and implosions, sometimes deafening and disastrous, can be your end.  Why?  The accumulated momentum in the system is hard to stop.  You have to think of all of the water, all the way upstream, and determine where all of that momentum will be absorbed.

In working on 12sprints, we had to change the system of SAP to account for some of our new models.  Elastic subscription services have a ton of ramifications for a traditional software company, from accounting to legal to privacy to support, and every time we reached a decision (closed a valve / redirected flow) we learned that other parts of the system had momentum we hadn’t anticipated.  And were there explosions?  That would be too strong a word, but there definitely turbulent debates and some bent metal in the expansion joints.

My conclusion?  Change is a lifestyle.  It is a lifestyle only survived by being relentlessly attuned to the rest of the system, and using a simple recipe when the pressure builds: be open, creative, and luckier than most.

And always wear waterproof clothing.

I get data in email attachments, in PowerPoint, in Excel, in Crystal Reports.  I read data on the web.   I get data on phone calls, over IM, from the mouths of customers.  I hear about data that other people have and ask, sometimes timidly, for a copy.  Like my brother’s MP3 hording I want to have it, just in case.  There is a comfort in having all of this data.  That is, until I need to find it.

Finding data is hard.  With Draconian email quotas and the difficulties of searching archives, it always becomes lost.  I can find my music now in iTunes, but what about my data?

More and more I use two very different tools to help me survive this information-saturated world: Evernote and SAP BusinessObjects Explorer.  Think of Evernote as your external brain, where you can store your every thought for perpetual retrieval, and Explorer as your company’s external brain, where every last bit of data can be retrieved instantly.  I’ll talk more about Evernote later, but this week seems to be all about BusinessObjects Explorer.

Yes, this week SAP BusinessObjects Explorer is being launched, but the technology antecedents to this powerhouse have been around a while.  It is comprised of a product previously known as BusinessObjects Polestar and some cutting-edge in-memory acceleration technology previously manifested in SAP Business Warehouse Accelerator.  Yet it is their combination is truly stunning.

I am an impatient man.  I love my Mac since I open it up and it is ready with no delay – I cannot wait for something to boot.  I have loved using Explorer over the last year since I can hit a web page and answer my questions as soon as I ask them – I don’t have to wait for someone to build a report.  I just have a conversation:

You say revenues are bleak for a certain product in the United States?  Let’s look at it by city.  Whoa.  San Francisco and Washington are down, when they are up for the rest of the business.  Let’s get California sales on the phone and see what’s happening – send them the link.  Richard, what’s going on here?  Oh, we didn’t run the right campaign?  Let’s budget for that next quarter, it showed good results everywhere else.

Fun with Data

Bingo.  In a 60 minute meeting, we can have 5 conversations like that, and have discussions based on substantive fact.  Coming from the data-desert of past jobs, this kind of knowledge oasis is intoxicating.

Yet combined with the in-memory technology the potential is breathtaking.  Terabytes are now your friend, they are not demons threatening to slow your life to a crawl.  Petabytes are an afternoon snack.  You don’t have to look at statistical samples, you can look at the truth.

So as I index the world around me, play a little for yourself online.  See what it is like to manipulate 1000 rows of Excel in this remarkable tool.  And then imagine what it would be like to manipulate the world.  That is, if you can handle the truth.

Elaine Scarry gave this subject a scholar’s attention in The Body in Pain, where she explained that pain nullifies the world around us — with extreme pain nothing exists but the pain.  This deconstructs the ego to a point where conversation is meaningless and information extracted in this state has one goal: to make the pain stop.  Say anything to make the pain stop.  In fact, there is a long history of torture being used to extract misinformation to support campaigns of misinformation.

While this simple fact is well established in research, it seems appallingly under communicated.  If it was well communicated, I imagine it would lead to this:

Interrogator 1: Should we do it?

Interrogator 2: Well, it doesn’t work.

Interrogator 1: OK then, let’s not bother.

The complex ethics simply disappear.

Now this next logical leap may offend some people - minor pains can have a similarly distorting effect.  I am not trying to equate the horrors of torture with unpleasantness, sometimes, of life, but simply pointing out that minor pains similarly distort “reality”.  And this has something to do with software.

Be careful what you measure in software development

There is an adage that people do what they are measured on.  This works well when the measures are good and objective.  But in software the measurements are often bad and subjective.  When you measure, you distort — just ask Heisenberg.  Do your best to measure what you care about as a business (e.g. Customer perception of a product’s value) rather than how you make the software (e.g. # of bugs per KLOC).

Be careful asking teams to get it done sooner

We all know about the mythical man month.  But if we all ask ourselves how often we pulled in a date just to have it slip back to its original target (despite this knowledge), we are all guilty of it.  And you can toss features out of the window all you like, you can’t always rob the calendar of days or weeks.  Things have their own gestation period — at some point we need to live with that.

Be careful when you think you know the truth

I consider myself a good listener, which I guarantee is true at least part of the time (such as when I listen).  But when I was involved in a platform release at a previous company, I was certain the team was telling me that we were in good shape on the release.   That is really what I heard.  I was later told that people felt that was the only answer I would accept.  This shocked me, since I was never “that guy.”  But unless you sanity check what you are hearing by spending enough time in the hallways and the cubicles and getting a feel for individual’s commitment level, you won’t really know.  In this case, simply having high expectations and articulating them passionately and consistently began to have a distorting effect — people began to play into my fiction with a supporting narrative instead of what they really believed to be true.

And so what?

We could listen to Matthew Alexander describe how brains are better than brutality for gathering intelligence, and think about how its lessons apply to the miniature information distortions in our own lives and work.  It comes down to developing real relationships with real people.  In software development, that means a trust relationship with the development team, with sales, with your customers.  Knowing when to push, but knowing when to listen.  And in software, that is often a tough trick to pull off.

When I hired a guy last year entirely over SMS I committed a gaffe - I received a Twitter DM (direct message) and hit reply, sending the reply to all of my followers.  This is a variant of a DM Fail, when people think they are sending a message to just one person but instead broadcast it widely.  In my case I uttered something relatively harmless like “req opened this week.”

Something more nefarious happened recently on Twitter: they actually sent DMs to the wrong people, detailed in TechCrunch.  Jason accurately called this a “breach of user trust,” but it was resolved quickly.  Twitter is not alone here.  A colleague of mine was using an esteemed Web2 product when they one day got a trove of someone elses messages dumped on their desktop over IMAP.  Only once, but once is all it takes.

This is an area where Web2 (I hate the “oh”) and Enterprise2 differ, since the “average risk of mistake” differs.  In other words, a breach of trust will, on average, cost you more in the enterprise.  Stewart Mader characterized this issue of trust well in his Tale of Two Tunnels, where he characterized Web2 applications as building tunnels in the earth, and Enterprise2 apps as building tunnels under water, with the failure of the latter catagory being quite different.  While we don’t care if we see the Twitter Whale Fail occasionally, “a crack or crumbling of a tool inside an organization is not seen kindly and raises doubts around the viability of the tool.”

But here is the point of this post: DON’T SCREW IT UP BY BEING TOO CAREFUL.  For those of us building Enterprise2 stuff, especially those of us that have been serving the enterprise for a while, then tendency is to be too careful.  I remember a discussion with the CIO of a major CPG customer a year ago.  I had energetically described how all of our activity feeds (I might have worked for a different vendor then) had three layers of security — first people opt in and share, then the assets are redacted based on ACL, then central XACML policies can do a secondary redaction.  The guy looked at me for a while, and said: Forget everything but sharing.  We want this thing to grow.  If people misbehave we can fire them.

Ironically, this CIO was talking about trust.  ”We can fire them” sounds negative, but he was actually saying “We trust our employees to do the right thing by the business.  And if they don’t we can fire them.”

This was a breath of fresh air.  Too often the systems that try to “keep us in line” cause us to abandon the official tools and use tools that work.  We want to comply, but we MUST get our work done.  A CIO that wanted to err on the side of trust (we’re talking people, not certificates) seemed modern indeed.

When designing systems keep this in mind.  Keep an audit trail, not a frustration trail.  Let people know they are making mistakes, don’t necessarily prevent them.  And if you trust them, you just might find that they behave a little more responsibly.

Now if I could just open this confidential attachment on my blackberry…

—

Times are tough.  Buckle down.  Buck up.  Be lean.  Be efficient.  Do more with less.

Yes, Indeed.  But before we all become super-efficient pieces of a super-efficient machine let’s take a moment for pause.  Because there is a downside of efficiency (or at least what the world around us often calls ‘efficiency’)

I think it was in graduate school when it first became clear to me that efficiency was great until there was a problem, and then it wasn’t so great.  I was studying complex water systems, and how to manage them to leverage the most capacity (electrical and consumption) while preserving fish happiness and keeping the land pretty.  Once you balance all of the uses into a finely tuned system, and make it reliable, people will build complex systems around that reliable water, and if it isn’t there… well complex systems start to break down.

What this amounts to is setting up systems to have the butterfly effect, since making systems more efficient generally creates more complex dependencies.

Consider George Monbiot’s recent interview with Fitah Birol’s, Chief Economist of the International Energy Agency.  Much of the world’s governments rely on assessments of the IEA regarding how long oil supplies will last, and it turns out they now think they were off, but only by a factor of 2.  They had modeled a decline in output of 3.7% per year, which they now think is 6.7% per year.  And we’re using more.  So we’ll be running out around 2020.  This means that “unconventional” sources of oil, like tar sands, would need to be processed into oil to keep the machine running, but don’t worry: that would only amount to an environmental catastrophe.  It is similar to the phases of drilling in conventional wells, where primary, secondary and tertiary recovery start to require more resources, be worse for the environment, etc.

Or consider the minor business dispute between Russia and Ukraine that led to freezing out Slovakia, the Czech Republic, Austria, Hungary and Romania over the past few days.  As gas flows efficiently between regions, complex systems begin to rely on it, so it better keep coming.

In other words: efficiency increases output, output begets demand, demand requires continued output, and the scenarios of Too Big to Fail, or the corollary, Too Big to Exist.

To my friends who work in software this is just obvious: we are asked to use resources “more efficiently” all of the time, which usually means very little spare capacity to handle unexpected events.  And the software business = unexpected events.

So my solution?  I endeavor to become less busy.  Take more breaks.  Chew food slowly.  Say no.  Which will of course require and enable “true efficiency,” but it might not look like that on the books.  And, of course, will never happen.

Internally, the labs team continue to innovate this offering in interesting ways. One way has been to add an API that allows applications, pages and tools to push prepared data to Polestar and embed the cloud based solution into their products. Which is interesting, but not as interesting at the amazing things that start to happen once that API happens.

For example, one internal team created this video of an internal test showing that Polestar could be embedded into a classic SAP applications.

This is just someone playing with an idea at this point (so don’t look for this built into your SAP applications just yet), but it really demonstrates the power of an API to generate the unexpected. Hopefully we will see lots more innovative uses of these API.

I always wanted a word for that, for flow. When I was developing software I always really loved that sense of “flow”, complete focus on solving a problem, creating the solution. You would surface from your work and wonder how it could be so late.

Today, I only really experience that while playing games. I think this is the magic of games, that they tap into this need for flow and allow us to get lost. They do this by providing such clarity around achieving our goal, constant learning, and the sense of momentum towards that goal that is immediately visible. This is wonderfully and humorously described by Daniel Cook in his presentation on “Rescue Princess 2.0″

Now I am in a role that focuses more on the meaningful aspects of happiness, working with people and ideas, influence and communication, littered with interruption and changing focuses, constantly working around what David calls “Rules that Suck” (RTS) - the typical life of any manager or executive decision maker.

My work life, and my home life as a parent, is no longer particularly filled with periods of flow: it does not lead me to learn what I need to succeed;  rarely do I get a sense of momentum towards defined short term goals; there is no visible scoring system; levels can take years to complete and perhaps worst of all there is no pause button.

The relentless piling of law upon law—the federal register has 70,000 ever-changing pages—does not make for a more just society. When even the most trivial daily interactions are subject to detailed rules, individual judgment is stifled. When rule-makers seek to eliminate small risks, perverse consequences proliferate. Bureaucrats rip up climbing frames for fear that children may fall off and break a leg. So children stay indoors and get fat.

The point of the article is that Obama’s likely appointment of Cass Sunstein to the White House could help address some of these issues, perhaps even leading to real (and needed) tort reform.

That got me thinking to business in general, and a recent study that we commissioned from The Economist showing that 70% of people need to work around their company’s established processes in order to effectively get their work done (Actually, when I pose that question to a group of successful people, the actual answer is 100%).

Rules are good.  Unless they suck.  When I’m confronted with rules that suck (RTS), I’m often reminded of Kierkegaard’s Fear and Trembling, which discusses when it is our individual obligation to rise above the rules of society and embrace a higher truth (yes, the subtleties of this particular example are distracting).

Kierkegaard was contemplating the relationship between Abraham and God when God commanded him to do something unthinkable (sacrifice his son).  When we are confronted with RTS we need to understand our equivalent of God.

Since my business is building software, and I think that software lives to serve the customer and make their lives better, Serve the Customer is my guiding principle when I’m confronted with RTS.  If a RTS serves the customer, then it must be for the greater good.  If a RTS prevents me from serving the customer, then I need to figure out some way around it.

RTS are the primary reason we all need to work around policies and practices all of the time.  Many RTS hearken from an increasingly outdated business culture, where command-and-control was the pervasive operational model.  One person decides, everyone else follows.  Rules are critical for large organizations to run effectively, but there is a limit: When processing a Purchase Order costs 10x what you’re purchasing, maybe letting people expense it is OK.

The best way to create Rules that Don’t Suck (RTDS) is to, where possible, respect the intelligence of the person they are targeted to.  With respect comes trust and understanding, and generally less suckage.  So let’s simplify tort law and enterprise business in one fell swoop: just empower people to use their brains and then make them accountable.

What is really interesting is that the data set is only 16 rows pulled from a website, yet you can quickly glean really interesting facts and trends. Thank you Fredrick - great video. As a Brit living in Canada, it seems I am doomed to hear about Hockey all the time. Clearly, I need to go and get some Rugby data and produce the next example myself.

Also, here is the spreadsheet so you can try this your self at http://polestar.ondemand.com

Alan Kay’s presentation at TED

Xcelsius allows you to build models in Excel, load them and dynamically visualize the information. So Excel and Xcelsius provides the modeling, now we just need to really really understand our business and create that library of working models.