Looking at Software

Finding an Accidental Cross Join in an Oracle Query

2015-02-03T19:26:00.001-08:00

Ever work on a query that joins multiple Oracle tables and one of them is cross joining on multiple rows resulting in multiple identical data rows in your output? Ever sit wondering like me which which table is at fault? Found a neat way to figure that out without having to actually think.

We start with something like this:

SELECT

list_of_fields

FROM

table_a a,

table_b b,

table_c c,

table_d d

WHERE

blah, blah, blah

Just add this to your list of fields:

a.rowid "A",

b.rowid "B",

c.rowid "C",

d.rowid "D"

(You don't have the "A", "B", "C" and "D" if you'd rather let all the columns be called "rowid" and count column positions--which is what I actually did.)

The output looks like gibberish, but on at least one of the columns the rowid will have at least one character different from the previous row within the same set of "identical" rows. That's the table at fault.

Best wishes!

The Failure of Large Business Applications

2010-12-26T07:43:00.000-08:00

There was once a day when do-it-yourself software ruled. Companies hired scores of programmers to create COBOL applications that drove the business cycle. Then along came the software vendor who promised massive savings by splitting development costs across multiple businesses. The argument made sense so it wasn't long before in house applications were being replaced by purchased applications. The teams of in-house programmers were released as companies looked forward to large software savings.

But the costs continued to rise. . . and rise. . . and rise. . .

Those teams of in-house programmers were replaced by armies of contractors and off-shore developers who customized, tweaked and upgraded the newly purchased applications. To put it bluntly business applications have been an IT disaster. It is not hard to see why.

Fact: Software vendors make money on initial sales

Result: Initial sales are driven by the addition of new features.

Fact: Software vendors make lots of money on support

Result: Upgrades justify support

Fact: Supporting multiple versions costs money

Result: Only the most recent versions are supported

Overall Result: Businesses are forced into an upgrade cycle that only benefits the software vendor and costs businesses enormous amounts of money. While initial acquisition costs have dropped the total cost of ownership has spiraled out of control.

I believe there are solutions--which I will address in an upcoming post.

Is a Software Launch Like a Rocket Launch?

2010-12-02T16:50:00.000-08:00

The minutes and hours leading up to the launch aren't very impressive. The rocket is slowly put into place. Equipment is checked and rechecked. Then all at once with a shuddering roar it happens.

Launching software systems can be very similar. First, good software requires detailed preparation. All possible scenarios must be examined , evaluated, implemented and tested. Secondly, regardless of the preparation when the rockets are ignited you never know exactly what's going to happen. Pieces shake loose, sometimes rockets even explode.

In the end the success of any kind of launch is based on the stability of the design and the preparation that went into it.

It's Not My Documentation

2010-01-04T18:12:00.000-08:00

Almost every software developer knows it's good to document their software.

Almost every software developer doesn't.

Almost every software developer is smart.

Therefore, it is smart not to document software!

I say that partly in jest, but I really do think software developers are just not ideal at writing documentation. It's not that we're bad at writing. Most developers I've known are actually pretty good writers. The real reason software developers make such poor documenters is we generally don't know how to think like our audience. Writing documentation is like writing code--understanding the needs of the end user is critical. Our technical training simply doesn't prepare us to do well.

I've never tried this, but it seems that an ad-hoc team made up of a documentation consumer (at the keyboard) and a programmer (to provide information) would make a powerful combination.

The Critical Moment in Software Design

2009-12-02T16:15:00.000-08:00

Imagine yourself in a meeting with a user (or group of users) discussing requirements for a new software feature. At some point--usually about 15 minutes into the discussion--the following phrase almost always comes up:

"But that doesn't happen very often."

It's often said with an awkward expression or laugh. This is the critical moment. Behind those non-committal words the user is really thinking--"These are details. This is my world. Does this geek care about my world? No, of course not." Of course behind our geeky glasses and awkward social graces we care a great deal about the user's world. The detail of the user's world is critical. Without detail there can be no accurate design. If the design is flawed then everything else will be broken too.

My response is typically like this:

"Often it's those things that don't happen very often that cause the most trouble. If you don't mind let's dig into this a bit more?"

I have never talked to anyone who was't thrilled to go into great detail once they felt that they had "permission" to get into the details.

The Missing Peice of Design

2009-11-22T06:51:00.000-08:00

We've know for years that defects are easier (and cheaper) to fix during requirements and design than during coding and test. Yet that knowledge doesn't seem to reduce the number of requirements and design defects .

What's particularly weird is that all other factors being equal most "lets do it"(1) projects that I've observed turn out to have significantly fewer requirement and design defects than projects with heavy investments in design.(2) Until now I haven't understood why. In a Eureka moment late last week I finally realized why.

The key is that design does not come from reality. Design comes from a mental model of the world. Like this:

Requirements and design problems are almost always are rooted in the disconnect between the mental model and reality. It makes logical sense. Designers are almost always smart, dedicated and detailed. Once a piece of information has become part of a designer's mental model it is unlikely that it will be forgotten or corrupted between the model and the design. The more time a careful designer spends polishing and refining the design the less time they have to develop and verify their mental model. Thus the very thing--careful preparation--that is suppose to reduce risk actually increases it!

Of course modern software developers don't (I hope) follow the simplified model above. Instead we follow an iterative model. Here is the classic Boehm model from 1988.

The problem with this model is it doesn't reflect the disconnect between reality and design. Instead I propose the following:

If the mental model is close to reality projects will slide cleanly into stage 4. If not, there is a jarring bump as the design crashes into reality. If your design crashes, watch out. It will be tempting to slap a patch in place and move into the next iteration. Don't. Stop and re-calibrate. Scrap the entire previous iteration if you must. In the end it will be a lot cheaper than working around a flawed design for the life of the software.

----------------------------------

1) By "Let's Do It" I don't mean unplanned, I just mean less formal.

2) This may sound suspiciously like a argument for the Agile development process and perhaps it is. I've never worked in an Agile environment so I can't say.

Who Should Write the Docs?

2009-08-16T18:25:00.000-07:00

User Documentation: It's tempting to hire an unemployed English major so that the writing is good. But the point of documentation is not good writing, it's useful writing. So skip on the writing expert and pick a system user. Someone who actually knows the pitfalls of the software in a real work environment will create a much more interesting and useful product. (Maybe hire the English major as an editor)

In Line Code Comments: I'm amazed at how many developers don't comment their code. I'm not talking about /* Start of loop */ -- oh duh. I could see that in the code. That kind of comment is stupid. What I am talking about is /* this logic is more complex, but it handles leap year where other alternatives don't */. In 5 years what would I need to know about this code? Document that. We need to demand of ourselves and our peers that undocumented code is unfinished code.

Compilation/Configuration Comments: For small shops with limited resources this is critical. Either the developer or whoever is in charge of deployment has to document this stuff. It's not fun. It simply must be a discipline.

Design Documents and Other Construction Artifacts: Nobody. No, I don't really mean that. But, as a general rule it seems that too high of a percentage of documentation time is spent generating documents that are useful for only a brief period during the early construction phases. These "big picture" documents are important. But there also needs to be detailed technical documentation describing the finished system. That step is so often left out.

Wiki, wiki, wiki: A document is never finished. Make it easy to update.

Congressional Website Overloaded

2009-08-10T18:20:00.000-07:00

I just wrote my Congressman and www.house.gov was reacting VERY slowly. Must be a ton of people sounding off about health care. I wonder what the numbers look like. Must be huge. Probably a big ugly security layer eating up 3/4 of the CPUs.

Why We Don't Document our Software

2009-08-09T11:07:00.000-07:00

It starts with a phone call, frantic email, instant message, or personal visit. System X is down. The programmer for system X left the company 5 years ago and nobody has touched it since. If it doesn't get fixed soon the widgets will start sprouting legs and marching on IT. They'll be torching the place shortly if you can't fix it fast!

Quick, where's the docs? Um. . .

"this space left intentionally blank"

Yeah, it really stinks. If it hasn't happend to you yet, just wait, it will. It's ironic since as a software developer I'm proud to work in a very rational field. Software Engineering is one of (if not the) most logical profession in the world. You would think we wouldn't allow this kind of chaos to happen. But we do. All the time.

Because we are so logical the fact that we rarely document our work therefore can not be perceived as illogical. As noted in a previous post common excuses such as: poor writing skills, too busy, etc, don't stand up under scrutiny. There must be a rational, logical explaination.

In economics they teach a principle called "opportunity cost". Whenever a choice is made there are always other options that are excluded. For example, during the years I studied software engineering I simultaneously lost opportunities to study other subjects. More pertinently, the hours spent writing documentation are not spent writing code.

Software Engineers are hired to write code. The flashier, cooler and showier the better. Writing documents may earn a polite "atta boy" from the boss (followed by "how's project X coming along"). The only way to earn real kudos is to deploy code or rescue the company from a dire emergency. (The irony is that many of these dire emergencies are created by undocumenttion.) From time to time we may out of guilt write some docs, but inevidably we gravitate towards the rewards, leaving behind a wake of sparsely documented systems. What is unusual is not the lack of documentation, but the documentation that does exist.

The solution is breathtakingly simple: treat documentation as the equal of coding. The open source community has already illustrated that this can be done. Since I haven't been a part of an open source team I can't verify this personally, but from an external view I see some factors at work that I don't see in software for hire.

the documentation develops in parallel with the code
documentaiton authors are rewarded equally with code authors (in non-material ways)
the documentation is highly visible to the people handing out the rewards

If we can find ways to implement these concepts inside software for hire shops my bet is that poor documentation will become a thing of the past.

P.s. Anyone out there already doing good documentation I'd like to hear some suggestions. Pleas post a comment.

Response to The World’s Economic Problems are Like 2 Cross-Wired Thermostats

2009-08-06T18:28:00.000-07:00

Recently one of my favorite bloggers Harwell Thrasher posted an interesting (and well written) article on economics. Unfortunately, many of the basic assumptions underlying the article are deeply flawed. Since these assumptions are generally believed by the majority of educated Americans I decided to dubunk them here in my blog. For many years I believe these myths too, so by no means am I degrading the great work Harwell does in his blog.

First, there is an implicit assumption that money is a constant scale–-similar to a temperature scale. The exact opposite is true. Money is a commodity with a value that varies greatly over time and place. There are countless examples—a lunch that costs $1 in El Salvador may cost $5 in Tennessee and $10 in New York City. More pertinant to the economic discussion a dollar spent on housing in some markets will buy twice as much house as it did a couple years ago. In general, during recessions the value of money increases dramatically. That increase in value is what triggers entrepenurial activity and re-ignites consumer spending. (i.e. "Wow, this is cheap--I'll buy it." As more sellers and buyers move into the markets the economy recovers.)

The increase in the value of money has a secondary effect of triggering a round of bankrupcies. (It's harder to pay off debts with increasingly valuable money.) Popular opinion labels bankrupcy as a bad thing. It is to the person going bankrupt. But for society at large bankrupcy is generally a possitive. It removes assets from (mis)management that ran it into debt (generally an indicator of future losses) and transfers those assets into the hands of managers who have money to invest (generally an indicator of future success). This reshuffling of wealth into the most capable hands helps lay the groundwork for recovery.

Second, there is an assumption that recessions/depressions do not end without government intervention. Historically the opposite is true. Recessions/depressions are self correcting. (Of which there are numberous examples--three in my lifetime alone and I'm not that old) The only time a recession won't self correct is when government intervenes. Most notoriously Japan’s Lost Decades and our Great Depression. Unfortunately, the Great Depression is so deeply impressed in our collective oral history as a unique event that we don't realize it was simply an ordinary recession that wasn't allowed to self correct.

Third, there is an assumption that recessions are financial events. The cause is actually a misallocation of labor. This misallocation is typically created by a credit bubble that draws resources away from productive activities into pseudo-productive activities. (Usually good activities, but WAY too much of them.) The solution is to reallocate the unnecessary bankers, construction workers and auto workers into productive activity.

The only government solution that might work is education/retraining. The other common government solutions all work directly against the steps needed to recover. Bailouts attempt to keep misallocated workers misallocated. Temporary diversions like building bridges to nowhere and crushing cars are only marginally better since they divert labor into temporary (and wasteful) activities rather than into permanent productive employment. Quantitative easing (printing money) is probably the worst government solution of all. As noted above a deflationary phase is critical to generating a recovery. To the extent that the Fed/Treasury succeed in preventing deflation they prevent recovery. Other financial controls like minimum price and wage controls have the same dampening effect since they prevent entrepreneurial activity from taking root.

From a political perspective the efforts of the Bush and Obama administrations (the short-term economic policies have been similar in both administrations) have been brilliant. However, the economic effect of these policies (if continued) will result in years of prolonged recession. (1)

It’s frustrating to see this happening and know that the 99.9% of American’s are simply not economically educated enough to understand the mechanisms of what is taking place. My only hope is that we will become frustrated enough that we’ll kick out the current leadership team in Washington (Democrats and Republicans) and bring in a new crowd.

1) I have to give credit to Michael Shedlock for introducing me to the concept of an "L" shaped recession--a crash followed by a generally weak economy that doesn't recover. His blog is well worth following.

Why Don't We Document Our Software? Part 1: False Excuses

2009-08-04T05:22:00.000-07:00

Quick what's the right answer:

a) We're bad writers

b) We don't like to write

c) We are too busy

d) All o the above

OK, trick question. The answer is: e) none of the above. Here is why:

a) Most software developers have at least a bachelor's degree from a liberal arts school. As a general rule we have at least as much training as the biology major writing the user documentation. Often we have more. Software developers write blizzards of email and fill up message boards with numerous comments. We can write.

b) If this were true why are so many open source projects some of the best documented systems out there? When programmers develop "for fun" we write documentation.

c) There is a kernel of truth in this. Everyone is busy. But the truth is we find time to do what's important to us (and our managers). Documentation just isn't.

In my next post I'll examine the real reasons we don't document our software systems.

Science and Art in Software Development

2009-07-21T05:23:00.000-07:00

Art is best appreciated emotionally. Science logically. The subject matter however can easily overlap. I can stare at the Mona Lisa's enigmatic smile or I can study 16th century clothing styles--all in the same painting.

Software development contains the same split personality. I beleive the best software developers understand when and where to apply both logic and emotionality. It's not unusual for me to be working on a problem and "feel" about it in a certain way long before my logical mind catches up.

Generally, those feelings are right. However--while emotions tend to form more quickly than logic. they also tend to be more extreme. Logic is required for proper balance.

Historically, the general public has seen programmers as all logic and no emotions. I'm not sure that's really true. I think our field is filled with emotional people, they are just emotional about topics that the public doesn't understand. Go .Net, Java Stinks!

(Feel the emotion?)

My Code or Someone Else's Code?

2009-07-09T09:08:00.000-07:00

Most software developers if given the choice of maintaining their own code or someone else's code, will choose their own. That's unfortunate. Editing other people's code is one of the best things that can happen to a developer. It's a great way to see what works and what doesn't.

Especially early in a career it's not always obvious what practices are best. Fixing problems in ten year old code is very revealing. My own coding style has become simple, spartan, structured and well documented. I didn't code that way in school or early in my career. I learned it as result of many, many hours of maintenance coding.

Working in someone else's code base is also good for the code. As I noted in my previous post, hacking in new code can be a very destructive process. When editing my own code it's very tempting to hack new features because I know where all the easy attachment points are.

It's not nearly as easy to hack in someone else's code without risking nasty side-effects. That forces a full re-evaluation of the code and often significant restructuring. The result is that the code ends up containing the best of both developers.

Coding is Bad For Your Code

2009-07-07T21:58:00.000-07:00

There is nothing as destructive to a computer program as a good day of coding. An afternoon can wipe away a week's supply of simplicity and clarity. Gone are the clearly defined functions with well commented logic. In it's place is mess of IF and WHILE logic addressing all those special cases that didn't come up in the initial design.

We all know we "should" refactor. But the clock is a remorseless taskmaster. It's so easy to just hack in the feature we need and "design it better next time". Of course, there is no next time. Only now. There is no "design once, write once". Only design and re-design. Code by design, or hack without one--there is no middle ground.

Microsoft Live Search is Now Bing... I Guess Bling Wasn't Available?

2009-06-01T21:25:00.001-07:00

Bing.com is the new Microsoft search! Wow, I guess Bling wasn't available. . .

Very strange name for a service that has struggled from the beginning. I somehow don't think a name change will do much to improve Microsoft's search standings compared to Google. But, I do have to laud Microsoft for staying in the game.

Yes, “Live Search” was a mistake. Ever heard anyone say “I'm going to Live this?” Nope, neither have I. Somehow I don't think “binging” is going to join the pages of Webster's any time soon either, but at least Microsoft recognized that “Live Search” sounds too “corporate” and, yes, “dead”. Bing may be a bit off the wall—but so was “Twitter” and “Flickr”.

After a failure as large as Microsoft search most companies would have either thrown in the towel or buried their heads in the sand and declared that everything was just fine (while quietly letting the product die). By renaming their search service Microsoft is showing some real strength. Renaming is tantamount to admitting failure and most companies aren't brave enough to do that.

The question is—will this just be a marketing campaign? Will their technical guys blindly keep following the "safe" and empty path that Google has smashed down in front of them? Or will they actually hack out some innovations? The recent success of StackOverflow.com in a field very close to search is proof that innovations in search are still out there ready to be exploited--if Microsoft has the guts and vision to do it.

StackOverflow: Finding Success in Failure

2009-02-25T19:33:00.000-08:00

"what's so special about stack overflow that isn't already out there in the zillion other QA forums on the web?...Joel and Jeff need to read the book "Blue Ocean Strategy" and re-think this idea."

--My comments on StackOverflow 10 months ago

OK, I was wrong. Extremely wrong. StackOverflow has become one of the best resources for software developers around. It's the classic success story--great concept, well coded, explosive take-off. Over 100,000 questions have been posted so far with no end in sight.

If I believed in the typical "business book" script I'd go on about how the brilliant concept, the great developers, the teamwork and the pod casts came together to make this great software. . . but that would be a lie. Lots of start-ups have good ideas, great developers, good teamwork and podcasts. Then they fall flat.

StackOverflow had something very special that 99% of start-ups don't have--business skills. Jeff Attwood is the author of a successful blog. Joel Spolsky runs Fog Creek Software--in high cost New York City of all places--and also runs a major blog. Yes, blogging is a business--good bloggers know how to package and promote a "product". The hit counter is a brutal measure.

Every business is long on ideas and short on resources--particularly a start-up. Founders must brutally focus on what really matters. Which means they have to KNOW what matters. Experienced businesspeople aren't always right--but they are right a whole lot more often. Jeff and Joel were right.

StackOverflow is a businessperson's dream. Unlike most start-ups it is dripping with opportunities to make money. First, everything on the site is tagged and most users on the site are identified up front. That means advertising can be targeted, relevant and use less space. Secondly, StackOverflow has tapped into a powerful meta-concept. There is nothing about StackOverflow that explicitly requires it be a programming site. The StackOverflow framework could easily become a service made available to any paying company wanting a more effective forum.

Congratulations Jeff, Joel and team. I look forward to seeing whatever project you come up with next. I promise I will think long and hard before joining the chorus of nay-sayers next time!