The Programmer's Paradox

Death by Code

2013-05-20T16:00:00.001-04:00

A mistake I've commonly seen in software development is for many programmers to believe that things would improve on a project, if they only had more code.

It's natural I guess, as we initially start by learning how to write loops and functions. From there we move onto to being able to structure larger pieces like objects. This gradual broadening of our perspective continues, as we take on modules, architectures and eventually whole products. The scope of our understanding is growing, but so far its all been contained within a technical perspective. So, why not see the code as the most important aspect?

But not all code is the same. Not all code is useful. Just because it works on a 'good' day doesn't mean that it should be used. Code can be fragile and unstable, requiring significant intervention by humans on a regular basis. Good code not only does the right thing when all is in order, but it also anticipates the infrequent problems and handles them gracefully. The design of the error handling is as critical (if not more) than the primary algorithms themselves. A well-built system should require almost no intervention.

Some code is written to purposely rely on humans. Sometimes it is necessary -- computers aren't intelligent -- but often it is either ignorance, laziness or a sly form of job security. A crude form of some well-known algorithms or best practices can take far less time to develop, but it's not something you want to rely on. After decades we have a great deal of knowledge about how to do things properly, utilizing this experience is necessary to build in reliability.

Some problems are just too complex to be built correctly. Mapping the real world back to a rigid set of formal mechanics is bound to involve many unpleasant trade-offs. Solving these types of problems is definitely state-of-the-art, but there are fewer of these out there than most programmers realize. Too often coders assume that their lack of knowledge equates to exploring new challenges, but that's actually quite rare these days. Most of what is being written right now has been written multiple times in the past in a wide variety of different technologies, It's actually very hard to find real untouched ground. Building on past knowledge hugely improves the quality of the system and it takes far less time, since the simple mistakes are quickly avoided.

So not all code is good code. Just because someone spent the time to write it doesn't mean that it should be deployed. What a complex system needs isn't more code, but usually less code that is actually industrial strength. Readable code that is well-thought out and written with an strong understanding of how it will interact with the world around it. Code that runs fast, but also is defensive enough to make problems easier to diagnose. Code that fits nicely together into a coherent system, with some consistent form of organization. Projects can always use more industrial strength code -- few have enough -- but that code is rare and takes time to develop properly. Anything else is just more "code".

Monitoring

2013-04-21T14:14:00.000-04:00

The primary usage of software is collecting data. As it is collected, it gets used to automate activities directly for the users. A secondary effect from this collection is the ability to monitor how these activities are progressing. That is, if you've build a central system for document creation and dissemination, you also get the ability to find out who's creating these documents and more importantly how much time they are spending on this effort.

Monitoring the effectiveness of some ongoing work allows for it to be analyzed and improved, but it is a nasty double-edged sword. The same information can be used incorrectly to pressure the users into performing artificial speedups, forcing them to do unpaid work or to degenerate their quality of effort. In this modern age it isn't unusual for some overly ambitious upper-management to demand outrageous numbers like 150% effort from their staff. In the hands of someone dangerous, monitoring information is a strong tool for abuse.They do this to get significant short-term gains but these come at the expense of inflicting long-term damage. They don’t care, they're usually savvy enough to move on to their next gig long before that debt actually becomes a crisis.

Because of its dual nature, monitoring the flow of work through a big system is both useful but also difficult. It is done well when it gets collected, but is limited in its availability. Software that rats out its users is not appreciated, but feedback to help improve the working effectiveness is. One way of achieving this latter goal is to collect fine-grained information about all of the activities, but only make it available as generalized anonymous statistics. That is, you might know the minimum and maximum times people spend on particular activities, but all management can see is the average and perhaps the standard deviations. No interface exists for them to pull up the info on a specific user, so they can’t pressure or punish them.

Interestly enough, when collecting requirements for systems, fine-grained monitoring often shows up. Not only that, but there is usually some 'nice sounding' justification for having it. Most of software development these days is oriented to giving the ‘stakeholders’ exactly what they want, or even what they ask for, but this is one of those areas where professional software developers shouldn't bow directly to the pressure. It takes some contemplation, but a good developer should always empathize with their users -- all of them -- and not build anything that they wouldn't like applied to themselves. After all, would you really be happy at work if you had to do something demeaning like punch a timecard in and out? If you don't like it why would anyone else?

Organization

2013-03-24T16:50:00.000-04:00

“A place for everything, everything in its place.”

Benjamin Franklin

As Benjamin Franklin pointed out there are two parts to organization. The first is that absolute everything needs to fit somewhere. With software this really translates to having a solid 'reason' for every little bit in the system, be it config files, methods, data etc. and a reason for its location. It all needs its place, and often for that it also needs some level of categorization. "It doesn't matter" is synonymous with "it's not organized".

This includes names, conventions, even coding styles. Everything. Each tiny piece of work, each little change should all have its place. It should all have a reason for being where it is, as it is.

The second half of the quote is just as important. What's the point of having an organizational scheme if its not being used. As new things are added, they need to be added into their place. Sometimes it’s clear where they belong, but often times it hasn't been explicitly documented. Assuming that lack of documentation equates to lack of organization (and as such it is a free for all) is a common failing amongst inexperienced coders. Things can be organized with having an accompanying manual.

Failing to keep a development project organized is the beginning of the end. Disorganization is probably the worst aspect of technical debt because it is a direct path into various forms of spaghetti. And spaghetti is a quagmire for time, either to fix bugs or to extend out the functionality.

Part of software development culture over the last couple of decades has been a strong desire for 'freedom'. Freedom is great, but used improperly it just becomes an excuse for creating disorganization. For instance, utilizing the full freedom to create a new screen in a much fancier way than the existing ones is really just breaking the organization of what is already there. It's like helping someone dry the dishes in their kitchen, but then insisting on placing the clean stuff in any other location than where it actually belongs. It isn't really helpful is it?

It's true that in some regards a well-organized existing development project is a boring one. If the work isn't exceeding the existing organizational bounds then it can really just fit in like all the other pieces. But success in development doesn't come from making any one small piece of the system great, it's an all or nothing deal. The system is only as good as its crappiest screen or stupidest functionality. Thus changes to enhance it or its organization can't be selective. The whole thing needs to be fixed or it’s actually just making it worse.

If you're evaluating an existing development project, disorganization is easily the best indicator of the future. A small project can get away with disorganization, but anything larger absolutely needs organization to survive. If it is lacking or failing, then the whole effort is in deep trouble.

Deep Underground

2013-03-03T12:59:00.000-05:00

There are two distinct sides to software development. Since software is a solution to someone’s problem -- a tool to help them in some way -- it lies in a specific business or ‘domain’. Its purpose is tied directly to helping automate, track and organize very specific information. But the software itself is constructed using parts like computers, networks, libraries, programming languages, etc. so it is composed of and limited by ‘technology’. Both the domain and the technology are necessary ingredients in being able to help users, rather than make their issues worse.

If a software development project goes wrong by mostly ignoring the domain issues we usually say that it was built in an ‘Ivory Tower’. It’s a nice metaphor to describe how the developers may have completed the technical side, but they were too far up and away from the domain problems. The software is useless because the creators didn’t bother to dig into the details.

But what about the opposite problem? What if the developers pay close attention to the domain, but fail entirely to properly handle the technical issues?

Since that is fundamentally the opposite of an Ivory Tower we can flip the terminology. An ‘Ebony Dungeon’ project then is one that delves deep into the heart of the domain, but so deeply that it ignores the technology side. We often see this in in-house projects where the business or domain experts exert too much influence over the process, techniques and construction of the software.

Most domains have to tie themselves closely to some form of revenue stream. Those ties mean that they need to react quickly to changes. That sets up a culture of just focusing heavily on the immediate short-term, trying to be as malleable as possible. That works for business, but a large software development project is actually a very slow moving ‘ship’. It slowly goes from version to version, plodding along for years. While the business may need to wobble back and forth as its demands change, the only efficient path for development is to steer as straight a course as possible. There are a lot of moving parts in software and to get them all working properly they need to be organized; put into their proper places. A constantly shifting direction undoes any of this organization, which wastes time and causes severe problems.

A large system full of useful domain functionality isn’t actually very useful if it is technologically unstable. If it crashes frequently or is prone to serious bugs because of mass redundancies or if its performance is dismal, all of that existing functionality is unaccessible. A smaller more stable system would be far more effective at helping the users. The features available should work properly, be complete and be organized.

A very clear symptom that a project is trapped in an Ebony Dungeon is that most of the decisions keep getting punted back to the domain experts. “We’ll have to ask the business what they want”. If the project is balanced then at least 50% of the effort is related to the technical side. That includes using industrial strength components and algorithms, keeping the code base clean and insuring that the operations and installations side are built up to an automated level as well. Technical debt is unavoidable, but it needs to be controlled and managed with the same importance as any other aspect of the project.

In areas like GUIs, industry conventions should trump individual's preferences, so that the screens don’t become a sea of eclectic behaviors and that the functionality isn’t randomly distributed throughout the screens. Failure to properly organize the functionality at the interface level will cause a failure to use the functionality at the user level. Proper organization of a huge number of features is an extremely difficult problem that takes decades to master. A domain expert may understand the functionality requirements, but organization is just as, if not more, critical.

Being trapped in a dungeon exerts a lot of pressure on the programmers to create code as fast as possible. This usually manifests itself as a significant amount of “copy and paste” programming. Old code is copied over and then hastily modified with a small number of differences. We’ve known that this style of development is extremely bad for decades, but it is still commonly used to satisfy time pressure. Programmers need time to understand and refactor their work if its going to be extended properly. A rushed job is a sloppy job.

An Ivory Tower system misleads its backers because the real problems don’t become apparent until the users start working with the system. An Ebony Dungeon system also misleads its backers because it starts off fast and agile but slams into a wall when the work is to be extended. What appeared to be a big success in the early days ends up dying a premature death, usually costing way more than it should.

The software industry has swung rather heavily towards Ebony Dungeons lately. It’s easy to do because domain experts over-simplify the real work involved to build a system then they get mislead by the rapid progress. Without an experienced development crew it becomes easy to miss all of the symptoms of a project in deep trouble. Most that are dying get caught up in tunnel vision, thinking that just a few more features will turn the direction around and save the project. But “just a few more...” is actually the problem.

The best way out of a dungeon is to properly partition the system requirements. Everyone talks about ‘user requirements’ but they are only a small subset of what’s needed for a successful system:

user requirements
management requirements
technical operations/release requirements
support/debugging requirements
development/programmer requirements

If you factor in all of the different requirements necessary to build and run the system, you see that ‘features’ are only one aspect of the work. If you ignore the other four (or more) categories than obviously there will be serious problems with the system. If all of the requirements come from the domain experts, since they don’t know about or understand the other issues they won’t place any importance on getting them done.

The real art in building systems is to not go too high or too low, but rather to build on solid ground that is accessible to everyone. Towers and dungeons are equally bad.

Systemic Breakdown

2013-02-23T14:49:00.001-05:00

One of my favorite parts about building software is that I am free to indulge my endless curiosity. I prefer to dig deeply into the domains I’ve tackled so that I can tailor my solutions to actual problems, not just the high-level perception of them. The world is wonderfully complex place and it can be fascinating to delve into the details and underlying history.

To really solve a person’s problem with some code running on a computer you have to understand the root causes. A 10,000 foot view is just too high to grok the details, which often become counter-intuitive as you descend. As I've switch from domain to domain, I've obviously found huge differences between the issues but I’ve also found many underlying consistent patterns.
For this post I thought I'd pull back a little and discuss one of these larger veins running through much of my work. I’ll pick a very abstract example, but only because I prefer not to antagonize both my past and future employers.

We can start with the distinction between formal and informal systems. Formal systems are such because the underlying rules are rigid. All things must follow the rules (or they are invalid and no longer within the system). A few examples are mathematics and running software programs.
Informal systems on the other hand often involve people, although not always. The rules are mostly followed, but there are always exceptions. Informal systems are not rigorously constrained. There are a huge number of examples of informal systems, but any sort of social interaction -- companies, governments, etc. -- form the basis of many of them. Basically all you need is some 'objects' (entities, nouns, etc) and a set of rules that are applied to them.

For an overly simple example I keep coming back to cars driving on a highway. The underlying road supports a number of vehicles that are attempting to get somewhere, each with a unique destination. There are mandated constraints such as following the speed limit and staying in particular lanes. Many highways these days are challenged with a lot of problems: congestion, accidents, speeding etc. They are fairly simple systems, but their behavior is fairly complex.

Because it doesn’t really matter, I’ll focus my discussion loosely around the highways in Ontario, Canada that I drive regularly. Please don’t get too lost in the specifics, I’m only using these examples to illustrate the larger patterns that are common amongst similar and very different informal systems. Cars, rules and roads just help as reference points, nothing more.

A key constraint when driving on the highway is a speed limit. The rules and limits on speeding were set long ago, most likely when cars were considerably less sophisticated. So for instance a speed limit of 100 km/h was probably created long ago as a protective measure to decrease the likelihood of serious accidents. When highways were poorly paved and cars crude, no doubt these rules made a great deal of sense and contributed significantly to the safety of the roadways. These days, for a well-maintained highway, modern cars which handled better are quite capable of navigating the same terrain at a significantly higher speeds. That is, 140 km/h for many drivers in a modern cars is well within their ability to react correctly to hazards, road conditions and other vehicles. Because of this, in Ontario at least, 'speeding' is epidemic. Everybody exceeds what seems like the relatively slow limit of 100 km/h.

The response to people increasingly speeding was the creation of traffic police whose role is to enforce the rules. We can see this as a subsystem that sprang into life because of a degeneration of drivers in obeying rules like the speed limit (although there are probably many other causes). This new subsystem quickly figured out that ticketing people, beside enforcing the rules, was extremely profitable. And even more profitable if they put less effort into the overall ‘general’ enforcement but instead choose to hang out at inconvenient locations where people tend to speed naturally. Thus ‘speed traps’ sprang up from perhaps rather noble goals, but financial incentives quickly had considerable influence. Once create the local drivers quickly figured out where they were located, so they shifted their behavior to speed only in areas that they knew were safe.

Summarizing: this informal system rule for speed limits sprang up, degenerated, was buttressed with a new subsystem of ticketing but that too degenerated. The key point here is that when we are analysing the creation of speed traps we need to go beyond just the obvious cause, that more people were violating the speed limit. Getting to the 'root' causes is extremely important in analysing informal systems. It is easy to say that speed limits and traps exist because people are speeding more often, but that's not really capturing the whole picture. Rather the deeper root cause is that modern cars started allowing people to drive at higher speeds. They handle better, have bigger engines and can accelerate rapidly. The root cause for speeding is an advance in technology and that technology was getting applied by a large number of drivers to violate a pre-existing set of rules, which is a downstream consequence.

What's particularly true of informal systems is that their interactions are extremely complex. To fix the speeding problem by just creating traffic police is not enough, and because it didn't really fix the problem the traffic enforcement directives gravitated towards finding their own usefulness (generating money). Thus adding speeding tickets to the overall system did not really improve it in any significant way. Some people probably speed less, but most people are likely to just be more picky about where they are speeding. At various times and locations speeding is reduced, but many drivers are probably over-compensating now for the added inconvenience. People aren't going to stop speeding for instance, and their expectation for travel time isn’t going to change.

A solution like raising the speed limit wouldn't fix the issue either. The newer cars may allow good drivers to go faster, but highways contain more than just good drivers. They contain many poor ones, without the ability to safely handle higher speeds. It's perhaps this constraint that keeps the authorities from changing the rules. A faster highway might be significantly more dangerous if there are enough drivers on the road that can't react fast enough. So the average or less-skilled driver keeps the status quo intact, but as in the case of Toronto people simply moved forward and start creating their own rules of thumb for how fast to drive. Many in TO believe for example that +10 km/h for the city and +20 km/h for the highway are acceptable. This is so common that in the absence of any traffic police nearby, this is frequently the pace of most vehicles. Thus a breakdown in the informal system causes a creation of another even less formal system built on top.
Informal systems based on people always morph as time progresses. Some systems such as the highways find their own equilibriums by spawning off new subsystems. Some systems fluctuate, but generally drive themselves downward. We know how to make things more complex, but we seem to lack in the ability to reduce complexity. A degenerating informal system can often acquire enough unwarranted artificial complexity that its only path forward is to explode. The issues simply can't be fixed anymore.

The idea of speeding tickets was no doubt someone’s best effort to deal with the speeding problem. But as is true for all complex informal systems, the whole systemic complexity exceeds the ability of most, if not all, people to visualize and correctly modify it. Speeding tickets created speed traps. Speed traps are a rather grey aberration of the laws of highways, since they aren't helping to keep locals from speeding, but rather they are preying on unsuspecting strangers. This causes a hierarchy of drivers, creating a division between 'us' and 'them'. In that sense it is morally questionable, since a civilized society these days demands that the rules are applied equally to all people, not just some. A speed trap functions because it is based on ignorance. Non-local drivers following the normal rules of thumbs are caught simply because they strayed too far from their own locale. Stated that way, the enforcement part of society is actually preying on people for revenue, not to uphold the honor and virtue of the rules.

If technology got traffic into trouble, it is likely that it will also get it out of it. Automatic driving may allow cars to chain together into larger loosely affiliated trains headed into common directions. If that comes to pass cars will scout for a suitable train, then join up. As that becomes more common, the efficiencies of the highway -- which are poor now and getting worse -- will start to improve creating less traffic congestion. These trains of cars will follow limits set by computers, so speeding tickets and speed traps will become a thing of the past. Of course some people will continue to 'manually' drive, but official enforcement of any remaining rules will likely shift to some other basis, like asserting that speeds of 140 km/h are 'reckless' rather than 'speeding'. The rule remains, but gets generalized to include a wider collection of behaviors.

Informal systems are everywhere. They form the basis of all human organizations and they fracture into an endless variety of associated subsystems. These days computers have made it possible to quickly introduce new poorly thought-out rules and enforce them, so the underlying complexity has exploded. Most large companies, for instance, are so outrageously complicated that no single human could hope to know or understand the basis behind even a fraction of the rules.Thanks to badly designed software and archaic historic rules many of these are in various stages of systemic breakdown, with rules degenerating and inefficient subsystems spewing off everywhere to replace high-level but hopeless directives. Computers have the capacity to solve problems for people, but their dark side is that they can allow for massive increases in artificial complex that can quickly get gouged into the status quo. Ambitious, but misguided people tunnel onto little aspects of the overall problem and seek change, but once the system has gone beyond a human's ability to comprehend, most such changes ultimately make the problems worse not better.

Getting back to traffic, if someone really wanted to stop speeding and installed a massive number of high-tech cameras to cover every inch of the road, they might think that they'd be able to stop anyone, anywhere, from speeding. But sorting through that footage would be daunting, and just creating the infrastructure to fine and collect all of the tickets would be massive. So, ultimately there would still be blind spots, and over time these would become known. People, hoping to reduce their travel time would fly recklessly through the blind spots because they were forced to slow down once in the spotlight. Then some areas would become safer for driving, but some, well... they'd just push the envelop for danger until they too caused various spin offs. To fix complex system breakdowns the one thing that definitely doesn't work is to narrow the focus down to something believed to be 'tangible'. That type of tunnel vision only begs for disaster.

Quality and Scale

2013-01-19T15:54:00.001-05:00

[LACK OF EDITOR'S NOTE: I wrote and posted this entirely on my iPad, so there are bound to be spelling and formatting problems. Please feel free to point them out, I'll fix them as I go.]

I use a couple of metrics to size development work. On one axes I always consider the underlying quality of the expected work. It's important because quality sits on at least in a logarithmic space. That is small improvements in quality get considerable more expensive as we try for better systems. On the other axes I use scale. Scale in also at least logrithmic in effort.
My four basic catigories for both are:

Software Quality
- Prototype
- Demo
- In-house
- Commercial

Project Scale
- Small
- Medium
- Large
- Huge

Software Quality

The first level of quality are prototypes. They are generally very specific and crude programs that show a proof of concept. There is almost no error handling, and no packaging. Sometimes these are built to test out new technologies or algorithms, sometimes they are just people playing with the coding environment. Proototypes are imortant for reducing risk, they allow experience to be gained before commiting to a serious development effort.

Demos usually bring a number of different things together into one package. They are not full solutions, rather they just focus on 'talking points'. Demos also lack reasonable error handlng and packaging, but they usually show the essence of how the software will be used. Occasionally you see someone release one as a commercial product, but this type of premature exposure comes with a strong likelihood of turning off potential users.

My suspicion is that in-house software accounts for most of modern software development. What really identifies a system as in-house is that it is quirky. The interface is a dumping ground for random disconnected functionality, the system looks ugly and it's hard to navigate around. Often, but not always, the internal code is as quirky as the external appearance. There is little architecture, plenty of duplication and usually some very strange solutions to very common problems. Often the systems have haphazaardly evolved into their present state. The systems get used but its more often because the audience is captive, given a choise they'd prefer a more usable product. Many enterprise commercial systems are really in-house quality. Sometime they started out higher, but have gradually degenerated to this level after years of people just dumping code into them. Pretty much an unfocused development project is constrained by this level. It takes considerable experience, talent, time and focus to lift the bar.

What really defines commercial quality is that the users couldn't imagine life without the software. Not only does it look good, it's stunningly reliable, simple and intuitive to use. Internally the code is also clean and well organized. It handles all errors correctly, is well packaged and requires minimal or no support. Graphic designs and UX experts have heavily contributed to give the solution both a clear narrative and a simple, but easily understood philosopy. A really great example really does solve all of the problems that it claims to. Even the smallest detail has been though-out with great care. The true mastery of programming comes from making a hard problem look simple; commercial systems require this both to maintain their quality and to support future extensions. Lots of programmers claim commercial quality programming abilities, but judging from our industry very few can actually operate at this level. As user's expectation for scale and shortened development times have skyrocketed, the overall quality of software has been declining. Bugs that in the past would have caused a revolt or lawsuit are now convientently overlooked. This may lead to more tools available, but it also means more time wasted and more confusion.

Project Scale

It is imossible to discuss project scale without resorting to construction analogies. I know programmers hate this, but without some tangilble basis, people easily focus on the wrong aspects or oversimplify their analysis. Grounding the discusion with links to physical reality really helps to visualise the work involved and the types of experience necessary to do it correctly.
A small project is akin to building a shed out back. It takes some skill, but it is easily learned and if there are the inevitable problems, they are relative well contained. A slightly wonky shack still works as a shack, it may not look pretty but hey, it's only a shack. Small projects generally come in around less than 20,000 lines of code. It's the type of work that can be completed in days, weeks or a few months by one or two programmers.

A medium project is essentially a house. There is a great deal of skill in building a house; it's not an easy job to complete. A well-built house is impressive, but somewhere in the middle of the scale it's hard for someone living in the house to really get a sense of the quality. If it works well enough, then it works. Medium projects vary somewhat, falling in around 50,000 lines of code and are generally less than 100,000. Medium projects usually require a team, or thet are spread across a large number of years.

You can't just pile a bunch of houses on top of each other to get an apartment building. Houses are made of smaller, lighter materials and are really scaled towards a family. Building an apartment building on the other hand, requires a whole new set of skills. Suddenly things like steel frames become necessary to get the size beyond a few floors. Plumbing and electricty are different. Elevators become important. The game changes, and the attention to detail changes as well. A small flaw in a house, might be a serious problem in an apartment building. As such more planning is required, there are fewer options and bigger groups need to be involved, often with specializations. For software, large generally starts somewhere after 100,000 lines but can also get triggered by difficult performance constraints or more than a trvial number of users. In a sense it's going from a single family dwelling into a system that can accomodate significatly larger groups. That leap upwards in complexity is dangerous. Crossing the line may not look like that much more work, but underneath the rules have changed. It's easy to miss, sending the whole project into what is basically a brick wall.

Skyscapers are marvels of modern engineering. They seem to go up quickly, so it's easy to miss their stunning degree of sophistication, but they are complex beasts. It's impressive how huge teams come together and managed to stay organized enough to achieve these monuments. These days, there are many similar examples within the software world. Sytems that run across tens of thousands of machines or cope with millions of users. There isn't that much in common between an apartment building and a skyscaper, although the lines may be somewhat blurred. It's another step in sophistication.

Besides visualization, I like these categories because it's easy to see that they are somewhat independent. That is, just because someone can build a house doesn't mean they can build a skyscaper. Each step upwards requires new sets of skills and more attention to the detail. Each step requires more organization and more manpower. It wouldn't make sense to hire an expert from one category and expect them to do a good job in another. An expert house-builder can't necessarily build a skyscraper and a skyscraper engineer may tragically over-engineer a simple shed. People can move of course, but only if they put aside their hubris and accept that they are entering a new area. This -- I've often seen -- is true as well for software. Each bump up in scale has its own challenges and its own skills.

Finally

A software project has an inherent scale and some type of quality goals. Combining these gives a very reliable way of sizing the work. Factors like the environment and experience of the developers also come to play, but scale and quality dominate. If you want a commercial grade skyscaper for instance, it is going to be hundreds of man-years of effort. It's too easy to dream big, but there are always physical constraints at work, and as Brookes pointed out so very long ago, there are no silver bullets.

Potential

2013-01-06T13:51:00.001-05:00

Computers are incredibly powerful. Sure they are just stupid machines, but they are embodied with infinite patience and unbelievable precision. But so far we’ve barely tapped their potential, we’re still mired in building up semi-intelligent instruction sets by brute force. Someday however, we’ll get beyond that and finally be able to utilize these machines to improve both our lives and our understanding of the universe.

What we are fighting with now is our inability to bring together massive sets of intelligent instructions. We certainly build larger software systems now then in the past, but we still do this by crudely mashing together individual efforts into loosely related collections of ‘functionality’. We are still extremely dependent on keeping the work separated, e.g. apps, modules, libraries, etc. These are all works of a small groups or individuals. We have no real reliable ways of combining the effort from thousands or millions of people into focused coherent works. There are some ‘close but no cigar’ examples, such as the Internet or sites like Wikipedia where they are a collection from a large number of people, but these have heavily relied on being loosely organized and as such they fall short of the full potential of what could be achieved.

If we take the perspective of software being a form of ‘encoded’ intelligence, then it’s not hard to imagine what could be created if we could merge the collective knowledge of thousands of people together into a single source. In a sense, we know that individual intelligence ranges; that is some people operate really smartly, some do not. But even the brightest of our species isn’t consistently intelligent about all aspects of their life. We all have our stupid moments where we’re not doing things to our best advantage. Instead we’re stumbling around, often just one small step ahead of calamity. In that sense ‘intelligence’ isn’t really about what we are thinking internally, but rather about how we are applying our internal models to the world around us. If you really understood the full consequences of your own actions for instance, then you would probably alter them to make your life better...

If we could combine most of what we collectively know as a species, we’d come to a radically different perspective of our societies. And if we used this ‘greater truth’ constructively we’d be able to fix problems that have long plagued our organizations. So it’s the potential to utilize this superior collective intelligence that I see when I play with computers. We take what we know, what we think we know, and what we assume for as many people as possible, then compile this together into massive unified models of our world. With this information -- a degree of encoded intelligence that far exceeds our own individual intelligence -- we apply it back, making changes that we know for sure will improve our world, not just ones based on wild guesses, hunches or egos.

Keep in mind that this isn’t artificial intelligence in the classic sense. Rather it is a knowledge-base built around our shared understandings. It isn’t sentient or moody, or even interested in plotting our destruction, but instead it is just a massive resource that simplifies our ability to comprehend huge multi-dimensional problems that exceeds the physical limitations of our own biology. We can still choose to apply this higher intelligence at our own discretion. The only difference is that we’ve finally given our species the ability to understand things beyond their own capabilities. We’ve transcended our biological limitations.

State of the New Machines

2012-12-28T19:46:00.001-05:00

I'm typing in this post from a cottage in northern Ontario. This is my first post on my iPad, connect via 3G to the world. I couldn't imagine 20 years ago, while lugging home my 30 pound Compaq "portable", that someday I'd be miles off the grid, yet connect to the world via a wafer thin screen resting on my laptop. Hardware, it seems, has made amazing progress.

The irony is probably that my tablet is packed with retro apps, most of which are throw backs to the early days of PC computing. While hardware has screamed ahead, software has, well, lagged. Still, there are a few apps that impress me, mostly the stuff coming from Autodesk. The rest however...

Theory and Practice

2012-11-25T21:14:00.002-05:00

Nearly three decades ago, when I started university all I really wanted to learn was the magic of programming. But my course load included plenty of mathematics and computer theory courses, as well as crazy electives. “What does all this have to do with programming?” I often complained. At first I just wished they’d drop the courses from the curriculum and give me more intensive programming assignments. That’s what I thought I needed to know. In time I realized that most of it was quite useful.

Theory is the backbone of software development work. For a lot of programming tasks you can ignore the theory and just scratch out your own eclectic way of handling the problem, but a strong theoretical background not only makes the work easier it also is more likely to withstand the rigors of the real world. Too often I’ve seen programmers roll their own dysfunctional code to a theoretical problem without first getting a true appreciation of the underlying knowledge. What most often happens is that they flail away at the code, unable to get it to be stable enough to work. If they understood the theory however, not only is the code shorter, but they’d spend way less time banging at it. It makes it easier. Thus for some types of programming, understanding the underlying theory is mandatory. Yes, it’s a small minority of the time, but it’s often the core of the system, where even littlest of problems can be hugely time intensive.

The best known theoretical problem is the ‘halting problem’. Loosely stated, it is impossible to write some code that can determine if some other code will converge on an answer or run forever (however one can write an estimation that works with a finite subset within a Turing Machine and that seems doable).

In its native form the halting problem isn’t crossed often in practice, but we do see it in other ways. First is that an unbounded loop could run forever. An unbounded recursion can run forever as well. Thus in practice we really don’t want code that is ever unbounded -- infinite loops annoy users and waste resources -- at some point the code has to process a finite set of discrete objects and then terminate. If that isn’t possible, then some protective form of constraint is necessary (although the size should be easily configurable at operational time).

The second way we see it is that we can’t always write code to understand what code is trying to do. In an offbeat way, that limits the types of tools we can use in automation. It would be nice for instance if we could write something that would list out the stack for all possible exceptions in the code with respect to input, but that would require the lister to ‘intelligently’ understand the code enough to know the behavior. We could approx that, but the lack of accuracy might negate the value of the tool.

Another interesting theoretical problem is the Two Generals Problem. This is really just a coordination issue between any two independent entities (computers, threads, processes, etc.). There is no known way to reliability get 100% communication if the entities are independent. You can reduce the window of problems down to a tiny number of instructions, but you can never remove it entirely. With modern computers we can do billions of things within fractions of a second, so even a tiny 2 ms window could result in bugs occurring monthly in a system with a massive number of transactions. Thus what seems like an unlikely occurrence can often turn into a recurring nuisance that irritates everyone.

Locking is closely related to the Two Generals Problem. I’ve seen more bugs in locking than in any other area of modern programming (dangling pointers in C were extremely common in the mid 90s but modern languages mitigated that). It’s not hard to write code to lock resources, but it is very easy to get it wrong. At its heart, it really falls back to a simple principle: to get reliable locking you need a ‘test-and-set’ primitive. That is, in one single uninterrupted single-threaded protected operation, you need to test a variable and set it to ‘taken’ or return that is it unavailable. Once you have that primitive, you can build all other locking mechanisms on top of it. If it’s not atomic however, there will always be a window of failure. That links back to the Two Generals Problem quite nicely, since where it becomes an issue is when you can’t have access to an atomic ‘test-and-set’ primitive (and thus there will always be problems).

Parsing is one of those areas where people often tread carelessly without a theoretical background, and it always ends badly. If you understand the theory and have read works like The Red Dragon Book then belting out a parser is basically a time problem. You just decide what the ‘language’ requires such as LR(1), and how big the language is and then you do the appropriate work, which more often than not is either a recursive descent parser or a table driven one (using tools like lex/yacc or antlr). There are messy bits of course, particularly if you are trying to draft your own new language, but the space is well explored and well documented. In practice however what you see is a lot of crude split/join based top-down disasters, with the occasional regular expression disaster thrown in for fun. Both of those techniques can work with really simple grammars, but then fail miserably when applied to more complex ones. Thus being able to parse a CSV file, does mean you know how to parse something more complex. Bad parsing usually is a huge time sink, and if it’s way off then the only reasonable option is to rewrite it properly. Sometimes it’s just not fixable.

One of my favorite theoretical problems is the rather well-known P vs NP problem. While the verdict is still outstanding on the relationship, it has a huge implication for code optimizations. For people unfamiliar with ‘complexity’, it is really a question of growth. If you have an algorithm that takes 3 seconds to run with 200 inputs, what happens when you give it 400 inputs? With a simple linear algorithm it takes 6 seconds to run. Some algorithms perform worse, so they may take 9 secs (3^2 -- three squared) to run, or even 64 seconds (4^3 -- four to the power of three). We can take any algorithm and calculate its ‘computational complexity’ which will tell us exactly how the time grows with respect to the size of the input. We usually categorize this by the dominant operators so O(1) is a constant growth, O(n) is growing linearly by the size of the input, O(n^c) is growing by a constant exponent (polynomial time) and O(c^n) has the size of the input as the exponent (exponential time). The P in the equation is a reference to polynomial time, while NP is rather loosely any growth such as exponential that is larger (I know, that is a gross oversimplification of NP, but it serves well enough to explain that it references problems that are larger, without getting into what constrains NP itself).

Growth is a really important factor when it comes to designing systems that run efficiently. Ultimately what we’d like is to build is a well-behaved system that runs in testing on a subset of the data, and then to know when it goes into production that the performance characteristics have not changed. The system shouldn’t suddenly grind to a halt when it is being accessed by a real number of users, with a real amount of data. What we’ve learned over the years is that it is really easy to write code where this will happen, so often to get the big industrial stuff working, we have to spend a significant amount of time optimizing the code to perform properly. The work a system has to do is fixed, so the best we can do is find approaches to preserve and reuse the work (memoization) as much as possible. Optimizing code, after its been shown to work, is often crucial to achieving the requirements.

What P != NP is really saying in practice is that there is a very strong bound on just exactly how optimized the code can really be. If it’s not true then there would be no possible way you could take an exponential problem and find clever tricks to get it to run in polynomial time. You can always optimize code, but there might be a physical bound on exactly how fast you can get it. A lot of this work was best explored with respect to sorting and searching, but for large systems it is essential to really understand it if you are going to get good results.

if it were true however, amongst many other implications, that would mean that we are able to calculate some pretty incredible stuff. Moore’s law has always been giving us more hardware to play with, but users have kept pace and are continually asking for processing beyond our current limits. Without that fixed boundary as a limitation, we could write systems that make our modern behemoth's look crude and flaky, and it would require a tiny fraction of the huge effort we put in right now to build them (also it would take a lot of fun out of mathematics according to Gödel).

Memoization as a technique is best known from ‘caching’. Somewhere along the way, caching became the over-popular silver bullet for all performance problems. Caching in essence is simple, but there is significant more depth there than most people realize, and as such it is not uncommon to see systems that are deploying erratic caching to harmful effect. Instead of magically fixing the performance problems, they manage to make them worse and provide a slew of inconsistencies in the results. So you get really stale data, or a collection of data with parts out of sync, slower performance, rampant memory leaks, or just sudden scary freezes in the code that seem unexplainable. Caching, like memory management, threads and pointers is one of those places where ignoring the underlying known concepts is most likely to result in pain, rather than a successful piece of code.

I’m sure there are plenty of other examples. Often when I split programming between ‘systems programming’ and ‘applications programming’ what I am really referring too is that the systems variety requires a decent understanding of the underlying theories. Applications programming needs an understanding of the domain problems, but they can often be documented and passed on to the programmer. For the systems work, the programmer has to really understand what they are writing, for if they don’t, the chances of just randomly striking it lucky and getting the code to work are are nearly infinitesimal. Thus, as I found out over the years, all of those early theory courses that they made me take are actually crucial to being able to build big industrial strength systems. You can always build on someone else’s knowledge, which is fine, but if you dare tread into any deep work, then you need to take it very seriously and do the appropriate homework. I’ve seen a lot of programmers fail to grok that and suffer horribly for their hubris.

Best Practices

2012-11-18T09:57:00.002-05:00

One significant problem in software development is not being able to end an argument by pointing to an official reference. Veteran developers acquire considerable knowledge about ‘best practices’ in their careers, but there is no authoritative source for all of this learning. There is no way to know whether a style, technique, approach, algorithm, etc. is well-known, or just a quirk of a very small number of programmers.

I have heard a wide range of different things referred to as best practices, so it’s not unusual to have someone claim that their eclectic practice is more widely adapted than it is. In a sense there is no ‘normal’ in programming, there is such a wide diversification of knowledge and approaches, but there are clearly ways of working that consistently produce better results. Over time we should be converging on a stronger understanding, rather than just continually retrying every possible permutation.

Our not having a standard base of knowledge makes it easier for people from outside the industry to make “claims” of understanding how to develop software. If for instance you can’t point to a reference that says there should be separate development, test and production environments, then it is really hard to talk people out of just using one environment and hacking at it directly. A newbie manager can easily dismiss 3 environments as being too costly and there is no way to convince them otherwise. No doubt it is possible get do everything all on the same machine, it’s just that the chaos is going to extract a serious toll in time and quality, but to people unfamiliar with software development issues like ‘quality’ find that they are not easily digestible.

Another example is that I’ve seen essentials like source code control set up in all manner of weird arrangements, yet most of these variations provide ‘less’ support than the technology can really offer. A well-organized repository not only helps synchronise multiple people, but it also provides insurance for existing releases. Replicating a bug in development is a huge step in being able to fix it, and basing that work on the certainty that the source code is identical between the different environments is crucial.

Schemas in relational databases are another classic area where people easily and often deviate from reasonable usage, and either claim their missteps as known or dismiss the idea that there is only a small window of reasonable ways to set up databases. If you use an RDBMS correctly it is a strong, stable technology. If you don’t, then it becomes a black hole of problems. A normalized schema is easily sharable between different systems, while a quirky one is implicitly tied to a very specific code base. It makes little sense to utilize a sharable resource in a way that isn’t sharable.

Documentation and design are two other areas where people often have very eclectic practices. Given the increasing time-pressures of the industry, there is a wide range of approaches happening out there that swing from ‘none’ to ‘way over the top’, with a lot of developers believing that one extreme or the other is best. Neither too much or too little documentation serves the development, and often documentation isn’t really the end-product, but just necessary steps in a long chain of work that eventually culminates in a version of the system. A complete lack of design is a reliable way to create a ball of mud, but overdoing it can burn resources and lead to serious over-engineering.

Extreme positions are common elsewhere in software as well. I’ve always figured that in their zeal to over-simplify, many people have settled on their own unique minimal subset of black and white rules, but often the underlying problems are really trade-offs that require subtle balancing instead. I’ll often see people crediting K.I.S.S (keep it simple stupid) as the basis for some over-the-top complexity that is clearly counter-productive. They become so focused on simplifying some small aspect of the problem that they lose sight that they’ve made everything else worse.

Since I’ve moved around a lot I’ve encountered a great variety of good and insane opinions about software development. I think it would be helpful if we could consolidate the best of the good ones into some single point of reference. A book would be best, but a wiki might serve better. One single point of reference that can be quoted as needed. No doubt there will be some contradictions, but we should be able to categorize the different practices by family and history.

We do have to be concerned that software development is often hostage to what amounts to pop culture these days. New “trendy” ideas get injected, and it often takes time before people realize that they are essentially defective. My favorite example was Hungarian notation, which has hopefully vanished from most work by now. We need to distinguish between established best practices and upcoming ‘popular’ practices. The former have been around for a long time and have earned their respect. The latter may make it to ‘best’ someday, but they’re still so young that it is really hard to tell yet (and I think more of these new practices are deemed ineffective then promoted to ‘best’ status).

What would definitely help in software development is to be able to sit down with management or rogue programmers and be able to stop a wayward discussion early with a statement like “storing all of the fields in the database as text blobs is not considered by X to be a best practice..., so we’re not going to continue doing it that way”. With that ability, we’d at least be able to look at a code base or an existing project and get some idea of conformity. I would not expect everyone to build things the same way, but rather this would show up projects that deviated way too far to the extremes (and because of that are very likely to fail). After decades, I think it’s time to bring more of what we know together into a usable reference.

Work Smarter...Not Harder

2012-11-04T13:41:00.003-05:00

I’ve always loved this quote:

"Work Smarter...Not Harder"
Allan F. Mogensen

But what does it really mean, particularly when it comes to software development?

get organized immediately and cleanup often
being consistent is more important than being right
be very sensitive to scale (bigger systems mean less shortcuts)
utilize the full abilities of any technology or tools, but don’t abuse them
automate as much as possible, spend the time to get it reliable
minimize dependencies, accept them only when there are really no other options
read the manual first, even if it is boring
research before you write code, avoid flailing at a problem, seek expertise
choose to refactor first, before coding
delete as much code as possible
encapsulate the subproblems away from the system, spend the time to get it right
break the problem cleanly, fear special cases
apply abstraction and generalization to further reduce the work
think very hard about the consequences before diving in
fail, but admit it and learn from it
don’t be afraid of the domain, learn as much as possible about it
focus on the details, quality is all about getting them right
accept that complexity breeds below the surface, if you think something is simple then you probably don’t understand it yet
know the difference between knowing, assuming and guessing
everything matters, nothing should be random in software
don’t ignore Murphy’s law
a small number of disorganized things doesn’t appear disorganized until it grows larger
reassess the organization as things grow, updating it frequently as needed

Working smarter is most often about spending more time to think your way through the problems first, before diving in. Under intense time pressure, people often rush to action. There are times when this is necessary, but there is always a cost. Rack up enough of this technical debt. and then just servicing it becomes the main priority, which only amplifies the time pressure. Thus any gains from swift action are lost and working harder won’t undo the downward spiral.

Being smart however can prevent this from occurring. Yes, the pace is slower and getting the details right always requires more effort, but a minimal technical debt. means more resources are available to move forward. Eventually it pays off. Being smart isn’t just about thinking hard, it also requires having enough data to insure that the thinking is accurate. Thus, acquiring more information -- dealing with the details -- is the key to being able to amplify one’s thinking. In software development, what you don’t understand can really harm you and it’s very rare that you can ignore it.

Setting Up Development

2012-10-27T15:19:00.002-04:00

While it is not always true, you’ll frequently find that in restaurants there is a reasonable correlation between the quality of the food and the management of the kitchen. In my youth I had always found that a messy kitchen often meant low quality food and/or bad service. Of course it’s not always that simple, but usually a disaster area behind the curtains percolated out to the rest of the organization. Since I prefer to ‘do things well if I have to do them’ when searching for jobs back then, I stayed far away from disorganized, sloppy or dirty restaurants, even if it was well-hidden behind the scene.

The same is essentially true in software development. Projects that are set up correctly have strong processes, gravitate towards best practices and control the client’s expectations are considerably more likely to produce quality software. The correlation isn’t always as strong though, since I have seen smooth development shops that still produce fugly software. But I’ve never seen a badly organized shop produce excellence.

One far too common problem I’ve seen is that the client expectations are out of whack. In the printing industry people often say “Fast, Cheap or Quality, pick two” as the means to explain the types of trade-offs necessary when purchasing print work. In software we get the same saying but it ends with “pick one”. I figure it’s worse for our industry because our work is considerably longer. Print jobs are often measured in hours, days or perhaps weeks, while software is usually measured in weeks, months and years. I’ve seen far too many software projects where clients insist on all three and then have a hard time understanding why some “fiddling” with a few screens might require more than a trivial effort.

A development project that is set up well not only attends to the implementation and distribution of software, but it also must control the client’s expectations. They need to be reasonable. Failure to do this means too much pressure on features and quick fixes, which means accumulating a heavy technical debt. Left unchecked, the project spirals downwards until it combusts.

You can see the symptoms of that type of pressure directly in the code base. Poor analysis, little design, poor use of tools and overly frequent ‘micro’ releases. As well, there is generally a high bug count and a lot of time burned on ‘second-line support’. If developers are busy with band-aides then clearly they’re not coding new stuff. Too much support and too little features just amplifies the pressure, which eventually becomes so toxic that big things start to break.

Getting out of this type of spiral can be difficult or impossible. Resetting the expectations of people who are already disappointed is an extremely tough task. However, just like those reality TV shows where they help people in serious financial debt learn about handling their finances (like Til Debt Do U$ Part) in a more reasonable manner, the ticket to really fixing the development problems is to tackle the technical debt, which mostly means being cost conscious (features & make-work) and reducing lifestyle expectations (release dates). Digging out takes a massive amount of time and means a lot less progress, but it only gets worse if it is put off for longer.

Reversing the slide means revisiting the development and release procedures. It also means lots of cleanup and refactoring. And it usually means revisiting all of the development tools to upgrade their usage to known best practices. For confidence, a few low hanging features might be tossed in as well, but all major changes need to be put on hold until the implementation and release process stabilizes. A smooth development effort should produce minimal support effort. It should be reliable and repeatable. The code should be clean and consistent.

Getting back to the kitchen in a messed up restaurant, not only do you need to clean up the work area and replace any defective equipment, but you also need to set some processes in place to insure that it doesn’t happen again. Once that back-room problem has been dealt with, you can consider a new strategy and/or a new menu, but neither of those will help if the kitchen lacks the ability to execute.

You’d think that new projects were less vulnerable to this type of problem, but most often the pressure to get something working right away overrides the desire to set things up properly. The developers start letting little issues slip and then after enough time, they get hit with ‘death by a thousand cuts’. The most visible “source” of the problem is the client expectations, but they only become overwhelming because of a lack of management. Still it is ultimately up to the developers to ‘insist’ that the technical debt not be ignored. At some point it will always be dangerously large, so at very least ‘cleanup’ should be a recurring work item. Personally I like to start each new development iteration with some significant cleanup, mostly because its boring and it gives one a chance to take a breather after the last release. If that becomes a steady habit, while there is always technical debt and time pressure, at least it never builds up to toxic levels.

In the end its not just what you code, but how you code it and how you leverage the development and systems tools as well. Digging out of technical debt is an on-going problem and mostly the fixed time allocated to do the work should be relative to the size of the debt. It doesn’t need to happen in one big effort, but it can’t be ignored indefinitely either. A smooth development process means that the developers can spend more time contemplating the behavior of the code, which is really the only route to quality. A disorganized process means a strong head wind which is only overcome by dangerous short-cuts. Thus smooth is good and it is a primary requirement to achieving quality. You can’t have the later without spending time to get the former.

Programming Style

2012-10-15T14:30:00.002-04:00

Does it really matter what the code looks like? The short answer is an emphatic ‘yes’.

It matters because ultimately writing code is about discipline and details. If the code is just slapped together, really messy and there are issues like extra blank lines or extra variables laying all over, it reveals an awful lot about the author. Perhaps they were in a huge rush or even a panic. Maybe they are just lazy or don’t care. Maybe they really don’t understand what they are doing. Any which way, their state-of-mind while doing the work was such that they lacked the self-discipline to clean up their work and thus were probably not particularly good with playing attention to the all-important details either. Messy code is almost always fragile and usually a bug fest.

Code is basically knowledge and intelligence encoded into a sequence of steps for the computer to execute. It is only as smart as the programmer that wrote it, and it is only as flexible as they conceived of the runtime environment. If the programmer’s style if haphazard or their thinking muddy, then the code will suffer the same faults. If however, the programmer’s understanding is crystal clear and they grok the environment, then the code will stand up to a lot and keep on working.

Just getting something to barely work for a extremely limited set of circumstances is not good enough to be considered professional quality. Working code has to work correctly for all circumstances; for all inputs; for all changes in the environment. It needs to be able to stand up to the full range of things that happen in the real world. If it doesn’t, that’s an indication of a lack of quality.

What experienced professional programmers always come to understand is that quality starts in the code. If you want a beautiful system that is a joy for the users, then the code underneath needs to not only be smart and well designed, but it also needs to look marvelous as well. The ‘platinum’ standard of coding comes from taking a complex problem and making the code look simple. Elegance comes from both the readability and the underlying skill used to make the answer look obvious. A well-written piece of code hides the messiness of the details carefully. It misleads a reader into thinking it took just a fraction of the time to get written. It leverages abstraction to avoid redundancies, but not at the cost of becoming obfuscated. It basically takes the knowledge of the programmer and encapsulates that into a small, tight region of the system that works correctly and makes it easy to extend later. It draws clean, hard lines through the problem and then ties them up with the lines in the solution. And because it is easy to see its sole purpose, it makes it really easy to detect and fix bugs. Quality software comes from quality code.

But it is not just quality that is affected by bad code. Testing, bug fixes and support are all heavy drains on software development. Work spent in these areas is work that is no longer available for new development. Crappy-code bugs are way harder to find and fix, so as a consequence they chew through more resources. It actually costs more to write bad code than doing a good job the first time around, line for line.

And often projects get caught up in a vicious cycle of coding fast and recklessly then loosing too much time dealing with the inevitable problems, causing them to go right back to fast and reckless again. That type of cycle feeds back into morale, and the whole project does a slow spiral down the drain.

As for programming style, mostly it doesn’t really matter which conventions are chosen. Some conventions do make improvements in clarity or readability, and help to make inconsistencies more obvious, but so long as all of the code is consistent, it can always be refactored to a better convention at some point.

Getting all of the variable names the same, laying out the lines with the same whitespace formatting and finding strong conventions for naming core parts like methods and objects establishes a strong foundation for the code. Normalization and abstraction help to either hide the details or put them forth in a way that inconsistencies are noticed easily. A solid architecture decomposes the system in a way that makes it easy to trace bugs quickly back to isolated parts of the code base. All of these things help in reducing the number of bugs, and also reducing the time it takes to fix them. And because of them, it opens up more time and resources for new development.

Building a high quality software system always means playing close attention to the millions of details floating about the project. It is not enough to just hack through the work, it is a detail-oriented process, which often needs great time and patience. All development projects come with outrageous demands and too little time, but being sloppy doesn’t help with these issues. Part of the great skill involved in being a professional programmer is the self-discipline to do the right thing, even under intense pressure. Sloppiness and short-cuts, while tempting are just putting off today, what will always cost more in the future. If the goal is quality, then that has to percolate into every tiny detail. Fast and cheap is always crap in software.

Knowledge Bases

2012-10-10T15:11:00.001-04:00

To me ‘intelligence’ is basically raw human thinking power. It’s our ability to work through problems. In order to employ intelligence effectively in our world, we need data. And that data needs to be structured and interconnected to make it usable to us. Usable, organized data is what I take to be ‘knowledge’. Its all ready to be utilized.

Our various endeavors are categorized into a huge number of fields (or domains) such as law, medicine, finance, physics, math, biology, etc. each of which is mostly an independent ‘base’ of knowledge about the field. What’s really interesting about most knowledge bases is that they are easy to over-simplify and also frequently counter-intuitive in their depths. An outside perspective can easily lead to the wrong conclusions. It’s not until you are steeped in the details, that you’re able to apply your understanding correctly.

Without enough underlying knowledge, intelligence is just hand waving. You can think about something as deep and as long as you want, but your conclusions are unlikely to be reasonable.

Software development is extremely complex. Not only do you need a strong understanding of the underlying technologies and how to use them, but it also cuts across many other knowledge bases.

A software system is a solution to a problem in a user domain. If it doesn’t solve their problems, then it is only adding to them. It consists of a huge number of details, all of which need to be organized and fit into the final work. Managing these pieces, usually under tight time-lines requires a special set of skills. Software is also very expensive to build and maintain, so finding the money to pay for enough resources is always tricky.

Putting this together we see that software spans the following knowledge bases:

Technology
User domains (all?)
Management
Business

Most programmers tend towards believing that technology is the most significant issue in software development, but usually the problems start in other areas and feed back into the development work. For this post, I’ll go through all four areas in the order that I find have the most impact for big projects.

Management

A large software project has millions of details that are all being juggled at the same time. Keeping most of those balls in the air, without losing track of them, is necessary to deliver a working system.

A good manager is constantly running around, making sure that all of their resources are moving forward, and that any roadblocks are eliminated as swiftly as possible. Some people see software management as a higher creative input role where they just have to inject ideas, but that’s never actually helpful. Most development projects have more than enough ideas, what they need is organization, process and to keep everyone on the same page. In that sense a good manager accepts their role as herding all of the cats in the same direction, while making sure the path forward is clear.

Development meetings are a good place to assess management. If they are retrospective and basically just a forum for people to catch up with what has already happened, then the project is not well-organized. It shouldn’t be necessary for the manager to catch up, if they have been paying attention as they should.

If, however, meetings are really planning sessions for the next upcoming work, then they contribute by identifying the issues long before they become significant enough to derail the process. Plans don’t always work as expected, but planning is a necessity when you are dealing with long running work. Even small tasks in development can take weeks or months, and the worst possible outcome is to keep shifting directions before any of the work is completed properly. What’s started should be finished, or it shouldn’t have been started in the first place. Only a long-term plan will avoid time-crushing ‘make-work’.

Since development flows through different phases, the management of each has to change as well. A basic development iteration has five parts:

Analysis
Design
Implementation
Testing
Distribution

Analysis for example is a never-ending effort about depth. Wherever you skimp on analyzing the requirements, it will come back to haunt you as ‘scope creep’. However it would be rare to actually have enough time and patience to fully analyze everything before the work starts. Analysis is also very sensitive to the approach; ask the right questions and you get the right answers. Ask the wrong ones and you get misleading information. Managing analysis means ensuring that both the right questions are being asked, and that the answers are being organized. Experienced analysts need little intervention, but the quality of the work isn’t known until later in the implementation phase, when it is often too late.

Design and Implementation are really about making sure all of the programmers are on the same page; that they are working together well and that they have all of the input necessary to keep them moving forward. For large projects this is basically about team building. A system built by a dysfunctional group is naturally a big ball of mud, and the messiness of the work creates a huge of amount of extra make-work to keep it all from falling apart. A good manager needs to keep the team working, arbitrate disputes and enforce the process and standards even when the time pressure is intense. They also need to protect the programmers from any outside interference to insure that they have enough calmness to be able to think clearly and deeply about their work.

Testing is all about quality vs. time trade-offs. It is impossible to test to 100%, so there is some lower percentage of testing that has to be accepted. Often that might even be just a fixed block of time. The biggest problem during testing is to keep everyone working. Testing is often seen as boring -- particularly after an intense development slog -- and the developers start to lose their attention to the details. Bugs get noticed, but then forgotten. Managing here means dragging people around, holding hands, calming nerves and just trying to get the process as constructive as possible. It can also means having to make brutal choices about quality vs. timeliness. Sometimes the software has to be shipped, even if it is not as good as it should have been.

Distribution of software usually involves an operations dept., or selling the system. Because it is usually forgotten until the end, the distribution is rarely well-planned. Management here needs to insure that the software is supported well enough, but not at the cost of disabling future development work. The handling of feedback and bug reports can all be set up in advance, and most projects require both a standard release and an emergency fast-track process. Most issues take some time, but some need an ‘all hands on deck’ approach in order to deal with them before the situation escalates.

Business

A large software project takes many man-years to build and for most projects the work is never done. It just goes on, year after year. Hiring programmers isn’t cheap and you need a lot of other support staff in the project as well including: project manager, system admin, specialists, etc. Commercial quality systems also need roles like: graphic designer, UX expert, editor, translator, etc.

All together, to build something large that isn’t a weird eclectic mishmash of disorganized functionality means having to shell out a huge amount of money. For software, money is time. That is, if you have enough money, you can hire the necessary resources and experience to get the work done, given some reasonable time frame. A lack of money, means that you have to do more with less, and often times it’s that time pressure that leads people to take ill-advised shortcuts, which will eventually scramble the project so badly it can’t be saved.

As such, setting up a development project always means having to figure out where the money is going to come from. And as is always the case with money, what strings are attached to it. Nothing comes for free.

The business world is very subtle, complex and constantly changing. Most techies who encounter it greatly oversimplify its nature, shades of gray and depth, which generally leads to unrealistic expectations of how things ‘should’ work. Some people have a better intuitive feel than others, but for most people it is best to just write the whole domain off as ‘irrational’ so that they won’t make any false assumptions.

Because it is volatile, the business world inflicts a lot of short-term influence on software development. This conflicts with the long-term nature of the work, so it requires making a large number of very difficult trade-offs. If the work always bends to the short-term concerns it will quickly become a mess. If it always sticks to the long-term, it will likely lose confidence and funding. Thus it needs to perform a very difficult balancing act between the two, so a good leader that finds the right trade-offs will end up taking flak from both sides. If both sides are a little unhappy, then it’s likely balanced properly.

For most people, the ability to balance the business and technical requirements is learned from long, hard and brutal experience. Intuition is usually bias to one side or the other. Even the greatest intelligence can only ‘guess’ its way through the complex interactions, getting more wrong than right. So all that is left is learning from experience; from the successes and mistakes of the past. And it takes some painful introspection to really be objective about the many causes of failure. People like to blame others, but within the spidery web of development, all things are related and no one is immune from influence.

User Domain

Software starts with a problem that needs to be solved. That problem is always domain specific, such as a financial system, inventory or even social networking, etc. Most domains have their own unique set of terminology, usually steeped in history. They often have ‘rules of thumb’ or dirty little secrets lurking in their darkened corners. What they never are, is laid out in a nice rational manner all ready to be modeled in software. Often the data is poorly understood, the process disorganized and each organization within the domain is slightly different. There are always rules, but they are not always followed rigorously, nor particularly logical.

Thus the core problem in software is taking some ill-defined ‘informal system’ and finding a reasonable mapping to a very rigid formal system modeled inside of a computer. In practice this makes the analysis of an existing domain one of the most crucial parts in arriving at a functional software system. However, because it is always gray and messy, it is also the part of the process that people ignore the most often. They just start coding, and hope that somehow, after a while, the answers will come to them. Sometimes that works, but more often getting off on the wrong foot is extremely fatal. If you build too far from the actual answer, then you’ll have no choice but to do it all over again. But since people don’t like to restart their efforts they usually flail at the code, hoping it will somehow work itself out. However if they started really badly, they could bang on it forever and still never get something that works.

A very common problem in development is that the users rarely know what they want, or what would work correctly for their problems. Some have strong opinions, but they often lack a full understanding of the consequences of their choices. Most flip-flop faster than the development can be completed, so tying the process too closely to their wishes frequently results in a mess of half finished, or poorly thought out code. However the converse is also true, code written in an “ivory tower” far away from the users most often oversimplifies the core problems making it somewhat less than helpful. To bridge this gap requires domain experts, and often a very deep understanding of the real problems. Everything the users say is important, but not all of it needs to be taken literally. They usually understand their own informal systems, but the mapping to software, and the code itself are best left to people with plenty of experience. It is easy to write code, but extremely difficult to know what code is right for the solution.

Technology

Last, but not least are the technologies being used. Most projects involve anywhere between 3 and 30 major technologies, and a slew of minor ones. Each technology is eclectic in its own unique way and needs some time and experience to discover its strengths and weaknesses. People often focus on the programming language when assessing skills, but in most projects crafting conditionals, loops and slicing & dicing code are the easy parts. There is skill involved in solving the endless array of coding puzzles, but unless the project is pushing the bleeding edge of technology, it becomes significantly easier as one gains experience. Unfortunately, getting heavily seasoned (>15 yrs) programmers is difficult, since they are expensive and often leave the industry.

Software development is hugely affected by scale and by the desired quality of the final work. For scale I usually break it down as:

small - 1 developer, <30 lines="lines" span="span">
medium - 1-4 developers, <99 lines="lines" span="span">
large - 5-20 developers, <1 lines="lines" span="span">
huge - teams of developers, millions of lines

Projects don’t jump scale easily, they often require a nearly complete rewrite to go from one size to another. Disorganization and bad practices often work OK for small and medium projects, but they become fatal beyond that.

As well as scale, the desired quality is important. I generally see it as:

prototype - rough proof of concept
demo - a small working set of features, mostly works.
in-house - eclectic, lots of rough edges, inconsistencies and operational issues
commercial -- solid, dependable and beautifully designed by graphic designer / UX experts. Looks good, works correctly.

Each increment in quality takes considerably more work and requires specialists to focus on their particular strengths. Products often sell although they are just in-house quality, but then they are vulnerable to stronger competitors. Users may not complain directly about inconsistencies, but they do generate a negative impression of the system. Badly architected ‘balls of mud’ can often degenerate in quality as programmers randomly slap weak functionality into the corners. Poor development practices tend to be reflected in the interface, so often the outside of the system is a good indication of the state of the code base.

A rather dangerous development trade-off often comes in the choice to build or buy (paid or free). Depending on the technical specs, building is often extremely time-consuming, but in my career I’ve seen more people fail because of their choice to buy. All technologies are a collection of their author’s eccentricities, and often these play a dominate role in their usage. Buying specialty libraries and packages usually works well because you don’t have to acquire the knowledge to build them, but for the core parts of the system,if you depend on someone else’s solution you limit your options going forward. That can drive the architecture and constrain the path forward in dangerous ways.

Getting a commercial grade large or medium system to users is always a team exercise. It takes a large group of people, all with different specialties, to make it happen. As such, the team dynamics become crucial in determining the outcomes. Badly functioning teams usually produce badly functioning systems. Rogue programmers may get a lot of opportunity to express their creativity in their work, but they often do it at the expense of the overall project. Multiple different coding styles and no consistency generates high bug counts and stability problems. Once a project gets going it only continues to work if all of the people involved are on the same page, which generally means strong leadership, a reasonable process and a well-defined set of objectives. Efficiency means long-term goals, even if the short-term is volatile. Initial speed can be gained from brute force hacks, little or no reuse and heavily siloed programmers, but each of these accrues significant technical debt, which eventually bogs the project down into a mess. If the work is getting harder over time that often means disorganization, no architecture, redundancies and/or no long term plan, which if left unchecked will only get worse. A well run, well thought-out development effort will build up a considerable number of reusable common components, which if documented will guide extensions and new features. Architecture sets this commonality and management enforces its usage.

Finally

What sets software apart from most other professions is that it requires a larger cross-section of other knowledge bases to keep it successful. The danger in software is that from an outside perspective, it all looks so easy. You just have to throw together a bunch of simple instructions for the computer and chuck it all onto a server. But that type of over-simplification has always lead to disasters, caused by people who don’t have enough experience to respect the underlying complexity and trade-offs required for successful development. Most other professions are usually managed by people who have moved through the ranks. Software is often special, in that the most experienced developers are rarely put into full leadership positions. More often it is business people or domain experts that attempt to drive the projects forward, although few have the necessary prerequisites. Lack of knowledge usually equates to bad choices, which always mean more work and a much higher likelihood of failure. The complexities, knowledge and work involved in software development are easy to underestimate, so the failure rate is obscenely high.

Serializations

2012-09-30T11:53:00.000-04:00

For the purposes of this post I’m going to over-simplify things considerably by saying that all of the stuff that people learn get stored in their brains as a series of ‘models’. A model in this case is essentially a large set of ‘symbols’ and a whole lot of ‘interconnections’ (relationships) between them. Structurally we could view this as a graph (from graph theory), or we could see it as a relative multidimensional mesh of some sort. It doesn’t really matter for this post, either perspective will do.

As we progress through life we collect facts and relationships which help to build up these models. Sometime, we cross-link them to each other forming larger models, although there is no requirement to do so. We can have many models that are essentially independent from each other.

So what we have in our minds is heavily interlinked multidimensional data that is used to drive our actions and behaviors. Often we want to communicate parts of this to the others around us. In order to accomplish this, we pick a starting place in a model and then traverse through the nodes in some complex fashion. We take these paths and we ‘serialize’ them into a stream of sentences which we speak out loud. No doubt the whole process is significantly more complex than what I am describing, but the important aspect is that we are linearizing our vast internal models down to a string of ‘symbols’ that we then communicate to others, who as they listen hopefully update their own internal models with what we’ve sent.

The fact that we are serializing paths through our models has a huge number of interesting consequences. An obvious one is that there can be many many different paths through a model. We see this quite easily because often different people will find very different ways to express the same thoughts. Often too we can see from the paths, that two people may have similar models, but they differ in parts. It isn’t always obvious whether the differences come from the model or from the serialization.

Another consequence is that people listening to the stream (or reading the stream) don’t always update their model or they possibly update it incorrectly (due to lack of attention, ambiguities or other interference). And of course, we do hold multiple conflicting models in our heads precisely because we are seen to so frequently have contradictions in our own understanding. Things we’ve learned in one context can contradict things we have learned in another. If we become aware of that, we could possible intertwined or merge the models, but we not always aware of these contradictions.

The serialization output itself is interesting because we have found many different ways of representing aspects of these models. The most obvious is that we use different verbal languages. Within languages we also break out into different subsets of terminology that are specific to a domain, such as law, computers, medicine, etc. As well, we have learned to serialize aspects in more fundamental formats such as music, painting or poetry. These ‘rawer’ forms generally communicate deeper aspects of our models that are oriented around our underlying emotions. We also have more rigorous representations such as mathematical notation, which we’ve broken down into subsets we call ‘branches of mathematics’ (each often associated with one or more formal systems). People commonly see these types of ‘formal’ serializations as the ends themselves, but really they are just linear representations of multidimensional models that happen to be abstract formal (rigid) systems. Reading and understanding a string of new mathematics results in an update into what we know internally in our models.

One of the most fascinating achievements of the twentieth century was our success in instantiating our own abstract formal systems. Prior to this, mathematics produced models in our heads and we used them to help us explain the world around us. They didn’t have physical manifestations, although we could serialize them to paper and pass them on. But once Alan Turing actuated the notation into a construct that we could create physically, at least one of our formal systems left the abstract world and landed right into the middle of the real one. Turing Machines became physical devices and these ‘computers’ are gradually finding a place in every corner of our lives.

That this was possible reopens the question of whether or not these types of abstractions are just purely abstract or are just generalized manifestations of our reality. That’s a pretty long-winded way of saying that our thinking abilities could be entirely bounded by our universe. That we might not be able to create internal models that are purely abstract. That we can’t think outside of the box, if the box is our own physical reality. However, notions like ‘infinity’ and ‘perfect’ do appear to be disjoint from our reality, so they’re unlikely to really exist, just approximations to them. We can go a little outside of the lines, but it leaves one wondering about how far is too far?

To control our computers we create software, but in a funny sense our creation of software seems diametrically opposed to our own handling of our internal models and serialization. That is, we first create the serialization (the code), then we set it running to achieve a time-based multi-dimensional behavior. So the running software is a model just like the ones we have in our head. The code can still be transferred from one machine to another, communicating through a huge variety of ‘programming languages’ but at some point in the future it might be possible that the runtime model of the system could be updated rathe replaced and reloaded as we do currently. That shift from just instantiating new instances, over to communicating between long-running models could possibly amplify the usefulness of software systems by orders of magnitude. We’ve already started down the road to dealing with ‘big data’, but we’re still focused on the serializations and not yet the construction of massive models.

The Sapir-Whorf hypothesis suggests that our thinking may be altered by the languages we use for communication, but one might easily think that it is the other way around. Some technical subset of language contains significant localized model primitives, which no doubt drive our internal model construction in very specific ways. Thus if one becomes a lawyer and reads a lot of law, eventually they’ll start ‘thinking like a lawyer’. The primitives they communicate with are very specific sub-models that drive how they build up other models. So, if a language focuses more heavily on a eclectic subset of primitives, as it is used for communication, it will affect how people build their internal models. We see this also with computer programmers. They spend their days constructing formal systems, and they often apply that back to the informalities of the world around them giving them a rather inflexible black and white perspective on the rather grey aspects of the world around them.

Another interesting aspect to this perspective is that mathematics, as a set of communicate symbols, could be seen as no harder to learn than any other language. At the high level that is essentially true, but some aspects of formal systems do rely on difficult underlying notions; Douglas Hofstadter called them ‘Strange Loops’. They seem to contradict our build-in intuitional models, causing contradictions. However once enough of these are strange loops are resolved and understood, and are freely available to a person, the rest of mathematics really is just about depth. The models get larger (mathematics is huge), but are still communicable. A question often arises about whether ‘programming’ itself is mathematics, and this too resolves itself. Since the programming languages describe a formal system in a similar manner as mathematical systems describe a mathematical branch, both of them share a great deal of commonality. Theorems for instance, are roughly analogous to libraries of code in that they are higher representations of significant underlying detail. Larger primitives that can be welded more effectively. And since they are both bound by the same constraints as any formal system, finding complete sets of non-overlapping primitives in both models is nearly identical. But at the same time, they are also too different languages, with the same gulf between them as say English and Chinese. Knowing both is possible, but translating between them is harder (although since they have significantly less ambiguities then vocal/written languages much of the difficulty is reduced).

This perspective that we take multidimensional data and serialize it to to a linear stream provides an interesting framework around a large number of seemingly diverse interactions between people. It not only helps characterize our communications but it also gives us an insight into how we think about the world and how we share that amongst ourselves. It is a simplification, but it does help to bind together language, mathematics, computers and our own intelligence. And it gets back to that recurring underlying duality that comes up over and over again between ‘static’ and ‘dynamic’, between ‘nouns’ and ‘verbs’, and between ‘data’ and ‘code’. As we dig into our reality and generalize what we have learned we often return to the same fundamental patterns, which one could easily suspect are really manifestations of our physical reality.

Cooperation vs. Competition

2012-08-12T13:37:00.002-04:00

I read this excellent article in the July issue of Scientific American. It was called “The Evolution of Cooperation”. The basis of the article was centered around game theory, but the essence of it helped me to better frame my understanding of several deep issues. Often there are interesting abstract high-level constructs that once understood, nicely overlay a sea of information, leading to a better understanding.

Our modern era stresses the benefits of competition. From the earliest ideas of a ‘free hand’ that would fix people on a level playing field, we have been inundated with the notion that things would be better if we were freer to compete. Those notions have always bothered me, since inevitably in order to gain the upper hand, people bend towards pushing the boundaries of the rules. Eventually going too far. Thus it is no surprise that the Olympics needs very serious drug testing, since the rewards of winning often outweigh the risks of getting caught. Competition may start out fair, but there are always people willing to abuse it, and gradually over time as more of them do better, the rest follow. Eventually it always degrades.

So these philosophies of unfettered competition, either sound naive or they seemed to be pushed by people who are already at the margins of fairness. They’re already bending the rules and they’d like to get away with bending them harder. I do hear what they are saying, but I suspect it to either be self-serving spin or a failure to accept the full range of expected behaviors from our species.

What the article clarified is that competition really only exists on the back of cooperation. That is, if there are no rules, there are no rules to bend. There is no competition, just chaos. So in order to compete, initially everyone has to agree on a set of rules. Thus cooperation is by far the deeper principle. It must be in place first. In that sense, it seems more than obvious as a concept, after all we’ve come together to form societies, countries, companies, etc. and these entities are all held together by rules that define how one should behave within them. By sheer scale, it seems that we cooperate far more often than we compete, but if one only reads modern literature, it would seem if the opposite were true. No doubt that is because the adherents of competition are louder and more vocal than those for cooperation.

What fascinated me about the article was that their game theory models predicted that cooperation tended to reinforce cooperation. That is, if it gets root somewhere, it tends to get larger, and while the two ebb and flow with respect to each other, the formal system model tries to balance them out in its stead state. Again, not really a surprise, but it does heavily contradict those that preach progress through increased competition. It points towards that philosophy as being unbalanced (and thus unsustainable).

So far all of what I’ve said has been applied generally across the behavior of our organizations, what does that have to do with software you’re wondering? The underlying cultures of software development have always tended towards programmers getting more ‘freedom’. That’s a recurring theme and one that has always bothered me. Freedom in its simplest form is just a lack of rules. And as society shouts so frequently, a lack of rules means that it is easier to compete. So the essence of our cultural values in programming is towards individuals competing against each other in the guise of being able to push their own creative boundaries for their solutions. It’s every programmer for themselves.

That is fine when the output is limited to the amount of work a single individual can do. So if we wrote programs that were no longer than one man-year’s work of effort, the programmers would do best if they were free to code these in whatever eclectic manner they choose. However, the era of small software ended a long time ago. What we are most often building now is built on a huge number of man-years of effort. Big systems. The little stuff exists already, it is the big stuff we are struggling with.

In all the different projects I’ve worked on, one thing has always remained true. If the project is large, it will absolutely fail if the underlying programmers don’t all get one the same page together. That is, it is a team effort, and a poorly functioning team will never be successful. It doesn’t actually matter what the ‘page’ is, it doesn’t actually matter what the standards, style, architecture or technology are. All that matters is that a group of people come together and agree on how the system will be constructed, otherwise the process of constructing it quickly breaks down and the whole thing fails.

One easy way to determine who's really on a team is by what they say. The members of a team never lie to each other, even if what is being said is not pleasant. Lying is a deliberate misdirection and generally something that you might want to do to your competitors, so that you maintain an advantage over them. Telling the truth, as you know it (even if it turns out to be wrong) is what you do when you are cooperating. You are trading your understanding for theirs, so that it is shared and everyone gets on the same page. Withholding information, particularly in this case, is a form of lying. If you let an opportunity pass without speaking up about something important, then you are doing so for competitive reasons. You are deliberately choosing not to alter the direction and you are essentially lying about the fact that you don’t have any ‘other’ information to share.

Given a huge rise in expectations for software, and that the things we seek to build far exceed the complexity manageable by a single individual, getting better results for software development comes directly from getting programmers to cooperate more with each other. However, our culture (and certainly many of the discussions on the web) show that we programmers are highly competitive. We’d rather have our freedoms, and maximize them at the expense of the whole project failing. Or industry. And when that sort of problem happens, we’re quick to blame each other for the problem, rather than accepting that the root cause was a failure (on our part and theirs) to get onto the same page and cooperate with each other. We see this over and over in the industry. Excessive squabbling about how ‘theirs’ is wrong, and ‘ours’ is better. About ‘right’, and about ‘perfect’. Now one doesn’t expect to get cooperation at every level in the industry, our species would never accept that, but what we’ve seen so far is that that we are far more competitive than we should be. Everyone is out for their own personal glory (although ‘glory’ is not always defined in the same way) and as a result everyone heads out into their own unique direction, waving the banner of ‘freedom’. The consequences of this behavior have been obvious. The failure rate for software is stunningly high and there is more energy applied to telling others they are wrong, then there is for civil discussions on what would be better. The number of programmers in the field has been increasing, but the number of innovations in software (not hardware) have been dwindling. We occasionally see some interesting new ideas, but generally there are constrained to a small group of individuals. Little pockets, here and there. There hasn’t been a real major shift in technologies for well over a decade now. Just a few fragments.

So that article was really informative in laying a basis for learning how to improve things. Competition pushes us, but it also stagnates us. It motivates us, but it also limits us. If we want to move forward, then the only way to do so is via cooperation, and in order to cooperate we have to all be willing to all play by the same rules. The specifics of the rules don’t matter, just that we all play by them. It is sometimes a hard thing for people to accept, and certainly in our modern age it has become harder, but it really represents the only way to move forward. We can choose to all do things differently, but that choice also means that we’ve put a cap on what we can now do, and what we have now isn’t particularly impressive. Or we can come together to build really big spectacular things. Most of us want to build better software but in order to do that we have to give up many of our cherished freedoms. Excellence comes with a price attached.

Smart

2012-08-07T18:10:00.000-04:00

We’re dropped into the torrent of churning information, then driven along by its current towards destinations unknown. In this frantic melee we’re often able to grasp at nearby facts, both large and small. Some hold these for curiosity, but others are able to leverage them against the flow, easing their passage. Smart isn’t just what we remember, but rather it’s what we understand -- well-enough -- to be able to wield it effectively as we tumble through the darkness. Smarter is going beyond that to position ourselves for a chance at the better side of the next fork as it comes surging by.

Architecture

2012-07-30T18:19:00.001-04:00

When and why do you need an architecture?

If you are going to build something small or even medium in size, you can just wing it. Small stuff requires little architecture and medium systems most often crystallize along some crude architectural lines without much intervention. If you are going to build something large and you start without an architecture, what you will wind up with is a large series of disconnected and/or overlapping pieces that are impossible to stabilize. The project will die.

So what is an architecture?

In its most simplest explanation it is the overall organization of the underlying sections of code. For something to be ‘organized’ that means that there are a series of rules which categorize a specific layer to a ‘reasonable’ number of sub-pieces. Most often I prefer to refer to these as ‘lines’ and to leave the number of layers to be relative to the problem. An unreasonable number of categories is somewhat dependant, but lower numbers like 3 or 5 are nice, while larger ones like 20 or 30 are problematic.

How does one describe an architecture?

Like any good blueprint, an architectural description is a top-down listing of the lines and the layers. But since there are essentially two dimensions to an architecture there need to be two main roots to the design. The first is the underlying data in the system, while the second is the way the code is constructed to manipulate this data.

A top-down view of the data is essentially a hierarchical structure, starting with the major entities and gradually breaking them down into the details. The overall organization however needs be contained within a reasonable number of categories at each layer, so the major entities are most likely collected into a smaller set of categories at an abstract layer. That ‘organizational’ constraint flows all of the way down to the details. The data itself often cross-references other entities, but it does so internally (as data), thus it often be arranged in a hierarchy.

For the code the situation is similar. A hierarchical structure of categorizations forms the basis, however code is a little more difficult. To avoid redundancies many well-defined parts of the system will be shared at many layers, so here a hierarchy really fails to capture the expressive requirements of a well-designed system. Thus, although there is a root (a place to start), the lines in the code form a graph of categorizations, although these should still be presented as well-organized layers (unwinding this is part of the challenge).

So overall a well-documented architecture has two main sections that each descend down into the details. At each layer, there are a reasonable number of sub-pieces that are either specific or abstract (if necessary to be reasonable). A group of programmers should be able to walk their way through both parts of the document and use all of the contained information to solve all of the final coding-level problems.

If all is going according to plan, they still have significant problems to work on but all of the external decisions have been resolved and they understand how their individual pieces will interact with each other.

How does one create an architecture?

A software system needs a well-defined problem to solve. The underlying details, driven by this problem, come from analysis of the solution, which identifies the required data and the algorithms to operate on the data. There may also be analysis into the operation environment for the system and possibly the development environment as well. All of these details need to be sorted and arranged, then finally organized into layers. Although the final architecture is top-down, assembling the pieces in a bottom-up manner is often simpler and produces less redundancies. Generalization and abstraction are key to reducing the effort and getting it presentable. The architecture is ready when it’s depth matches the programming team’s capabilities.

Five Years

2012-07-28T08:59:00.000-04:00

It seemed so long ago when I first starting blogging. A number of other writers had reached the five year mark in their blogs and I can remember being very impressed. “There is no way I’ll be blogging for that long” I thought to myself. Five years is both a very long time, and a brief instance in one’s life, it is funny how time can seem so contradictory.

Five years actually came last year -- I had blogged a full year at another site before I switched over to Blogger and renamed the blog. At that time I was contemplating the type of post I should write. Should it be funny? I’ve never been able to get my sense of humor to come out correctly in writing (I am actually funny, it just doesn’t translate well). Should I write another of those rants on the ills of the software industry? We’re an immature industry, which drives me nuts, but there doesn’t seem to be much on the near horizon to be able to change that. Perhaps I should reflect on the sorry state of discussions in the Internet? Rude, stupid and mean often combine to interrupt the necessary flow of communication, but I do have a sense that slowly that is breaking; that once all of the newness diminishes, we may actually get down to sorting out the mess our world has become. So what then?

I took up writing because I tried to write a book but then soon realized that I had no real understanding of how to communicate my thoughts in ways that were readable. I had managed to get well into my late thirties without writing anything more than a few pages. I had always known that you don’t really understand something until you can explain it to others, and that verbal conversations doesn’t really allow for you to go deep into a body of knowledge. After a rather productive period back then, I had acquired a new sense of how to build things, but still lacked the ability to share that knowledge. So off I went, continuously stumbling, in the hopes that I could somehow make a difference.

These days I’m no longer so keen on changing our industry. I tend to write for myself, to clarify my own perspectives and to help me tie together the vast stream of loose ends I’ve accumulated over a long, and sometimes painful career. That leads me towards my interests, whether or not they are popular or desired, but I am happier now with what I am saying and slowly I think I’ve managed to improve the way I am saying it. Writing in this fashion is an excellent way to learn to clarify one’s knowledge and I’d easily recommend that more people pursue it. Fame and fortune won’t be yours, but there is a tremendous amount of growth and satisfaction that come from being able to see how you are progressing in your pursuit of wisdom. We only get better as when we acquire more knowledge, and it is far tougher to turn a stream of facts into really deep understanding than most people realize. Maybe when I get to my ten year post I’ll have something more enlightening to share ...

What is Complexity?

2012-06-18T18:56:00.004-04:00

This is going to be a long and winding post, as there are always fundamental questions that do not have easy or short answers. Complexity is one of those concepts that may seem simple on its surface, but it encompasses a profoundly deep perspective on the nature of our existence. It is paired with simplicity in many aspects, which I wrote about in:

http://theprogrammersparadox.blogspot.ca/2007/12/nature-of-simple.html

It would be helpful in understanding my perspective on complexity to go back and read that older post before reading further.

The first thing I need to establish is that this is my view of complexity. It is inspired by many others, and it may or may not be a common viewpoint, but I’m not going to worry in this posting about getting my facts or references into the text. Instead, I’m just going to give my own intuitive view of complexity and leave it to others to pick out from it what they feel is useful (and to disregard the rest).

Deep down, our universe is composed of particles. Douglas Hofstadter in “I am a Strange Loop” used the term ‘epiphenomenon’ to describe how larger meta-behavior forms on top of this underlying particle system. Particles form molecules, which form chemicals, which form into materials, which we manipulate in our world. There are many ‘layers’ going down between us and particles. Going upwards, we collect together as groups and neighborhoods, based in cities in various regional collections to interact with each other as societies. Each of these layers is another set of discrete ‘elements’ bound together by rules that control their interaction. Sometimes these rules are unbreakable, I’ll call these formal systems. Sometimes they are very malleable: thus informal systems. A deeper explanation can be found here:

http://theprogrammersparadox.blogspot.ca/2011/12/informal-ramble.html

If we were to look at the universe in absolute terms, the sum total of everything we know is one massive complex system. It is so large that I sincerely doubt we know how large it actually is. We can look at any one ‘element’ in this overall system and talk about its ‘context’; basically all of the other elements floating about and any rules that apply to the element. That’s a nice abstract concept, but not very useful given that we can’t cope with the massive scale of the overall system. I’m not even sure that we can grok the number of layers that it has.

Because of this, we narrow down the context to something more intellectually manageable. We pick some ‘layer’ and some subset of elements and rules in which to frame the discussion. So we talk about an element like a ‘country’, and we are focused on what is happening internally in it or we talk about how it interacts with the other countries around it. We can leverage the mathematical terminology ‘with respect to’ -- abbreviated to ‘wrt’ -- for this usage. Thus we can talk about a country wrt global politics or wrt citizen unrest. This constraints the context down to something tangible.

A side-effect of this type of constraint is that we are also drawing a rather concrete border around what is essentially a finite set of particles. If we refer to a country, although there is some ambiguity, we still mean a very explicit set of particles at particular point in time (inferred).

So what does this view of the world have to do with complexity? The first point is that if we were going to craft a metric for complexity then whatever it is it must be relative. So it is ‘complexity wrt a,b,c,.., z’. That is, some finite encapsulation of both the underlying elements (possibly all the way down to particles or lower) and some finite encapsulation of all of the rules that control their behavior, at every layer specified. Complexity then relates to a specific subsystem, rather than some type of absolute whole. Absolute complexity is rarely what we mean.

In that definition, we then get a pretty strong glimpse of the underpinnings of complexity. We could just take it as some projection based on all of the layers, elements and rules. That is of course a simplification of its essence and that in itself is subject to another set of constraints imposed by the reduction. Combined with the initial subsystem, it is easy to see why any metric for complexity is subject to a considerable number of external factors.

Another harder. but perhaps more accurate way of looking at complexity is as the size of some sort of multidimensional space. In that context we could conceive of what amounts to the equivalent of a ‘volume’, a spatial/temporal approach to looking at the space occupied by the system. This allows use to take two constrained subsystems and roughly size them up against each other. To be able to say that one is more ‘complex’ than the other

Complexity in this way of thinking has some interesting attributes. One of them is that while there is some minimum level of complexity within the subsystem, organization does appear to reduce the overall complexity. That is, in a very simple system, if the rules that bind it are increased, but the increase reduces the interactions of the epiphenomenon, the overall system could be less complex than the original one. There is a still a minimum, you can’t organize it down to nothing, but chaos increases the size of complexity (which is different from the way information theory sees the world). So there is some ‘organizational principle’ which can be used to push down complexity to its minimum, however this principle is still bound by the similar constraints that hold for any restructuring operation like simplicity. That is, things are ‘organized’ wrt some attributes.

Another interesting aspect of this perspective of complexity is how it relates to information. If complexity is elements and rules in layers, information is a path of serialization through these elements, rules and layers. That is, it is a linearized syntactic cross-section of the underlying complexity. It is composed of details and relationships that are interconnected, but flattened. In that sense we can use some aspect of Information Theory to identify attributes of an underlying subsystem. There is an inherent danger in doing this because the path through the complexity isn’t necessarily complete and may contain cycles and overlaps, but it does open the door to another method of navigating the subsystem besides ‘space’. We could also use some compression techniques to show that a particular information path is near a minimal information path. So that the traversal and the underlying subsystem are in essence as tightly woven as they could possibly be.

A key point is that complexity is subject to decomposition. That is, things can appear more or less complex by simply ignoring or adding different parts of the overall complexity. Since we are usually referring to some form of ‘wrt’, then what we are referring to is subject to where we drew these lines in the space. If we move the lines substantially, a different subsystem emerges. Since there are no physical restrictions on partitioning the lines, they are essentially arbitrary.

Subsystem complexities are not mutually independent of the overall complexity. We like to think they are, but in that all things are interrelated. However, there are some influences that are so small that they can be considered negligible. So for instance fluctuations on the temperature of Pluto (the planetiod) are unlikely to affect local city politics. The two seem unrelated, however they both exist in the same system of particles floating about in space, and they are both types of epiphenomenon, although one is composed of natural elements while the other is a rather small group of humans interacting together in a regional confrontation. It is possible (but highly unlikely) that some chunk of Pluto could come crashing down and put an end to both an entire city and any of its internal squabbling. We don’t expect this, but there is no rule forbidding it.

The way we as a species deal with complexity is by partitioning it. We simply ignore what we believe is on the outside of the subsystem and focus on what we can fit within our brains. So we tend to think that things are significantly less complex than they really are, primarily because we have focused on some layer and filtered down the elements and rules. Where we often get into trouble with this is with temporal issues. For a time, two subsystems appear independent, but at some point that changes. This often misleads people into incorrectly assessing the behaviors.

Because we have to constrain complexity, we choose to not deal with large systems, but they still affect the complexity. For the largest absolute overall system it seems likely that there is a fixed amount of complexity possible. One has to be careful with that assumption though, because we already know from Godel’s Incompleteness Theorem that there is essentially an infinite amount of stuff theoretically out there as it related to abstract formal systems. One could get caught up in a discussion about issues like the tangibility of ‘infinite’, but I think I’ll leave that for another post and just state an assumption that there likely appears to be a finite number of particles, a maximum size, an end to time and thus a finite number of interactions possible in the global system. For now we can just assume it is finite.

Because of the sheer size of the overall system there is effectively no upper limit on how complex things in our world can become. We could apply the opposite of the earlier ‘organizational principle’ to build in, what amounts to artificial complexity and make things more complicated. We could shift the boundaries of the subsystem to make it more complex. We could also add in new abstract layers would again would increase the complexity. It is fairly easy to accomplish, and from our perspective there is effectively an infinite amount of space (wrt a lifetime) to extend into.

One way of dealing with complexity is by encapsulating it. That is cleaving off a subsystem and embedding it in a ‘black box’. This works, so long as the elements and rules within the subsystem are not influenced by things outside of the subsystem in any significant way. This restriction means that working encapsulation is restricted to what are essentially mutually independent parts. While this is similar to how we as people deal internally with complexity, it requires a broader degree of certainty of independence to function correctly. You can not encapsulate human behavior away from the rules governing economies for instance, and these days you cannot encapsulate one economy from any other on the planet, the changes in one are highly likely to affect the other. Encapsulation does work in many physical systems and often in many formal system, but only again wrt elements in the greater subsystem. That is, a set of gears in a machine may be independent of a motor, but both are subject to outside influences, such as being crushed.

Overall, complexity is difficult to define because it is always relative to some constraints and it is inherently woven through layers. We don’t tend to be able to deal with the whole, so we ignore parts and then try to convince ourselves that these parts are not effecting things in any significant way. It is evident from modern societies that we do not collectively deal with complexity very well, and that we certainly can’t deal with all of the epiphenomenon currently interacting on our planet right now. Rather we just define very small artificial subsystems, tweak them and then hope for or claim positive results. Given the vast scale of the overall system we have no realistic way of confirming that some element or some rule is really and truly outside of what we are dealing with, or that the behavior isn’t localized or subject to scaling issues.

Mastering complexity comes from an ever-increasing stretching of our horizons. We have to accept external influences and move to partition them or accept their interactions. In software, the complexity inherent in the code comes from the environment of development and the environment of operations. Both of these influence the flow and significance of the details within the system. Fluctuations from the outside needs and understanding, drive the types of instructions we are assembling to control the computer. Our internal ‘symbols’ for the physical world align or disconnect with reality based on how well we understand their influences. As such, we are effectively modelling limited aspects of informal systems in the real world with the formal ones in a digital world. Not only is the mapping important, but also the outside subsystems that we use to design and built it. As the boundaries increase, only encapsulation and organization can help control the complexity. They provide footholds into taming the problems. The worst thing we can do with managing complexity is to draw incorrect, artificial lines and then just blind ourselves to things crossing them. Ignoring complexity does not make it go away, it is an elementary property of our existence.

Globals and State

2012-06-15T10:42:00.003-04:00

One of the great lessons learned -- long ago -- was that making variables ‘global’ in programming code was just asking for trouble. It is of course, easier to write the code with globals; you just declare everything global then fiddle with it wherever you want. But that ease comes at the rather horrendous cost of trying to modify the code later. Get enough different sections of code playing with the same global and then suddenly it is very complicated to ascertain what any little changes to the variable will do to the overall system. So we came to the conclusion that these ‘side-effects’ were both very expensive and completely undesirable. A lot of effort went into making it possible in most modern languages to batten down the scope for everything, variables, functions, etc. We turned our attention towards stuffing as much as possible into ‘black boxes’ -- encapsulation -- so that we minimize its interaction with the rest of the code base.

Another lesson often learned was that stateless code was considerably easier to debug than code that supported a lot of internal states. If you execute the code and each time its behavior is identical, it is fairly straightforward to determine if it is correct or not. If however, the behavior fluctuates based on changing internal state information, then testing becomes a long and drawn out process of cross-referencing all of the different inputs with the different outputs (a task usually short-changed leading to corner-case bugs). Test cases become complex sequences that are both time consuming and hard to accurately reproduce. Simple tests for stateless code means less work and better quality.

State changes can come from internal modification of variables, but they are most often triggered by things external to the scope of the code. Thus, function A modifies some state information, so that the behavior of function B changes. Generally the call to function A comes from somewhere on the outside of function B’s code block. This essentially forms an indirect reference to the state for function B, which relies not on a global variable, but rather a function that could be accessed globally. A global function. When we banished globals we did so for static variable declarations, however a code-based dynamic call is essentially the same thing. In a very real sense, any part of the program that is subject to changes either directly, or indirectly, that originate from other parts of the program is some type of global. Global data or global action, it doesn’t matter.

Ideally to make everything easily testable we’d like 100% of all arguments explicitly pushed into every function, and to support changes within the system we’d like 0% side-effects, so everything changed is returned from the function. Global-less, stateless code.

Often in APIs there are a large number of different primitives available. Different users of the API will access these subsets of functions in many different orders. In most Object Oriented (OO) languages this is handled by using something equivalent to set/get methods to alter the state of internal private variables, which other primitives use as values in their calculations. However, these methods are only available if the object is within the scope of the caller, so it has the effect of constraining their usage. So long as the object is not global, the methods are not either. The object becomes a local variable, interacting with it can be in any order necessary. However you can violate this easily by either setting the method calls to static or by creating the object as a Singleton. Either way introducing a global effect.

Another way to mess with things is to have the internal data as a reference to an object that is outside of the scope. When changes to that underlying object can occur anywhere in the code,this is another form of global manipulation.

In most systems, particularly if there is an interface for users, there is a considerable amount of mandatory state. Basically the computer is used to remember the user’s actions so that the user doesn’t have to keep supplying all of the contextual data over and over again. Depending on how the surrounding session mechanics interacts with the underlying technology, this can leave a lot of little pieces of required state laying all over the code. This of course is a form of spaghetti (variable-spaghetti) and it can be quite nasty because it gets placed everywhere. Cleaning this up means collecting all of the state information together into a single location for any given technical context. So for instance, in a web app there is likely one big collection of user state information in the browser, and another collection associated with the user’s session in the server. That’s fine and considerably better than having a huge number located in both places.

Long ago we identified that global variables were a big problem and in many circles we banished them. But I think we focused too hard on the ‘variable’ part, and not enough on the ‘global’ aspect. Global anything is a potential problem, anywhere. Like disorganization and redundancies, it is just a pool of gasoline waiting for a match. Software systems can be composed of an outrageous amount of complexity, and the only way to effectively deal with it is by encapsulating as much as possible into well-organized sub pieces. If you break that encapsulation... well, you are pretty much right back to where you started ...

Micromanaging Software Development

2012-06-07T16:42:00.002-04:00

Software development projects fail; often and badly. I’ve blogged a lot about the different causes so I won’t repeat most of that again, only to say if you include all of the various people involved in one way or another, at every level (including the end-users), the whole thing is known to be a very difficult exercise in ‘herding cats’. All of the problems stem from a lack of focus.

There are as many “theories” out there about how to prevent failure as there are ways to fail. They range all over the map, but a few of them rely on micromanagement at their core.

What is micromanagement? My first real introduction to it was from working in middle end restaurants as a kid. Most of them would be highly chaotic places but for their managers. The managers sit on the employees and make sure that they are all following the rules. They need to clock in when they start and clock out when they leave. They have to wash their hands, and the kitchen staff have to put on clean ‘whites’ and hairnets. Employees aren’t allowed to eat the food and they are always supposed to keep busy even if the restaurant is not. There are rules for preparing the food, rules for serving it and rules for handling the waste. As an employee your job is not to think, but to get your work done as quickly as possible, while obeying all of the rules.

When the manager is doing his or her job well, they’re running all over the place causing the restaurant to function like clockwork. A finely tuned machine for grinding out food and collecting revenues.

As a patron I’ve always appreciated a well-running restaurant with good food. As an employee I didn’t actually mind the discipline. At least you always knew where you stood and you didn’t have to think hard about stuff. After a while you just fell into the grove and did whatever the manager wanted you to do. Once work was over, it was over. You were done for the day.

What I realized is that doing a job like ‘line cook’ in a restaurant is a semi-skilled labour position. It takes a bit to learn the ropes, but once you’ve acquired the skills they are pretty much constant. Even into my middle age, I can still grill a ‘mean’ steak (I’ve mastered medium rare and rare. mmmm). Micromanagement is a very good way to manage non-skilled and semi-skilled labour, since it really comes down to a manager just extending their capabilities by directing the physical work of others. And if the micromanager is found to be annoying (as sometimes they are), changes in moral really don’t significantly affect people’s ability to get the job done. It’s probably one of the best way of handling employees in this type of position.

For decades now I’ve seen many a person look enviously at how these types of organizations function and then try to apply this back to intellectual jobs like computer programming. As I’ve often said, by far the bulk of most programming is ‘jogging for the mind’ so it doesn’t always require the uber-deep levels of thought that something like research or mathematics needs. From the outside this seems to confuse people who mistake it for being closely aligned with semi-skill labour. Both require some thinking so bought ought to be the same they surmise. There is however a huge difference. When cooking for instance, my main output is the way I am handling the food as I prepare it. When programming, the byproduct of my work is typing on a keyboard or swinging a mouse around, but the main output of my work is actually in ‘understanding’ the solutions I am creating. An outsider can see that I am using the correct techniques to cook and if they have acquired the same skill set, they can help me correct flaws in how I am working. An outsider however cannot peer into my brain and see that I am correctly understanding the puzzles I am solving and worse, depending on what I am building, they may need decades to acquire the same skill set, just to be able to critique my working habits.

Even if programming is jogging, and it gets considerably easier the more you do it, there is still a huge amount of learning involved. We solve a vast array of problems, on a wide range of equipment, with a plethora of different technologies. Too many things for any single programmer to master them all, and they change fast and frequently. Stop coding for long enough and suddenly the landscape looks completely foreign. It isn’t really and a good knowledge of the many theories of computer science can really do help to keep abreast, but still the amount of knowledge required is staggering.

In principle this means that a micromanager in software is unlikely to have more expertise than his employees, and that he has no way of judging how well his employees are working through the problems (until it is too late).

I’ve seen the first aspect of this solved by creating half-managers/half-programmers. In some sense it has been very common, since well-run teams are usually lead by the seniors in the group. Some organizations have just formalized this relationship and built on it.

The second aspect of this however is more difficult to deal with. Nothing is worse than letting a bunch of programmers run wild for months or years, only to find out that they really didn’t understand what they were supposed to build or how they should build it. A great many projects have failed this way. The micromanagement solution is to reduce the work to very tiny time increment, say a few days. The manager issues the work, the programmer delivers it and then it becomes very obvious if the manager/programmer communication and the programmer’s skills are both at a functional level.

The downside to this is ownership. Once you take away a long-term commitment to building something from the coders, they’re just drones to belt out the pieces. A skilled micromanager is required to direct the process, but the inspiration and quality of work from the programmers is likely to be lackluster. The job has become akin to working in a restaurant and of course once the day is done, the programmers rush off to do better things with their time. This of course can be a functional way to develop software, but it is at the far end of the spectrum caused by trading off development risk for increased surveillance. Lacking satisfaction, the turnover for programmers is high and few of them will have the ability to become micromanagers effectively.

There are no doubt circumstances where micromanagement works in software development, I’ve seen a couple of good examples. The work gets done, but from my personal experiences I’ve always seen that the overall quality suffers greatly for this. Attempts to formalize this relationship fail unless the significance of the manager’s role and exceptional level skill they require, is factored in correctly. Even then, with the manager basically over-extended and the staff turn-over often high, it’s no surprise that the organization is subject to significant volatility in both their output and quality. The risks have shifted from each individual programmer over to the relationship between the programmers and their handlers, but they are still there.

In general, although I find this type of organization structure interesting, I’d have to say that micromanagement is not a particularly effective way of getting over the herding cats problem. I’ve always felt that a well-educated and well-balanced team, with strong leadership is probably a more sustainable way to direct the work. A happy, engaged programmer is much more likely to catch significant issues before they’ve become too ingrained to fix. If they can transmit this upwards, while getting enough of a downwards context from management, they can be well positioned to deliver their individual work both quickly and correctly. Well that’s my “theory” anyways …

Another Stinkin’ Analogy

2012-06-05T18:08:00.002-04:00

Yea, yea, I know. Programmers hate analogies. But I think that this attitude leads one to miss their importance. Sure the world is based on details -- facts -- but these facts are just chunks of information rooted in our physical existence. And more importantly this information isn’t mutually independent, it is tied by context to all of the other information floating about. These ties form a meta-layer of information that binds together the underlying information. It’s these higher level relationships that things like analogies, metaphors, similes try to address at a higher level of abstraction. Sure there are simplifications involved, and there isn’t always a one-to-one correspondence with all aspects of an analogy, but what there is a relationship that can be learned, understood, and applied back to help organize and utilize the facts. Facts don’t help unless you can turn them into actual knowledge …

This analogy comes in two parts. The first is simple: programming is like solving a Rubik’s cube. That is, it is a puzzle that involves figuring out a series of steps to solve it. Like Rubik’s cube there are many solutions, but there is often a minimal number of steps possible. Also, like solving the cubes, there are some states between ‘randomized’ and ‘solved’ where some of the sides appear solved, but the whole puzzle is not. Of course, programming has a significantly wider array of puzzles within it then just this one that need to be solved. They come in all sorts of shapes and sizes. Still, when coding it’s not unlike just working your way through puzzle after puzzle, while trying to solve them as fast as you can. It can often be a race to get the work completed.

The second part of the analogy comes in after the puzzle has been solved. Some people just toss the completed work into a big pile, but big software development needs structure not piles. In that sense, once each puzzle is complete it is neatly stacked to form an architecture. Usually the architecture itself has its own needs and constraints. The puzzles form the substance of the walls, but the walls themselves need to be constructed carefully in order for the entire work to hold together. Sometimes you meet programmers that don’t get that second half of the work. They think it’s all about the puzzles. But it really should be understood that the user’s don’t use the puzzles directly, they use the system and it’s the way the system holds together that makes the difference in their lives. The code could contain the cleverest algorithm ever invented, but if that’s wrapped in a disorganized mess, then its effect is lost on them.

Ok, I lied there is a third part to this analogy as well. I didn’t pick Rubik’s cubes lightly as the underlying type of puzzle. In the second part, obviously the cubes can also act as bricks in a larger structure, but there is still more. For anyone that has spend time solving Rubik’s cube from scratch -- no hints -- they know that it isn’t any easy puzzle. You have to work really hard to work out the mechanics of the combinations that will leave a solution in place. There are, of course, lots of books you can buy to teach how to solve the cubes quickly. Once you get the rules, it’s not very hard at all in fact these days kids compete for speed in times that are quite frankly somewhat unbelievable. Some programmers will argue that using Rubik’s cube as the iconic puzzle is a gross simplification, but I choose it on purpose since it has a rather dual nature. If you don’t know how to solve it, it is a hard puzzle. But if you do your homework first, you can get it down to an incredible speed. That ‘attribute’ holds true for programming as well. Computer languages are simple in their essence, and all libraries are essentially deterministic. If you dive in without any context, it is a difficult world where it almost seems magical how things get built. But if you gain the knowledge first to solve the types of problems you are tackling, then you can belt through them quite quickly. In a prior post I referred to it as ‘jogging for the mind’. It’s not effortless and it takes time, but it’s not nearly as hard as it is if you have no context. Write enough code, and coding becomes considerably faster and more natural.

In turning what we know into knowledge we need to collect the details together then find the rules and patterns that bind them. It’s this higher level that we apply back to our lives, to hopefully make them better in some way. This applies back to software development as well. Programming can be very time intensive. We rarely get enough time to really bring the work up to an exceptional quality. And it’s this deficit in resources that drives most of our problems in development. To counter this, we need to be as effective as possible with our limited resources, and it is exactly this point that comes right back to the analogy. I’ve spent my life watching programmers struggle in a panic to solve Rubik’s cube-like problems faster than is possible, but mostly because they didn’t just stop and find some reference on how to actually solve the puzzles first. They’ve become so addicted to doing it from scratch that they see no other way. Meanwhile, they could have saved themselves considerable stress by just looking up the answer. And to make it worse, because of the time constraints, they often settle on getting a couple of sides nearly done, not even the whole puzzle.

There are, of course, a huge variety of Rubik’s-like puzzles that we deal with on a regular basis, but easily the majority of them are well-understood and have readily available answers scattered about the World Wide Web. The time taken to find an answer is just a fraction of the time required to solve it yourself, and the more answers you find the more puzzles you can breeze through. The puzzles are fun and all, but what we’re building for the user is the structure of the system. Getting that done is considerably more fun.

How do you design software?

2012-05-29T17:10:00.001-04:00

Simon Brown started an interesting thread of discussion:

http://www.codingthearchitecture.com/2012/04/27/how_do_you_design_software.html

In one reply, Gene Hughson added:

http://genehughson.wordpress.com/2012/05/17/getting-there-from-here/?goback=.gde_1835657_member_116375634

I figured I’d also take a shot at explaining how I usually design systems.

Please keep in mind that design is a highly creative process, so there are no right or wrong answers. What works in one case, may not in others. It is also highly variable based on the scale of the team, the project and the system. In smaller systems you can get away with cutting corners that are absolutely fatal in big ones.

I’ll answer this with respect to greenfield (new) software projects. Extending an existing system is similar, but considerably more constrained since you want all of the new pieces to fit in well with the existing ones.

For context, I’ve been building systems for over twenty years. In the last 14 years they have all been web apps and have all been destined to be sold as commercial products. The domains have changed significantly, but even though they are directed at different problems the underlying architectures and design goals have been very similar. I like to build systems that are dynamic. By that I mean that they do not know in advance what data they will be holding, nor how will be structured. There is always some static core, but below that, the exact nature of the data depends what is added to the system as it runs. At the interface level I usually build in some form of templates or scripting (DSL), so that the users are enabled to highly customize their workflows. For data stores, I’ve done OODB, NoSQL and generic schemas. I prefer NoSQL style solutions, but RDBMSes are useful for smaller systems. I consider a system successful when a user can essentially create their own personal sub-system with its own unique schema, quickly and easily using only the GUI. If their work requires long delays, programmers or operations involvement then I’ve missed the mark.

The teams I usually work with are small. I’ve been on big teams, but I find that smaller ones are generally more effective. Within these teams, each programmer has their strengths, but I try to get everyone to be as general as possible. Also, for any specific section of code, I try very hard to insure that at least two programmers know and can work on it. In the past, specialist teams with no overlap have had a tendency to collapse with moral or staffing changes. I prefer not to be subject to that sort of problem.

For any design the very first thing I do is identify a problem to solve. You can’t create an effective solution, if you don’t understand the problem.

Once I ‘get’ the problem, I go about deciding on the technologies. As the materials that make up the solution, the technology choices play a significant role in determining the system’s architecture. Each one has a ‘grain’ and things go considerably smoother if you don’t go against it. They also stack, so you should pick a collection of parts that work well together. Beyond affecting development, the choice often pays a significant role in sales as well, which is a key aspect when the work is commercial. Systems built in specific technologies sell easier in most markets.

For technologies that I have little or no experience with, I always go off and write a few prototypes to gain both experience and to understand the limits. Software capability is usually over-sold, so its worth confirming that it really works as required, before its too late.

With the technologies in hand, I then move onto the data. For all systems, the data is the foundation. You can’t build on what isn’t there, and an underlying schema works best if it isn’t patchy and inconsistent. The key thing to know about the data is its structure (schema/model/etc). But knowing how it gets into the system, its quality, volume and frequency are all important too. Even if the system is brand spanking new and cutting edge, chances are that earlier developers have modeled at least the major aspects of the data, so I find it crucial to do both research on what is known and analysis on what is out there. Mistakes in understanding the data are always very painful and expensive to correct, so I like to spend a little extra effort making sure that I’ve answered every question that has come up. One of the worst things you can do in software is to ignore stuff in the hopes that it will get sorted out later. Later is usually too late.

I work with two basic models for the data. The first is based on the idea that there is some ‘universal’ schema out there that correctly corresponds to the specifics of the data, no matter where it is found or what it is used for. The second is the subset of this model that is specific to the application I am building. If I’m utilizing an RBDMS to hold static elements, I generally try to make the schema in it as universal as possible. I may skip some entities or attributes, but certainly the model for any core entity is as close as I can afford to get it. Keep in mind that I am also constantly generalizing based on my understanding to get up to a higher level abstract perspective for as much of the data as I can. Generalizations cost in terms of the amount of work, the system performance and they slow down the initial stage of the development, however if they are well chosen they reduce the overall amount of work and provide a significant boost in development speed as the project matures.

Now, it is always the case that over time whatever I build will grow, getting larger and more complex. This usually means that the model of the data will grow as well. I can’t predict the future, but I can insure that the bulk of these future changes will be additions, not modifications or deletes. Doing so smooths out the upgrade path and avoids having future development hampered by technical debt that’s grown too large to be able to pay down.

With the base data understood, since I am usually designing large systems I generally move onto the architecture. The individual puzzles that are solved by the code always need to come together in a coherent fashion at the system level. Get this wrong and the system descends into ‘meta-spaghetti’, which is usually fatal (it can be fixed, but given that it is unpleasant work it is usually avoided until its too late).

I usually visualize the architecture as a very large set of lines and boxes. The lines separate different chunks of code, forming the basis of encapsulation and APIs. The boxes are just independent ‘components’ that handle related functionality; sometimes they are off the shelf, sometimes they need to be specifically written for the project. There are all sorts of ways of diagramming systems, but I find that laying it out in this fashion makes it easy to distribute the workload and to minimize overlap between programmers.

I always start by drawing the most natural lines first. For example, if it’s a web-app, there is a client and a server (you have no choice). Given that I’m keen on getting as much reuse as possible I try to avoid partitioning the system vertically based on attributes like ‘screens’. Most screens share a mass amount of common code, so avoiding duplicating it is usually a significant factor in many of my designs. I want one big chunk of underlying code that is called by some minimal code for each screen. Getting that in place usually results in several other sets of lines for the architecture.

With web apps, depending on the specific technologies, there can be some communication between the client and the server. If there is, again I want just one piece of code to handle it. It’s all communications code, so it doesn’t need to know about the data it is passing, just that it is passing it and handling errors. That’s easy to say, but sometimes with this type of piece, there can be several non-intuitive lines required to make sure that it really works deterministically.

Another place where redundancies play havoc is in getting the data to be persistent. Again, I focus on avoiding redundancies, since they are nearly as costly as in the screens, but unlike the screens I am way more tolerant in dividing any unrelated data into verticals. So long as there is a layer above that brings it all together in a consistent manner I don’t mind that the specifics are handled differently based on the underlying type of data. That’s usually a choice based on security or the distribution of work.

In all systems there are ‘big’ computations that grind through specific data. I generally label these as ‘engines’. I set them into the design as black boxes. What’s inside is less important then how it fits into the system. The technologies required usually dictate how these are peices integrated, but encapsulating them means that they can be built independently of the rest of the system. That is generally a big help again, when it comes to distributing the work or scheduling the releases.

Documentation generally depends on the environment and the size of the team. For a small, close knit group of very experienced developers, the major lines sketched on the back of a napkin can often be enough. Well, that and an ER diagram of the schema and some type of mock up of the screens done by a real graphic designer. Basically all of the parts that have to fit together nicely with each other. For some systems, I’ve whacked out the major pieces first then staffed up to flesh out the full system. That works well if you know where you are going and have enough time to articulate the lines in the code.

These types of techniques limit the overall size of the system or stretch out the development time, but they tend to set a tighter path towards reuse.

Sometimes I’ve done full specs. In those cases I’ll go down to the depth that I think is safe, but that is dependent on how the work is organized and who is doing it. So far I’ve always known who was doing the work and what their skill level was, but if I didn’t I’d likely go all the way down to the Class level, as in “here are the classes you are going to build”. It’s better to be too explicit initially, then be unpleasantly surprised later.

In most specs I prefer bullet points, tables and the occasional diagram. Anything but flowing text. Whatever gets the point across with the least amount of work and is easy to skim. The point of a spec is generally to get one or two people to accomplish a specific task, so I shy away from making it large, pretty or all inclusive. It just needs enough relevant details to create the code and nothing more. Flowing descriptions and justifications belong in high level documentation and have a very different audience.

Most of the architectures I have designed have been significantly abstracted away from the user requirements. The requirements affect the data and drive the number and types of engines, but up to this point they haven’t really entered into the picture. Most of the initial work has been about the technology and the data. It’s usually around this point that I try to sit down with a few real users and see what they are doing on a day-to-day basis. I may have found the problem right away, but now its time to actually dig into its ugly side. This manifests itself most often as the generation of lots of screens and some fairly serious extensions to the data model. If the architecture is holding water, neither of these are significant problems. I knew they were coming and now I want them.

From a user interface perspective I am usually aiming to simplify the interface as much as possible. It makes the user experience better, but it also reduces the coding work and testing. Well, not always. Some simplifications come from building more sophistication into the backend. The system holds a deeper more complex perspective on what the user is actually doing, so the user doesn’t have to hold it themselves. Those types of design issues generally land into the category of just being more data or engines, so there is usually a place for them to roost, long before they’ve been articulated.

Pretty much, if the architecture is doing it’s job, the growth and extensions to both the code and data are landing in previously defined parts of the system. To ease coding collisions I generally push building the code from the back to the front. The schema gets extended first, then the server, then the front-end. That also avoids creating fancy features that don’t map to the existing data.

Of course, there are always issues and problems. Things never really work according to plan, they take longer than expected and designs are rarely comprehensive enough to cover all of the extensions. My only rules are that if the design is wrong, we have to admit it as early as possible and then fix it properly as soon as possible. The longer you wait, the worse it gets. But it should be noted that any software development effort is always part of a much larger context (organizational/motivation), so often this larger context gets priority over the design and development issues. It’s these outside influences that make success so tricky to achieve.

That just about covers it from a high level. If all is working correctly, the analysis takes a bit of time up front and the development starts off slowly (from an interface perspective), but then generally the project falls into a comfortable steady state where each new extension gets easier to do than the last one. There are sometimes speed-bumps cause by a jump in scale or some ugly technical debt. Those have to be dealt with as early as possible. It’s worth noting too that in the beginning, there is usually a lot of moaning about time and progress, so it becomes important to veer away from the ‘right’ way to grab low hanging fruit (usually demos or throw-away features). But it’s always important afterwards to redirect the project back onto a more stable, long-term trajectory. Knowing when to bend, and figuring out how to undo that later before it becomes a huge problem, is extremely difficult and take considerable past experience to make viable choices.

New Layout

2012-05-22T16:55:00.000-04:00

Just to keep life interesting, I've changed the template on my blog to Blogger's dynamic template.

One consequence is that I now need to send the full posts in the feed (I didn't before because I wanted people to visit the site so I could track them).

Another consequence is that DISCUS isn't supported, so for a short time I've turned off my comments. Once I figure out how to get DISCUS working again, I restore them.

Given these issues I may change the blog back to a simpler template, but I figure I'll leave it this way for a few days until I decide. Enjoy (it's quite an entertaining template :-)

UPDATE: Seems like DISCUS isn't going to support this template for a while, so I think I'll just open the blog up to comments in Blogger and then import them over later, when DISCUS is ready.