The Programmer's Paradox

New Layout

2012-05-22T16:55:00.000-04:00

Just to keep life interesting, I've changed the template on my blog to Blogger's dynamic template.

One consequence is that I now need to send the full posts in the feed (I didn't before because I wanted people to visit the site so I could track them).

Another consequence is that DISCUS isn't supported, so for a short time I've turned off my comments. Once I figure out how to get DISCUS working again, I restore them.

Given these issues I may change the blog back to a simpler template, but I figure I'll leave it this way for a few days until I decide. Enjoy (it's quite an entertaining template :-)

UPDATE: Seems like DISCUS isn't going to support this template for a while, so I think I'll just open the blog up to comments in Blogger and then import them over later, when DISCUS is ready.

Bag O'Tricks

2012-05-08T18:39:00.003-04:00

There are at least two different approaches to computer programing.

The first approach comes from slowly building up an understanding of coding ‘tricks’. These are simple ways to solve simple problems. Initially, people start with the basic language features: assignments, conditionals, loops and functions. As they figure them out these go into their Bag O’Tricks. Then they start adding in language library functions, like string handling, files, data-structures etc. Gradually as they learn more, their Bag O’Tricks gets larger and larger. Many people move on to adding higher-level paradigms like design patterns. Most add in specific tricks for different technologies, like databases, networks or frameworks. Over time programmers end up with a fairly large collection of ways to solve lots of sub-problems within different languages and technologies.

When confronted with a new problem, they quickly break it down into sub-problems, continuing until the pieces are small enough to be solved with their existing Bag O’Tricks. If they are curious folk, they generally try to learn more tricks from examples or their follow programmers. They collect these up and apply them as necessary.

This is a very valid way of programming, but it does have one significant weakness. At any time during their career, a programmer’s Bag O’Tricks contains only a finite number of different solutions. They can arrange them in different ways, but their capabilities are limited by their tricks. That works wonderfully when the problems are the same or substantially similar to ones they have dealt with in the past.

The trouble comes when they encounter a problem that is of a new or different caliber. What happens -- you can see this quite clearly in a lot of code -- is that they start applying their tricks to the sub-problems, but the tricks don’t pack together well enough. These solutions become very tetris-like, basically odd fitting blocks with many gaps between. Of course, past success clouds present judgement and since the programmers have no reasonable alternatives -- given the ever present time constraints -- they keep heading down their chosen path. It’s the only path they know. When this goes wrong, the result is a bad mess that is unstable. A problem outside of the scope of a programmer’s tricks is one that they aren’t going to be able to solve satisfactorily. The industry is littered with examples, too numerous to count.

The second approach to programming is to drop the notion that ‘code’ is “the thing”. That is the key, to let go of the idea that creating software is all about assembling lists of instructions for a computer. Yes, there is always ‘code’, but the code itself is only a secondary aspect of a larger issue. The ‘thing’ is what is happening ‘underneath’ the code. The root of everything in the system. The foundation.

Right down at the bottom is data. It is what the users are collecting, what the database is storing and what people are seeing on their screens, reports, everything. All coding problems can be seen in the light that they are just instructions to take data -- stored in one place -- and move it to somewhere else. Along the way, the structure or shape of the data may have to change as part of the move. And on top of the data, there may be a requirement for ‘dynamic data’ that is calculated each time it is used, but this is only to avoid storing that data redundantly. Ultimately it is all about the data.

So the second approach is to forget about the code. It’s just a vehicle for getting data from somewhere, transforming it and then passing it on. The list of instructions is meaningless, the system is all about how data flows from different locations, is manipulated and then flows elsewhere. You can visualize the entire system as just data moving about, going from disks to memory, heading in from the keyboard and heading out to the network, getting dumped to the printers, being entered and endlessly modified by users. You don’t really need to understand the specifics of the algorithms that tweak it as it moves, but rather just its starting and final structure. The system is the data, and that data is like a car, where the code is simply the highway that the car follows to get to specific locations.

This second approach has considerable advantages. The best one is that a programmer seeing their work as just taking data D and getting it to D’ is no longer restricted by their finite Bag O’Tricks. Although they can permute their tricks endlessly, they are still heavily restricted from solving particular problems correctly. But a transformation from what is essentially one data-structure to another is a well-defined problem. There may be some sub-algorithmic issues involved in the transformation, but once broken down into discrete pieces, figuring out the code or researching how to do it properly are very tangible activities. So the programmers are in a good place to solve the system problems correctly, rather than just trying to endless combine tricks in the hopes that most issues go away.

Another major advantage is that a data perspective on the code allows for easy and natural optimizations. The programmer is no longer combining pieces, which often throw the data through unwanted gyrations. Instead the data goes directly from point A to point B. It’s a straight line from one format to another. As well, the programmer can widen their scope from just a localized problem all the way up to ‘every use’ of a particular type of data, anywhere in the system. This opens up huge possibilities for macro-optimizations that generally provide huge boasts to the overall performance.

One common difficulty in software development is system upgrades. The code upgrades really easily, you just replace a block you don’t like with a block that you do. Data however, is a major pain. Upgrades force merges, backfilling and endless restructuring. If you are initially focused on the code then the upgrade problem gets ignored, where it quietly grows and becomes dangerous. Focusing on the data however brings it front and center. It becomes just another sub-problem of moving the data from one place to another, but this time across versions rather than just inside of the system. It makes tackling a big upgrade problem no worse than any other aspect of the system.

Added to all of this, it is far easier to visualize the data moving about in a system instead of seeing a mountain of poorly organized code. This makes architecture, debugging and testing far simpler as well. For example, a large inventory system with lots of eclectic functionality becomes conceptually simple when viewed as just a way to collect and display very specific items. This twist then leads to ways to combine and organize the existing functionality so that it is easier for the user to wield. Generalizations flow naturally.

Over the years, I’ve seen many a programmer hit the wall with their current Bag O’Tricks approach. Their ability to correctly solve problems is limited, so it is easy for them to get into a position where it becomes convoluted. However, seeing the data first breezes right through these issues. It becomes very quick and convenient to either manipulate the data into the correct form, or to determine if such manipulations are even possible (there are many unsolvable problems). Getting back to the earlier analogy, if you don’t have a viable car, you don’t really need to consider which off-ramp would be best.

Often I like to refer to programmers who rely solely on their Bag O’Tricks as having ‘one eye open’. The programmer may be very good at coding, but they’re too constrained by the limits of their existing tricks. If they spend their career staying within those boundaries, there are no problems. But if they want to get out there and build spectacular things that people will love, then they’ve got to get that second eye open as well. Once they’ve done that, they are no longer limited by what they know, just by the available time and their ability to correctly analyze the problem space. A whole new world of possibilities opens up. They just have to learn to change their perspective.

Structured Questioning

2012-04-08T15:36:00.000-04:00

I worked with a person who believed that the secret to getting things done was just to make a list, then start checking off items as they were done. For anyone that’s worked on a large software development project, they know that it’s just not that easy. Different items depend on each other, progress varies, people come and go, and that leaves a dynamic shifting landscape that continuously results in having to reorder the work. It’s far more of an exercise in juggling a huge number of balls, then it is an act of just marching through a list.

But still, for most people in the project they are usually just tackling one, two or even three items at a time. From their perspective, their traversal through the work is very list-like. It’s just a matter of getting the things from the ‘to be done’ category into the ‘completed’ one. Order is very important, but only if you are looking to get the most work completed with the least effort.

I’ve been thinking about this lately, but with a slightly different spin. Big projects generally require a massive amount of analysis to get finished. Failure to complete that work, results in serious scope creep problems. If you only kinda know what to build, as you progress and learn more that new knowledge radically changes what you are building, invalidating some of the earlier work. That’s not very efficient. A little more up front analysis always results in a lot less reworking.

So what is the best way to ‘structure’ analysis, so that you know you’ve completely covered the problem space?

Ultimately, I think graph theory holds the answer to this type of problem. A graph is an abstract collection of vertices and edges. The vertices (or nodes) are interconnected to each other via edges. A graph is independent of any strict positional layout; the vertices aren’t fixed to any dimensional locations. The ‘where’ of the nodes isn’t important, only their relationship to each other. You can of course constrain a graph dimensionally, a planer graph for instance is one that can be fully represented in 2D with no edges crossing each other. For larger graphs with more edges, each can be constrained within some specific N dimensional space (with some higher dimensional equivalent to ‘crossing’ defined).

What makes graphs so useful is that they can be used to model multi-dimensional problems without getting tied down into any spatial positioning issues. They simply represent the interrelationships between different entities. They’re pure information.

Applying this to analysis is a matter of figuring out what entities the vertices represent. That comes pretty naturally if we realize that a question/answer format is strong way of structuring complex informal systems. The question sets a constrained context, while the answer iterates through all of the associated details. We see this commonly in technical writing when the author uses a dialog between two parties to explore issues at depth, or a group of people create a ‘howto’ reference document to make it easier for people to navigate quickly to the answers. Both these forms provide a question/answer structure to express large or complex information.

Using this perspective, we can simply set each vertex to represent a question/answer pair. That sets the structure, but by itself it really isn’t all that useful. Depending on the question, answers can often result in a slew of more questions. These sub-questions are important, and related, but fall out of the original context defined by the question. That is, any question/answer pair has a relationship to many other question/answer pairs. There is some meta-relationship between them.

While the underlying structure of all of these related question vertices is a graph, what we are interested in isn’t their overall relationship structure, rather we are interested in making sure that as we traverse all of these questions in the analysis process; that we’re not dropping any important balls. So while the underpinnings are a graph, what we need is closer to a list of questions and answers.

Working that in is as simple as just iterating through this information as a list of vertices + all of edges. That is, we set the question as a unique identifier for the vertex, the answer as the content and include a list of other related questions. So it looks like:

Question: “...”
Answer: “...”
List of related Questions: [ “...”, “...”, ...]

At this point it may seem like we’ve spun right around into a ‘trivial’ approach, but I suspect that the overwhelming simplicity of this structure hides some of underlying depth. With a careful selection of the syntax of the questions, we leave ourselves open to being able to gain some confidence in knowing just how much ground is covered by a list of these ‘structured questions’. We can for instance, constrain the types of questions by spatial or temporal bounds. Thus some questions can be independent of time, essentially nouns, that set a context in our physical world. These come in the form of ‘who’, ‘where’ or ‘what’. We can also include temporal questions, based around verbs, that seek answers for causal relationships in the past or predictions of the future. These are of the form ‘why’ or ‘how’.

Strict structuring of questions both insures that the answers are significantly ‘atomic’, while also making it easier to identify alternative or ambiguous questions that need to be aligned, removed or rewritten. This is necessary considering the goal is to provide some structure that leads to a confidence that the problem space has been fully explored. Redundancies, if not detectable, lead to concerns that the process has fallen into analysis paralysis. Chaotic exploration most likely results in large undetected gaps (that in software don’t get found until it’s too late).

All of this brings us full-circle back to the problem of trying to frame the analysis. If we were interested in analyzing an area related to building new software, we’d start with what is essentially the root of all of all such questions “What problem are we trying to solve with a computer?” An answer to this, for a common problem like social networking, might be something like “Allow the users to communicate effectively with each other via software running on the computer”, which is simple enough that it immediately generates a slew of sub-questions:

Who are the users?
Why would they use this software?
How do they use the software?
What does it mean to ‘communicate effectively”?
etc.

Then we can take this collection of sub-questions -- constrained relative our root question -- and start breaking them down into answers, and more related questions. This effectively brings us back to the idea of ‘just creating a list and working through it’. There is a process to move forward, constraints to prevent endless analysis and a means to know that the work is approaching ‘completeness’. We just keep iterating the list, until the questions dissipate or become trivially uninteresting.

Now of course, initially it’s hard to know how many sub-questions will spin off an initial one. There is a combinatorial explosion in progress as each question spawns more questions. But this is why it’s so important that there are constraints. There are natural boundaries to the analysis, which if articulated early prevent an endless process that will go outside of the lines.

Inside of this space however, is a finite area that we are hoping to cover completely. As the analysis progresses, the growth of new questions will reverse. This shift provides a good indication of the remaining work. That is, if the process is initially inestimable, as more is known it will approach increasingly reliable estimates. The unknowns will gradually vanish with time. Tracking this progress, identifies the length.

In practice this might come to play as setting a fixed period for the analysis, and then mid-way through having to redo the original estimated effort, but hopefully enough is already know that this only has to happen once. And although the effort exceeded the initial prediction, the downstream cost savings in not having to cope with scope creep will put the project in better shape.

Overall, I like the idea of structured questioning, although I haven’t worked it into any real-world analysis yet. It just follows from my understanding that almost any ‘organization’ is critical to making large projects work. The key is that it is incredibly simple, yet it is highly constrained by a defined structure. These two attributes always combine well. The things that work in practice need to be easily understood, and they need to provide enough value that people gravitate towards them because they help, rather than hinder the process.

Organization

2012-03-30T17:31:00.001-04:00

There is one simple, yet fundamental rule for organization: if someone hasn’t explicitly organized it, then it is disorganized.

I believe that one can infer this from the properties of our physical reality. We know for instance, that entropy always wins. What starts as chaos, ends in chaos. Order is a temporary state of affairs. So, without explicit action, order is highly unlikely.

This rule is especially important when building software. A reasonably large system may consist of millions of individual details, all of which need to be embedded into the system. The scope of any development project includes all aspects of analysis, design, implementation, testing and operations. It includes the process of completing the work as well as the details that go into it. There may be regulatory or licensing issues floating about as well. Each area may not appear daunting, but when taken together there is an awful lot between the genesis of an idea and actual utilization of it for practical purposes.

All of these different aspects need coherent organization. They need to be collected, sorted, categorized and normalized so that they are available to the builders, operators and often, users of the system.

In a small project, you can easily get away with a casual approach to managing these details. If there are three or four parts to something, they don’t need any explicit arrangement. But as the size of the project increases, the number of parts grows extremely fast and thus juggling, say a few hundred unorganized parts easily results in a significant number of unexpected problems. And these problems may cascade into other problems.

This effect of scale often means that sloppy habits learned from small projects generally lead to overconfidence in big ones. Just because some aspect didn’t require organization when the project was a mere 20,000 lines, doesn’t mean that it won’t when it balloons to a few hundred thousand lines. And millions of lines requires something else altogether.

As the project size increases, the effects of disorganization become more pronounced. They require more effort to control, and can balloon into more formidable problems if left unchecked for too long. Thus in serious software development, scale is everything. It’s the first thing you investigate, and it’s always the most significant aspect of any large project. The only way to tame this rampant complexity growth is via organization.

Realistically, it doesn’t matter how it is organized, so long as that organization is consistent and sustainable. But it does matter that the organizational system spans the entire scope of the effort. You can’t just organize a sub-part of the project or process and hope that it will somehow magically propagate to the other areas. You have to cover over all of the details, and all of the work getting done, and you have to insure consistent application.

And it also matters how deep the organization goes. You may have some higher level methodology, and be very organized right at the bottom in terms of the code, but if what’s left in the middle is left untouched, eventually it too will cause significant problems. Every part of the process, architecture and sub-problems needs some coherent organizing principle.

Once the dust settles, what usually brings down ambitious software projects, or grinds them to a standstill, is an explosion of complexity. Fundamentally there is nothing that you can do about that, other then partition it carefully and encapsulate it into manageable sub-parts. But it is possible to prevent any secondary complexity caused by disorganization. And it is this avoidable artificial complexity that generally spirals out of control. When stopped the project becomes tractable, but when ignored it combinatorially explodes as the project expands. A little bit of organization goes a long way ... and enough of it may save the project from a premature death ...

Lost in Thought

2012-03-14T11:46:00.003-04:00

Computers are stupid. They take massive lists of instructions and mindlessly process them one by one. That’s purely mechanical, the result of billions of nearly invisible gates flip-flopping in a sea of electrons. A consequence set in motion by these sets of instructions.

Software -- when it is written well -- can be quite intelligent. It can remember things for you, help you organize them, control devices and even assist you in interacting with other people. Where does this intelligence come from?

The act of ‘programming’ a computer is the rather long and often painful process of assembling larger and larger lists of instructions. To get intelligence embedded into these lists it must come from the programmer. Some of these ‘smarts’ are just derived from long understood ways to combine the instructions. Some of them come from the programmer’s intuitive sense of how the world works. Some of them come from the knowledge passed onto the programmer other people who are familiar with the problems.

Thus, the source of any intelligence in the software is the thinking done by a programmer on the vast amount of knowledge acquired in relation to the solution. Programming -- often shorted to ‘coding’ is thinking, and the by-product of this thought is code.

Some code requires deep intensive thought. The programmer needs to visualize the interactions of the data in their mind, in order to linearize it down into reliable instructions. They need to explore all of the dark corners that get implied by the computer’s behavior.

With experience, most code is pretty light thinking, the mental equivalent of ‘jogging’ for the mind. It’s just smaller collections of conditionals and loops that solve very specific sub-problems. When the number of these types of pieces explodes, the heavy thinking shifts towards ways of reining in their organization and consistency. Solutions to the micro-problems in coding are most often black and white, they have definitive answers. While solutions to the larger organizational and context issues tend to revolve around difficult trade-offs, where no answer is completely satisfactory.

One of the main consequences of programming just being thoughts is that muddled or distracted thinking directly determines the quality of the code produced. If someone is rushing through a sub-problem, their answer may be fragile or they may have actually solved the wrong problem. When elegance is achieved in software development -- an increasingly rare occurrence -- it is a consequence of a well-thought out approach and a consistent implementation. The programmer understood a problem so well, that they were able to express it in its most simplest terms, and their own self-discipline was so strong that they dotted all of the ‘i’s and crossed all of the ‘t’s. The details were attended to, no matter how small.

Of course loud environments, excess stress, long hours and rampant disorganization are all easy disruptors for the ability to think clearly. So is rushing through the work as quickly as possible.

For any coding veteran, they can immediately see the underlying quality of thought that has gone into the work, whether it was an interface, API or code-base. The poor quality of workmanship almost ‘glows’ if you know what you are looking for. An interface with endless out-of-place functionality tacked all over its design speaks of disorganization and coders that don’t work well together. Inconsistent primitives for an API cry out the pain of wave after wave of quick hacks to slap on poorly thought-out extensions. A tangled mess of repetitive code sings a sad song of confusion or a rush to just get it barely working. These existing examples of other people’s work not only instruct the computer, but also tell the tales of how they were constructed and how much care and effort went into that process. They reveal far more than just functionality.

Programming is thinking. Poor quality thinking results in poor quality code. Rushed thinking results in poor quality code. Distractions result in poor quality code. Bad morale results in poor quality code. Ultimately the utility of a piece of software is dependent on how intelligent it is, and that comes directly from the efforts of the people that built it. Crap quickly tossed together in a hyperactive sweatshop is, well, crap. If you really want to help people solve their problems, you have to spend a lot of time thinking very deeply about a solution that works. There is no other way.

Working Environments

2012-03-12T17:52:00.003-04:00

I’ve become increasingly interested in the different types of working environments currently available for programmers and software developers. My past has included a wide variation of project sizes, companies, technologies and system quality. That gives me a sense of how much variance is out there in different development shops, but my exposure has been limited to companies in Toronto, London and Waterloo. I’d like to know what other environments exist, and how people feel about them.

For anybody interested in describing where they work or have worked in the past, you can comment on this post or if you’d rather, you can send me email directly. If I get enough feedback, I’ll put together some type of summary in a post (but no company names).

Questions about different environments:

What type of development work is going on? What’s the domain and the core functionality? Is the system sophisticated? What sort of planning and design occurs? How long does that extend into the future? What’s the methodology used when building?

Is the culture to get it down quickly, or is it to get it done right? How long do people spend analysing vs. building vs. testing? How much research goes into the underlying algorithms? How about interface standards? How is technical debt. handled? What types of documentation is getting done? What about code reviews?

Who is making the product decisions? Who is making the technical ones? Who does the analysis? What role do domain experts play, if any? Who is making the interface choices? If the project is large, is there consistency to the interface? Are there graphic designers? Editors?

How many bugs are being found? Is there a separate QA group? A separate operations dept? When bugs are fixed, do they ever reoccur again? Do related problems occur frequently?

Are projects coming in close to their schedule? How often are releases happening?

What’s the overall environment like: fun or serious? Is it expected that everyone works overtime? How much overtime do people normally work? Do they encourage people to have a healthy work/life balance?

What is the office environment like? Is it quiet? Private? Are the software developers all together? What about management? Are there technical managers? Do they code as well?

How does this environment compare to others you’ve experienced?

Complexity and Decomposition

2012-02-24T18:33:00.000-05:00

The standard approach to getting a big project accomplished to identify the problem, come to an overall solution, then keep breaking it down into smaller, more manageable pieces. Once these are small enough -- specific tasks -- all that’s left is to march through them one-by-one and complete them. This is quite a reasonable way to approach problems where the problem isn’t heavily hierarchical in nature and the sub-parts are mutually independent or at least mostly independent.

If however, the problem is multi-dimensional in its nature, and there are a significant number of different ways to decompose it, this doesn’t work. If there are at least two logically consistent ways to decompose a problem then the underlying pieces are interdependent. These cross-dependencies means that “linearizing” the problem and marching through each piece in sequence will result in a significant amount of redundant effort. Beside the extra work, redundancies reek havoc because people are inherently inconsistent. These resulting differences in output (and any consequential problems) amplify the complexity, and again result in extra work.

An example of this is a problem that can broken down into A, B and C. It can be further decomposed into 1, 2, 3 and 4. So if we decompose lexically first, we get the following pieces:

A1, A2, A3, A4, B1, B2, B3, B4, C1, C2, C3 and C4.

If we decompose numerically we get:

1A, 1B, 1C, 2A, 2B, 2C, 3A, 3B, 3C, 4A, 4B and 4C

Thus we have a two level hierarchy which falls into 12 different sub-pieces that we need to complete in order to get the project done, with two reasonable ways to slice and dice it.

While this type of problem exists generally with any type of labor, it is very noticeable in software development. As such the concept of ‘polymorphism’ arose. The essence behind this concept in software is that for a number of similar but different types of data, they all share a common interface so that the code working on them doesn’t need a specific implementation for each different type. Thus, this approach addresses a decomposition on a finite number of static data-types. But the concept itself is general. It is also not restricted to things that are particularly static, nor data. Thus it can be applied to dynamic data as well. Data that is mutating in unexpected ways can still share a common interface (and support reflection). But it can also be applied to static code, where all of the blocks of code share an identical interface. And of course, given those other two, it can apply to dynamic code as well.

While it is a commonly used and a very effective programming paradigm, it can also apply generally back to any basic problem decomposition. In our above example, there are 12 different pieces of work that need to be completed. But there are really only 7 different unique items in that work list. Their permutation may be a combinatorial explosion, but that doesn’t mean that the only way to approach getting the work done is to explicitly work through all 12 pieces. We might conclude that at maximum efficiency we need only complete 7 pieces of work, but I would suspect that in practice there is some other overhead required in order to bind together the sub-pieces. Still the expectation would easily be that a polymorphic approach to handling the work would lay somewhere between 7 and 12. If it were 8 for instance, that would mean that the most efficient way to solve the problem was 2/3rds of the amount of effort required to just pound through all of the pieces. That would be a considerable savings.

Now, at this point some readers are probably on the edge of their keyboards, but they likely fall into two different categories: a) those that disagree with there being a more efficient way of solving the problem, and b) those that see this whole discussion as obvious. I’ve certainly met both kinds of people in practice, although I think more people fall into the ‘a’ category. The reason I think this is more common is that it is easy for most people, when faced with a problem to adopt tunnel vision. That is, they don’t focus on the ABCs, nor on the 123s, but rather they go pick what they believe to be the one correct decomposition (lexical or numerical) then they narrow down their field of vision to deal only with a sub-problem like A1. While working on A1, they don’t want to hear anything about A2, and certainly don’t want to know, or even consider C4. They just accept that there are twelve things to do and start doing them.

Again, this shows up very clearly in programming; quite frequently. It is fairly common if you read a lot of code to see a programmer belt out very specific handling for A1 in one location, and then find another, completely separate, yet nearly similar A2 in another. Not only does this occur for code, but it also pops up in the data and the static data used for operational configuration. Software, these day, is highly redundant. It actually gets worse when you look at teams of programmers. The large to massive code bases seem to start with 2x - 3x redundancy and I wouldn’t even want to guess what their worst factors are. The just become ever increasing seas of redundant data and code.

Even though polymorphism is a well-known, popular approach to architecting code, It becomes very unlikely in most cases that a significant proportion of most code-bases is utilizing it across the board. There may be some sub-instances, but most often they are trivial uses. This continues to occur, while oddly one of the most famous modern adages of programming is the DRY principle -- don’t repeat yourself -- which came out of The Pragmatic Programmer book (and possibly series). It’s rather obvious that this wisdom isn’t getting followed on a regular basis quite simply because its really really hard to accomplish in practice. If it were being attempted more often, we’d see it dominating far more of the technical conversations. We see it show up more often in the vast amount of open code that exists. And a huge number of modern technologies not only violate this principle, but they actively encourage it. So while it is talked about, it is quite often swept under the rug in the rush to belt out the next version.

I do find this ironic because certainly in software, we’ve seen a huge growth in the expectations for what software can do, and in general given the rapidly increasing -- out of control -- complexity of our societies, you’d think more and more people would start to look for efficiencies. But it does seem that the opposite is true about human nature. That is, the more complex things become, the more tunneled the vision of the people who have to deal with them. Our species seems to retreat towards the shallows the moment things get too much for them. This, it seems, sets up a feedback loop, forcing people who took the long road to be late, thus forcing them to not analyze the next fork sufficiently and again choose a longer route. And then it never ends.

And it’s that type of negative feedback loop that I’ve seen in many software development projects, both at the coding level and the organizational one. The projects get so ground down by slapping on band-aids to unnecessary problems caused by redundancies, that they lack the resources to avoid creating them in the first place. The work degrades into a loosely coupled collection of disorganized functionality that mitigates most of its own benefits. Not only have they chosen the lest affective path, they’ve also opened up an ever widening sink hole for their own resources. And it seems that this type of problems comes directly from our own intuitive approach to solving problems. We are most likely to not make the right choices (and then far too stubborn to backtrack).

Software Clearing Houses

2012-02-16T18:04:00.000-05:00

I love the idea behind open source. If I’m building something that uses someone else’s work, being able to drop into their code, investigate, curse, and then work around their problem is a huge time-saver. Nothing is worse then wasting hours guessing at what weirdness lies beneath.

The problem is that this idea of ‘open’ mutated into the idea of ‘free’, and ‘free’ is not a good idea in a society that revolves around money. If you write some fabulous piece of code and give it away for free, not only do you not make money yourself, but you’ve also prevented other programmers from making money by writing something similar. Not all of us are lucky enough to get funded by other means, some of us need to pay mortgages and bills and such. We do this by getting paid to work. By writing code for a paycheck. If everything is free, we’re going to have to find some other (less agreeable) way to pay the bills.

A slightly worse problem is that as more and more stuff becomes free, more and more of the low hanging fruit disappears. What that means in reality is that it becomes harder and harder for programmers to go out on their own; to start their own companies. Instead the control of the industry shifts to the big players, who have little incentive to innovate. If you can write something small and profitable, then you get the freedom to experiment. If you can’t, then you’re stuck for life in a big sweatshop writing broken code for people who don’t get computers. I’ve definitely seen this trend in the industry over the last few decades. The really innovative works have nearly vanished and been replaced by more and more sloppy re-works of existing wheels. Not only that, but the profits come from the upper levels of software, so the lower ones get stagnant as the bugs get permanently frozen into the code-base. Thus our software looks prettier, but because of the complexity increase, is dropping in sophistication and quality. And it’s not in a big companies’ interest to change this trend. They seem to make more money with lessor quality code.

So what can we do? My first suggestion is that we should push to get more and more software into openSource. That is an easy win, transparency promotes quality and ease-of-use. But at the same time, we need to attach a price to every single piece of code out there. I don’t think home users and developers should pay, I like that they ride for free, but for big companies -- making profits from our labors -- money should definitely return to our community. Money that we can use to innovate with.

The problem is that programmers aren’t business people and few of them really want to deal with business. What they want is to build really cool stuff and leave the hassles of collecting money to other people who enjoy it. To allow this, I think we need to set up ‘software clearing houses’. Programmers would deposit their code into these organizations, and the staff there would deal with the issues of wheeling and dealing in the business arena. The clearing house could deal with licenses, accounts receivable and accounts payable. They would be the repository of the running code and of the source code. They could collect bug reports, then deal with farming them back out to the communities that built the code. Basically they’d act as a middleman between a large number of developers on one side, and a large number of companies on the other.

Many of our current licenses ask for funds when the code goes into a commercial product, but not if it is being used in-house. One reason for letting the in-house users ride for free is that not doing so would result in a huge number of little payments that would all have to be coordinated. That would be messy for an individual developer, but if a clearing house represented a significant number of projects, libraries, utilities, etc. most big companies wouldn’t have a problem paying a single reasonable yearly lump sum amount. They’re a huge number of ways of structuring this type of arrangement -- fine verses course payments, etc. -- but what is really important is that it isn’t a burden to the companies, and the money is flowing to the developers.

A significant problem with relying on many of the newer openSource libraries is support, both for bugs and for on-going rust prevention. A clearing house could provide some assurances that they will contact the developers and try to get the issues sorted out. If that is unfeasible they could also contact other unrelated developers and get the code fixed or updated that way. Once deposited in the clearing house, the code could live on well past the author’s interest. It would also be less subject to dramatic shifts in design or licensing. If enough people were interested in the preservation of a fork, the fork would find an easy means to continue.

One problem for commercial developers is the proliferation of various licenses for libraries. There might be a great library to use, but the license may be vague or destructive. Often approaching the developers directly, results in outrageous financial demands thus making it impossible to utilize the work. Commercial developers are keen to make profits, and aren’t against sharing them fairly, but the commodification of software has dramatically lowered the margins. It’s getting harder and harder to make a profit directly on software, the monies come more often from the services and support side, particularly for software categories like niche enterprise software (5-20 clients). Thus payments to the authors of dependencies would be fairly small, and constitute a considerable overhead if there were a lot of them. This again would be fixed by clearing houses. Lump sum payments, or per-sale payments that were sent to a single clearing house that then disperses them to a large group of developers would allow the money to flow. If all of the works were under the same license and the terms were reasonable, then that would easily drop out another big problem for the commercial developers.

Another important point is that there should be many clearing houses. Competition is a good thing, but also some of the houses may specialized in providing access to particular types of code. Some industries are highly regulated and a house that could provide certified libraries would be hugely appreciated. Also license and support features could differ significantly, as well the underlying quality of the code. A house that only provided vetted high-quality libraries for instance, would be a very useful entity and save lots of development time currently used to evaluate the existing options.

I should point out that to some degree this idea already exists. Both Apple and Google have markets for apps that act as middlemen between the developers and the consumers. This seems to be working quite well (although it does also seem to have reduced the price of software). What I think we should do is expand that basic mechanism out to all code, all of the time. Pretty much everything would go through clearing houses, and for everything that is usable there should be some cost to it.

There are lots of other benefits as well, but my readers seem to really hate it when I go on and on :-) My key point for most people is: why kill yourself in the evenings and weekends to do great work that may end up making other people money for decades, if you are not going to get some share of the pie? Write something, deposit it into a couple of different houses, and if in a few years that provides the means to retire early then you are free to focus on the code you’ve always wanted to write, wouldn’t that be a great thing? You’re happy, the other coders are happy, the business’s are happy and the industry is happy. Everybody wins.

The First Principle of Software

2012-01-27T18:03:00.000-05:00

The very worst thing that any computer can do is lie to you. You have to be able to trust it when it says it will accomplish some work on your behalf. If it tells you one thing, then goes off and does something completely different -- causing a huge mess -- you have the right to be mad. If it shows you deliberately incorrect or misnamed data then you should be pissed. Except during hardware failure, computers can precisely record, manipulate and display data without any of the frailties of human nature. Users should be able to rely on this.

Thus the very first ‘most sacred’ principle of building software is that it should never, ever, mislead the user in any way. The software shouldn’t lie to them. That sounds simple enough but the current state of our industry is to wantonly ignore this principle.

Most programmers think of ‘users’ as the end-users, but they aren’t the only ones who rely on the software. Operations people are users too, and so are all of the other programmers that are directly or indirectly leveraging the code. All of these people are on the front-line of utilizing a programmer’s list of instructions to the machine. Behind them often stands another huge group of people who rely on the information coming out of these systems. Thus even a simple problem can affect a massive number of people. There are far more primary and secondary users than most programmers realize. If a programmer writes code that lies to people, they could be lying to a really large group of people for a very long time.

There are many different interpretations of ‘lying’ -- our modern age has really lowered the bar for this word -- but it is commonly considered to be ‘intentional’ false information. However a computer itself has no hidden agenda. It is just happily doing what it is told. So if there is ‘intent’ it lies with the programmers. Sometimes the programmers that assembled the initial list of ‘instructions’ are trying to be malicious -- viruses are a real and present danger -- but most often the programmer’s lack of knowledge, time pressure or just plain laziness are behind the the false results. Given what we know collectively as a profession, it is clear that these are all preventable excuses, which leaves ignoring them as intentional acts. Thus a programmer who deliberately ignores any long-established conventions about building robust software is intentionally misleading the users.

Software can lie to users in many different ways including:

Misnaming the data.
Misnaming the code and/or functionality
Misrepresenting the status of work.

The first issue is the simplest. If you write code to store people’s full names into a system, then that data should be named something appropriate like ‘fullName’. That applies not only to the database that stores the data, but also to any interface that handles it at any time. Wherever the data is stored or presented, any associated names should be valid. Now if you put the data into a field like ‘address’, which in this case isn’t related to someone’s full name in any way, then the data is misnamed in the system. You are lying about it. Also, if you invent your own abbreviation that is non-standard and only known by you, such as ‘sFlNm then you are also lying, since no one else knows what it means and it is essentially garbage text.

Of course, sometimes we want the code to handle a more generic range of data, so to do this we can lift up the terminology to something like ‘name’, ‘text’ or even ‘string’. That type of categorical lifting isn’t misleading, it’s just using a more general term that still pertains to the underlying data.

In a system where all of the long-term (persistent) data is stored in a technology like a relational database, anyone should be able to go directly to the schema, and if they understand the data model, know exactly what each and every datum stored there means. Most of these data-store technologies have the capacity to support reasonable naming standards, old limitations like 8 character names have long since vanished into the dredges of history, thus there is rarely a valid excuse for not naming stuff appropriately. Its a matter of professionalism.

But it’s not only the names of that data that matter. Modern software relies on millions, if not hundreds of millions of lines of code, all of it whipping data back and forth around the system. Since the software industry is subject to on-going growth and change, very little programming code is ever just worked on by a single programmer. So if you belt out some code with cryptic variable names, then leave, you are essentially going to lie to the next programmer that unfortunately comes along. If you’re not lying to them, then its because they’ve junked all of your work, which isn’t particularly nice either. Methods, functions, configuration files, processes, etc. all require correct names. If a professional programmer has to name things, then those names have to be correct.

One of the worst things software can do is insist that it completed some work when it didn’t. This is easily the trickiest issue when it comes to lying with software, but solutions to mitigate or prevent problems have been around for nearly a half a century.

For instance, we do have ways of insuring reasonable transactional integrity that can be implemented in products. However, getting this type of code correct requires significant knowledge of the underlying issues, theory and practice. In reality, few programmers ever bother to access this type of knowledge, so they end up re-inventing broken versions of it. This hasty behavior shows up in many areas of software development including transactions, locking, concurrency and distributed processing. Failures in these areas, coupled with bad error handling, often produce misleading results. The software ends up lying to its users. Except in very rare cases, a professional programmer with the right prerequisite knowledge can avoid this type of lie.

With the exception of some unavoidable coordination issues, there is no reason for a computer to ever lie to its users. Someone may have typed in the wrong values foe the data but the computer should never contribute to the problem. It collects stuff, distills it and then shows it. It doesn’t need to distort what it shows in unreasonable ways. So it is pretty clear that the people creating the instructions for the computer should follow suit. It’s a matter of professionalism. We don’t need to misname data, obfuscate our code or reinvent broken functionality. These mistakes are avoidable and avoiding them is the foundation of providing usable software. Almost all violations of the first principle of software are completely unnecessary.

An Informal Ramble

2011-12-22T16:41:00.001-05:00

[ Author’s Note: I really don’t know what to say. It was just one of those days. If you don’t feel like reading a broad rambling on loosely connected ideas, please don’t read this! ]

A good place to start -- as always -- is with a few definitions. For this particular discussion it works better if I create them loosely and use somewhat non-conventional meanings. I’ll try to explain why that helps as I go along.

The first thing I want to define is a ‘system’. For this discussion it is a set of rules that act on ‘stuff’, where I intend stuff to be nearly as vague as possible. Stuff could mean static objects, or it could mean dynamic processes, or anything in between.

Systems can be broken down into two loose categories: formal and informal. A formal system rigidly constrains the stuff, so that it stays within the space occupied by the rules. That is, stuff is in the formal system when it obeys the rules, and it is invalid or outside the system otherwise. Thus the rules are ‘formal’ and essentially unbreakable.

Informal systems, on the other hand, obey the rules most of the time but there are plenty of things that can occur within the system that are outside of the rules. That is, an informal system loosely defines what most of the rules are, what the boundaries are, but there are always exceptions and somehow by some ‘stickiness’ these still fall within the system.

To put these two definitions in more candid terms: a formal system is black and white and an informal one is just shades of grey. Both are systems and have what amounts to deterministic behavior, but one only approximately follows its rules, while the other is bound by them.

So, why these definitions? Because we can apply them to many things within our world. The rigid abstract formalisms of mathematics are obviously formal systems. The rules are the axioms, the definitions, the lemmas, the theorems, etc. that are built up on top of the stuff which is the underlying well-defined mathematical objects. There are many different branches in mathematics and all of them have a plethora of underlying formal systems. Formal systems can also be built on top of other formal systems, by creating relationships between them.

So a very simple formal system in mathematics is a set of operators like: plus, minus, multiply and divide, that works on objects like: the set of all whole (positive) numbers. What is interesting about this system is that the subtract operator, when used on a small number and a much larger one, will produce a new number that is no longer within the whole numbers. That is 3 - 10 = -7, which is a negative number.

There are a number of ways to view this, but again I’ll go a little unconventional and say that the above formal system has an ‘exit’ point into a larger formal system. That is, if I had chosen integers, which include the negatives then there would be no problems with the subtract operator. Thus we can consider the first formal system as being an incomplete subset of the second, with respect to subtraction. We can also consider it to be an incomplete subset of one based on the continuum of real numbers, with respect to division. This gives us a large set of structural relationships between the various systems. These distinctions may seem a little weird, but they will come in useful later.

In contrast to a formal system, an informal one is not so well-defined. There are rules and there is stuff, but there is also a great deal of flexibility built in. That is, there are no real ‘exit points’. If something does not fit within the system, somehow it still seems to remain there. The types of informal systems I am thinking of include all different sizes of groups of people. Companies, clubs, governments and all other organizations are informal systems. They have rules which bind them together, but do change from time to time. Other examples are markets, schools, disciplines, professions, soft sciences, etc.

You may note that all of the informal systems I’ve listed so far are based around people. That’s more of a coincidence rather than a deliberate intent, for there are plenty of systems out there that we have tried to rigorously formalize but our underlying knowledge is incomplete, or the basis of the system is just too chaotic. Evolution, physics and weather are good examples. These too are informal systems, but our underlying comprehension of the rules is gradually approaching formalization. That is, rather than an exit point, outliers get feed back into the informal system to enhance our understanding of it as it approaches, but never quite reaches, formalization.

And this brings up a very important point. At the bottom of everything, as far a s we know, are particles. And although we don’t know all of the rules that define their behavior, we are pretty sure that the rule set is static. Thus there is one underlying formal system that quite possibly binds everything. That is, nothing can happen unless it is a physical possibility. On top of this we get ever increasing ‘epiphenomenon’; larger and larger collection of particles that at their collective levels behave within a set of what are essentially meta-rules. So we get galaxies, planets, stars, etc. And built on these we get even more: water, earth, plants, people, etc. And on these we build up even higher to stuff like buildings, farms, etc. And some epiphenomenon that are less physical, but still bind together stuff like organizations, companies, countries, etc.

So it would seem that for all of our informal systems, they exists within the bounds of lower and lower systems that eventually fall back to one rather low-level formal system.

What is rather odd though, is that we are also aware of the multitude of mathematical formal systems out there, and these seem to exist without any ties to those based on the many layers of epiphenomenon ones. We do however, use them to explain the others. That is, physics is a set of formal systems that are remarkably accurate in explaining the physical world around us. So in that very sense, mathematics is the meta-formality that binds together the epiphenomenon with some degree of precision. Thus it has its roots in our physical world, even if it just appears as an abstraction based on our intellectual thinking capabilities.

Any real argument about the how tangible the formal systems of mathematics are has to take in consideration computers. For the first time, our species has manager to instantiate one of our previously abstract-only formal systems -- Turing machines and/or lambda calculus -- into a physical reality. Well almost, since computers are still bound by finite resources and by the occasional hardware failure. Regardless, it is still a tremendous accomplishment and their existence has been rapidly transforming our societies ever since. One might argue that how physics models the world is similar, but we need to consider that that link between math and reality depends on a person utilizing the models in their head, in order to act on the knowledge. Thus the formal systems that underpin physics stay internal to our minds. Computers, on the other hand, do their computations entirely independently of people, and thus are a concrete physical manifestations of a formal system. A very different ballgame.

So why are these definitions of formal and informal systems useful? One of the main tasks of software developers is to construct useful formal systems that solve problems for people working in informal ones. That is, any software we write must be formal in its construction, but its usage lies in one or many informal systems.

If you are writing a system to track how a company deals with its clients, the company and the behavior of the clients is entirely informal. Most things happen according to plan, but lots of exceptions occur and the nature of the relationship changes as the market expectations change. But those rules need to be formalized, so they can be coded, so the computer can execute a deterministic series of instructions with each interaction between the client and the company.

From this angle, you can see the problem right away. The informality of the base systems leads to difficulties in finding a formal one that both precisely matches them and remains matching for any prolonged period of time.

There are ways around this. The underlying formality of the computer still allows for it to be dynamic with respect to its resources. The resources may be finite, but these days memory and disk are so large that their finiteness isn’t a significant factor. So, instead of coding a large number of finite static rules to contain the system, the designers find dynamic ones that bend and shape as the underlying informal systems move about. One of the great failures of software development these days is to ignore this reality most of the time, and rather just blame the inherent mismatch on contrived issues like ‘scope creep’.

To simplify this discussion somewhat, we can combine any underlying informal systems together and take a subset. This somewhat monstrous informal system is the object that we need to mirror with a formal software one. As one could easily guess, this type of combination can easily be contradictory or incomplete. And that is an all too frequent reality when building software systems.

While the relationship between software systems and the informal ones that the software is trying to solve is interesting, these days I am more interested in the general behavioral characteristics of informal systems by themselves. One thing we can do is use the size of an informal system as the metric of the underlying complexity. It isn’t the only way to view complexity, but it does prove useful in understanding how and where the complexity grows.

As an example, we can look at law. Since we’re generalizing, we won’t distinguish between civil and criminal law, nor tie the discussion to any specific country or international organization. Law is basically a set of rules that the people of a society have agreed to follow. There are often a lot of rules, and generally through government action they are always increasing. The laws define which actions are right or wrong but that is really a black and white view, where the world is rather grey. So along with the laws are the precedents from legal judgements that together often define how the law is interpreted and applied. Both of these can also rely on a vast number of legal definitions, that do not necessarily match the more common definitions used by industries or people. To add to this, some laws or interpretations of them have never been fully tested in the courts, so there is some ambiguity as to whether or not they are valid.

Thus the informal system that binds the laws of a country consists of the laws themselves, the history of precedence and some unknown guesses or assumptions about what is, and what is not, legal. And this is constantly being added to with new laws, or new interpretations of the laws. Thus it is perpetually getting larger all of the time.

At least from the outside there doesn’t seem to be any real mechanisms to reduce the size of the system. I’m sure there are some instances of people replacing laws with simpler ones, or laws just getting ignored, but in general this does not seem to be the trend. The system just gets larger every year, there are more lawyers and judges, and the complexity get worse. New interpretations that form precedence get added, but the old ones hang around and can always be dredged up again.

The irony is that at some point the sheer size of the system prevents people from being aware of the full scope of it. That seems to contradict its usefulness in being there as a common set of rules that everyone in a society should obey. Why have rules if most people are not aware of them, or they are not properly enforced?

In a formal system, one could likely find an equivalent system, but with a less complex set of rules and then reduce it to it. In an informal one, while that option may exist, it doesn’t seem to be the path chosen. The system gets larger, it get more convoluted, and any earlier discrepancies instead of getting fixed just get built upon for the later works.

An insider’s perspective on a informal system is generally predicated on familiarity. They understand the rules, or at least a subset of them, and they accept that this is the way it was, and the way it will be. An outsider’s perspective, particularly if they need to dig down into the depths and see all of the ugly warts and bumps that are being ignored by others, is quite different. They can see the twistedness of the informal system for what it is. The gradual accumulation of the rational and irrational, over a long period of time.

For software developers, if they are building systems to handle very specific domain problems, they are the outsiders. That is, they come at it from a distant perspective, and they get to evaluate the rules based not on history growth, but rather a rational logical foundation. They have no choice, they must map the informal system onto a very rigid and precise formal one. Any impedance mismatches between the two systems is both obvious and problematic.

That does give developers a rather unique point of view. We’re not looking at informal systems like law from the vantage point of acceptance. We’re looking at it from the vantage point of complexity, and mapping it to something simpler. Software systems, almost by definition, are going to be simpler and less complex than the informal systems they are mirroring. That’s a product of the sheer amount of resources to build them and keep them running, but also because they generally focus on just a small number of problems contained within the informal system, not the full system itself.

From all of my experience digging to many different informal systems there are a couple of things which really worry me. One is that virtually all of the informal systems I’ve seen contain some type of irrationality. There is always at least one thing there that is just not the product of rational thinking. It just gets built up at the cross roads between other parts of the system. Nothing that would even be constructed deliberately. It’s easy to see these because any domain expert explanations are contrived, messy, hidden or seem to change. That usually points to some underlying irrational component.

The other thing that worries me is that there is always a way for the system to grow, but almost never a way for it to shrink. One can see the consequences of this all over our societies. Everything gets more complicated and more convoluted. People try to justify the growth, but these often are just hollow excuses.

One might consider computerization a simplification method, and sometimes it is. It is not infrequent that implementing a new computer systems means changing the process to some degree to fit the new limits of the code. However development of systems often goes on for decades, and gradually the complexity heads back to its former glory.

Computers in general make it easier to manage complex informal systems. They can do a lot of the grunt labour for people, and they can do it fairly accurately, if they were correctly instructed to do so. We can see this shift as well. Part of the transformation of computers has been the ability to manage informal systems of such complexity that they would have easily collapsed without the aide of a computer. Overall, however, I’m not sure that this is a good thing.

A while back I read in a blog somewhere, that once a complex informal system gets large enough, the only remaining option is for it to collapse. That is, it falls apart suddenly and dramatically. History seems to confirm this view, as we have seen the rise and fall of many a great civilization. At some point the system can no longer increase, it can’t remain as it is and there is no ‘release valve’ for complexity. There are no other options. It basically implodes on itself.

That notion, if true, does not bode well for our current societies. We can see that our complexity has dramatically increased over the last half century, and we can see examples of systematic failure showing up at ever increasing rates, but we seem to have no viable paths of fixing what is wrong. We should be concerned.

If there is some way out of our historic fate, my sense is that we would need to rely on both computers and aggressive simplification to get there. Computers, because they can show us alternatives to our current informal systems, and help us model and control the side-effects of changing what are essentially chaotic systems of n-dimensional variables. And with that initial knowledge, we can start the slow gradual changes needed to simplify our informal systems. The changes have to be slow, just because in general people don’t adapt well and their fragility generates another form of complexity. No doubt it is easier said than achieved, but given the alternative of imploding it doesn’t seem that bad.

There is lots more I could say about formal and informal systems. Defined this way, they provide a general underpinning for a great deal of what happens in our lives. And along with the way we model things internally they help to organize the way we see the epiphenomenon around us. Perhaps, someday I’ll get more of a chance to delve deeper into this area, although all things considered, keeping up with modern life’s requirements is probably too time-consuming and complex these days for me to get another chance :-)

The Engineering of Software

2011-12-19T17:50:00.002-05:00

The last thing I ever wanted to be -- twenty-six years ago -- was an engineer. When I started university I had heard that 70% of all engineers hate their jobs and that they were often bored at work. Boredom sounded horrible.

What I wanted, was to master the black art of coding. To follow in the foot steps of those early magicians I had seen in the computer magazines. The ones that crafted weird cryptic concoctions to do really neat things to their machines. I wanted to be a hacker.

It’s funny how time and experience temper one’s thoughts; how your actions change once you really understand the consequences; how much you grow once you see the larger picture.

Development shops span the range between the cowboys, who are just hacking at the code furiously, and the bureaucrats, that are so ground down in organizational paperwork that little actually gets accomplished.

In my younger days the cowboys where fun. It’s a dynamic environment. Things are happening. And they are happening fast. Well, OK, there are a lot of needless problems cropping up, but hey, that just adds to the excitement, doesn’t it? You could leave work feeling like you got something accomplished.

Well, that lasts for a while. Until you start to get caught in the same hollow accomplishments, over and over again. That is, you’ve fixed the same bug before, wrote the same code before or stuck a band aid on the same gaping wound before. Doing it once or twice is fun. Doing it repetitively, while giving up your life to endless overtime, starts to become depressing after a while. Then suddenly one day you find yourself looking down at the growing pile of band-aids before you and you start thinking “isn’t this all just a waste of my time and my abilities?”

Early on I moved away from the cowboys. They were driving me nuts. I landed deep in the heart of a reasonable engineering culture. Some people on the outside might have confused that development process with a classic waterfall one, but it wasn’t really. Although there was a significant design process up front, the last of the details were always worked out in the various iterations. It was actually a healthy mix. Some thinking, some paperwork, then a hard run at the code. Some testing, a release and then back to thinking again.

Later when I set up my own shop, it naturally started as chaos. Startups are sensitive to cash-flow problems so sometimes the only way to deal with that is an ugly short-term hack. But as time progressed -- every time I got the chance -- I applied long-term fixes. These gradually paid off and when I finally moved on, the place was humming along smoothly releasing a steady stream of mostly high quality code and the occasional patch within a day or two of finding a bug. Each development cycle was a bit easier than its predecessor. Everything was tracked. We had managed to built a big application. One that was far larger, and far more sophisticated than you would have expected from such a tiny shop. I learned a lot in those years.

I found that it isn’t all that complicated. If you boil it down again and again, software development keeps coming back to its roots in engineering. Not in the classic sense that I was afraid of when I was younger -- the large, slow, boring, mindless process -- but rather in the sense that as the work progresses, developers like myself needed to be hacking less, guessing less, and understanding more. The project may start with a lot of unknowns, but at some point it always gets down to routine engineering. That is, we should be doing the ‘right’ things that work because we know that they work, and fixing the little problems -- permanently -- before they become big serious ones.

My earlier fears were misplaced. Engineering itself is not boring. I’m not even sure where I heard that statistic about unhappiness. Engineering, particularly when it comes to something like software, is simply knowing what works and applying it correctly so that the project is successful. It isn’t hack and hope, it isn’t wild guesses, and it isn’t a spin job to cover up the fact that the system is horribly broken. No, quite simply, it is the path to dreaming up cool solutions and then implementing them as cool solutions.

Yes, there are boring times. There is a lot of digging to see what is already out there. No sense reinventing the wheel especially if it’s already decades old. Note that that doesn’t mean you can’t recode the wheel -- making a better one -- but rather that you don’t try to rethink the wheel, that work is already done and accessible.

There is sometimes paperwork. But if one focuses on documentation that’s useful, rather than on the documentation a management consultant would want, then it’s not so bad either. It helps to refine ideas and highlight missing corner-cases long before they become too serious to fix. A little design goes a long way and any organizational chaos, at any level, is always going to grow into a serious problem someday. What you’re missing, or not thinking about, often becomes really obvious when you try to write it down. It also helps you to understand the vocabulary of the problem and to empathize with your users (if you can’t easily explain it in a document then the users aren’t going to be able to understand it either).

There is a lot of thinking. And ironically, although programmers preach heavily about their need for freedom, many choose to turn off their minds when they code, resorting to as much brute force as possible. But at the end of the day, code is just the by-product of thinking about the solution, so it is only as good as the thinking that went into it. To build sophisticated things you have to have a deep understanding of the problem and you have to spend a lot of time working through it. Thoughtless code is dangerous code because of it lacks intelligence. It just causes stupid problems.

And of course, if you’ve really set a firm direction and you know where you are going with the development then the rest of it is just hard work. Little issues have to be sorted out, details need gathering, but the main flow of the development can be worked out in advance. The project may drift but either the long-term perspective still holds, or it is rethought again to insure that it is still valid.

I remember reading decades ago in Scientific American about how the pharmaceutical industry moved from art, to science and then on to engineering. The pills we pop these days are highly likely to contain exactly what they say they will. That wasn’t always the case. It took them a long time to go from alchemy to engineering. The article said that software also started as an art, and has some scientific basis, but it will eventually follow the same path to engineering. I remember the article because I figured I’d bail when that happened. Sounded boring. But these days, as I am surrounded by more software with increasing complexity and decreasing quality I realize that what I want most right now is to get a whole lot more of the engineering mindset into the software development industry. It is time, I think, that we started taking our profession seriously.

There is still room out there for the hackers and the cowboys. Sometimes a band-aid is the best choice, or the solution is so small it doesn’t matter how much of a mess it is. For quick one-offs I think hacking will always be common, but when I go out and rely on someone else’s code, on their thinking and on their intelligence, I would like to know that they weren’t just mindlessly flailing at the keyboard or that they skipped their homework and just tossed it together. It makes a difference. I don’t want to build on a house of cards. I don’t want to find out later that I’m dependent on somebody who didn’t really ‘get it’ at the end of the day. I don’t want that degree of instability in what I am doing. I can’t build great software on top of a mess; dealing with other people’s mistakes eats through my resources like a knife cuts butter.

So these days I am all in favor of us seriously moving into engineering. Not doing so is a major obstacle that has prevented software from living up to its true potential. I think we can do it, and do it in a way that it is not boring and it isn’t just paperwork. We can still build quickly, but we can do so from a place of understanding, not one of panic. It’s about time that software development grew up and became a real profession, one that is taken seriously. Even in our deplorable state we are changing the world, so can you imagine what we could achieve if our collective works didn’t suck?

Getting to the Truth

2011-12-10T20:02:00.001-05:00

One of the most enjoyable things about building software is that it requires the developers to dig around in other domains. If you’re writing financial software, you need to know how the industry works, understand the terminology, and all of the complex details underneath. If you build something for the printing industry, you need to grok their equipment and business models. Everything you write depends on knowledge from elsewhere.

For a naturally curious person like myself, this is a huge benefit. Basically a license to try and learn as much about the world as I can.

Every domain I’ve ever worked in has had a fascinating array of history and depth. You can’t actually solve any of their problems with software until you really understand them; until you get down to what makes them tick. Get to the roots. Get to the truth.

Software is very rigid and to automate any processes they need to be fully understood. To collect valid data you need to understand its structure, quality and volume. To make the interfaces usable you need to understand the environmental issues and the people. If you don’t get to the truth, the software either won’t work or it will cause more problems than it fixes. If your goal is to actually solve something, the only place you can do that is in reality.

But getting to the truth is a funny business. There are all sorts of people out there that will stand in your way. Sometimes it’s because they have something to hide, sometimes it because they don’t want to think about it. Sometimes they just don’t care or figure it isn’t necessary. The causes very.

Lately I’ve come to think of these people obstructing the work as ‘confused’. They are confused about what’s happening below, or above, or why you’re trying to find out, or even what value the software will bring to their lives. The reasons don’t matter, they are just confused.

Building a system on confusion isn’t any better than building an apartment building on a foundation make of foam. It’s obvious that it’s not going to work, even if it looks promising for a moment. If you build on fantasies or delusions or ignore hidden secrets, the result may run, but it’ll quickly drift away from being useful. It doesn’t matter how much data you collect if the data is scrambled or incorrect. And if you ignore the people who ultimately use the work, then they won’t use it or they’ll use it badly, causing massive support headaches. A forced, but miserable user-base can disruptive.

Software has the power to transform, but only if the embedded intelligence within it is correct, and that only occurs if the programmers were exposed to the truth. No truth, then there is no intelligence, and so the results are just burning through wasted resources. This is the fundamental rule that binds the code to something useful in the real world. We’ve known this for a long time, referring to issues like ‘ivory towers’ when talking about programmers who write systems far away from their users.

But this can also happen when the business or management players seek to block access to the truth. And it can also come about as a failure to properly complete reasonable ‘business analysis’, or because the domain experts are confused or lack enough experience. There are a multitude of different ways that confusion can enter into the process and obscure the truth. Distort reality. And without enough truth there is no understanding, which always prevents success.

Problems and Solutions

2011-11-30T13:51:00.001-05:00

The world is littered with a growing number of problems. Some of these problems are solvable by using a computer, some not. To solve a problem with a computer all it takes is to design and construct a software solution, then get people to use it. Easy?

Solving a problem with software is not nearly as simple as it looks. Many more solutions fail then succeed. The stats are ugly: 66% of all software projects fail, as do 90% of all startups (most of which are founded around software solutions). Why are these numbers so high?

I think the root of the problem is that a good software solution looks simple. It’s very easy to miss the depth of both the thinking and the work that have gone into making it possible. This leads to a lot of people casually tossing around ideas for solutions, with the expectation that they’ll actually solve a problem. But few of these ideas actually do, and in that very limited remaining subset, the execution and the details mean everything.

To actually solve somebody’s problem with software, the developers needs to look long and deeply at the underlying problem. They need to understand it, not only from the 10,000 foot view, but also right down to each of the tiny details. A shallow understanding of the problem is not nearly enough to solve it. The real issues are often counter-intuitive, hidden and not easily grasped.

Even more complex is understanding the nature of how people use technology. Without some type of incentive, many ‘features’ of a system are useless. Making software ‘sticky’ is a rather complex combination of various issues including stability, usability, psychology, feedback and empathy. Again a broad perspective is not enough to understand these depths.

As well, the developers need to understand the uses and limits of modern technology. Software is still fairly crude and marketing claims for it rarely match reality. Technological issues are often deeply rooted in their theory and history, so both of these need to be well understood before you can properly assess the limitations. You really need to understand these limits before you can apply the technology to solve a problem.

Running a system development project is no picnic either. Software development is like a large train moving slowly down the track, from station to station. You can’t just throw in a right turn here or there. You can’t really change its direction until you hit a station, and even then the number of other accessible stations is very limited. It is easy to miss these inherent and often dangerous properties.

Right down at the core, software is about programming. But just being able to program is not enough either. Some code is good, while some is just a disaster waiting to happen. Experience teaches one more about the code they shouldn’t write, then about what they should. Knowing what’s disorganized, or fragile, or outright dangerous isn’t well documented and sadly only comes with a tremendous amount of experience. Hacking one’s way through a problem space won’t work; won’t produce enough of a stable, coherent solution. Just a convoluted mess. Really understanding what that means in practice takes a lot of time and a lot of experience. It’s not something that people can just read about, or take a course in, or figure out on the fly.

So why is the rate of failure so high? Easy, most of the people out there proposing solutions don’t have a deep enough understanding of what works and what doesn’t, to be able to propose valid solutions. Most programmers are viewed as just ‘code monkeys’ whose task is to build someone else’s solution. So the success or failure of their labours depends heavily on the quality of the proposed solution. A crappy solution will either fail outright, or limp along for years. A good solution can be ruined by a lot of crappy sub-solutions, slowly mutating into something horrific.

It’s for this reason that I often distinguish between programmers and software developers. A programmer can code, but a software developer knows how to solve real problems correctly with software. There is a huge difference. It’s also for this reason that I’ve never liked the term ‘stakeholder’. It seems to imply that they have the right to choose the solution because they have a ‘stake’ in the outcome. But if they don’t have enough of a background to pick a valid solution, then the work will fail. I’ve also seen a great number of entrepreneurs out there with the belief that as business people they should be the ones to pick the solution. That somehow a business background is enough for them to fully grok the problem, the people and the technical issues. This is probably why most startups produce software that doesn’t actually solve any real problems. It just looks shiny and new. So they fail.

The takeaway from all of this is that if you want to create a software solution to solve people’s problems, you need to give the task to someone with experience in creating working solutions with software. An experienced software developer who understands the history, limits and possibilities of software. If you leave it to someone with no experience, the solution is unlikely to be valid.

The Big Idea

2011-11-22T17:47:00.001-05:00

Way back when I was a co-op student, I got a job at a small company that built analytics applications for statistics. My previous co-op employer had laid off the whole division, so I had to find a new gig. There were lots of choices back then, so I picked the company because I would get experience programming in C. It seemed like a good idea at the time.

At first the job was great. I got to work on an interesting little problem for a product about to be released. But as soon as that dried up, the trouble began.

One of the senior statisticians had been asking for resources so he could get a pet project done. He sat way back in a corner office, I hadn’t really noticed him before. In hindsight the location of his office should have been enough of a clue.

The re-assignment started out OK. He had a bunch of little programs he wanted written in C. Nothing complex, not too hard, and the only big challenge for me was the using the language well enough. I had experience in programming Pascal, but C was a little trickier.

My problems started when he begin explaining his ‘big idea’. Variables, he said, where really hard to create and name when coding. It takes time to think up the names, and then you have to remember the names you thought up. Instead, he proclaimed, the secret to great coding is to declare a bunch of global arrays at the top of each program. One for each data-type, e.g. int, float, double, char *, etc. Then all you have to do is index these to get to the correct variable. That is, if you have an integer variable that holds the number of iterations, all you need to do is remember that it was your seventh variable and type in ‘integer[7]’ and you’ve got it. No need to remember some complex name, you just need to know the type and when you created it. Simple.

He didn’t really understand why I recoiled in horror. This was, of course, the most absurd thing I’d ever heard in my short little career. I understood just enough to know why this wasn’t just a step backwards, but rather a quantum leap backwards. It was ‘off the charts’ crazy as far as I was concerned.

I tried to explain why this would render the code completely unreadable (and probably full of bugs), but I was just a student programmer so he dismissed it. I ignored his directive to implement it this way and wrote the code normally, you know with real variable names and scoping as tight as possible. But I wasn’t used to C and I used some of its rope to hang myself (my code was worse than flaky). And so my failure re-affirmed his belief that he was right. The situation grew worse.

Fortunately I was saved by the co-op work term coming to an end. It was a horrific experience, but on the bright side I did end up with C on my resume, which lead to my next job (where I apprenticed with someone who actually taught me how to code properly).

I learned a lot of different things from that experience, and even after more than two decades I’m still learning new things. But the most important of them was how easily it was for someone to get it horribly wrong. Although he had written far more code than I had at the time, he hadn’t written nearly enough to really understand what he was doing. A big idea is far away from actual understanding. But without the latter, the former is unlikely to be reasonable. And unfortunately for us, software development appears simple to those who haven’t plumbed its depths. Thus we get a lot of big ideas …

Thinking About It

2011-11-09T18:09:00.001-05:00

That dreaded moment comes in every software project development cycle. The plans are too ambitious, the time too short. After an exciting start, the groove descends into the valley of just plugging away, day after day, while trying hard to keep to an impossible schedule.

It comes every time, on every project (if it doesn’t please, please let me know where you work :-)

What I’ve witnessed over and over is that this moment pushes programmers into speeding up their pace. They convince themselves that what’s needed -- what would make it better -- is more code. So off they go, coding faster and faster, cutting more and more corners.

But that sense of panic is working against them. And against their goal of getting the software released.

In an abstract sense, one can view software development as the act of ‘thinking’ about how to solve problems in a robust manner. The code itself is just a by-product of that thinking. The work is problem solving, but the final output is code.

When you speed up programming, it comes at the cost of reducing thinking. The programmers just fall back into writing the most basic rudimentary splat of code that appears to run. Because of the rush, that code is inherently fragile; the little niceties and less obvious corner-cases are always the first casualties. Spaghetti is a real possibility.

Even worse ‘code re-use’ goes completely out the window. So now the system is rapidly getting filled with redundant code that is fragile, inconsistent and most likely convoluted.

As this first wave results in a mounting bug list, it gets plowed under by even more waves, each one going faster; each one with a deeper sense of urgency. But each one causing more damage. The bug list explodes, the specs change and the technical debt swells to some epic proportion, often ending the project or at very least, resulting in an unstable release that is a nightmare to use or upgrade.

And thus speeding up the coding is a recipe for failure.

What does need to happen is to either speed up the thinking (if that is even possible) or start slashing the goals. Deep, but well-thought-out cuts (to avoid painting oneself into a corner) can push the excess work into later cycles. So long as enough thought about the future is applied, the added technical debt can be minimized. Meanwhile the development pace should be maintained. Evenly.

Sure it’s scary, but with experience this point in the process can be anticipated and even managed properly. A code panic, on the other hand is most likely to lead to ruins.

Problem Solving

2011-11-01T11:44:00.002-04:00

"Mainframe guys?! What do they know? They’re dinosaurs, the industry passed them by years ago...” was my response twenty years ago, when I first got out of school.

Someone had recommended that I take a look at how the mainframe guys (and gals) had solved a particular problem way back, while I was still just finding my identify in high school. But I dismissed it far too easily. Things, I felt, were changing too far, too fast.

In hindsight, of course, it took me a long long time to understand how wrong I really was. There were indeed things that I could learn from all of that earlier intellectual work. Problems that were solved, done and dusted. And what I truly failed to grok was that the work was done when there were less expectations, and more of an ability to focus longer and deeper on getting better solutions.

“Don’t reinvent the wheel” isn’t about using libraries or other existing code. It’s about not re-solving problems that in many cases were solved decades ago. And even deeper, it’s about not just quickly solving them in a brute force manner if there is already an elegant solution that has been deeply thought about and made available.

Still, being young, I wanted to solve problems right away and I guess that orients one to search for low hanging fruit. But inevitably, as there has been a tremendous amount of water under the bridge already, pretty much all of the low hanging fruit has been tasted by others. Many others.

The trick then is to spend some time first, to see if the problem is really new and unique. If it is, great -- you can learn from what people already know, but if it’s not then avoid falling into the trap; try to utilize what was done before. Under time pressure that can seem time-consuming and dangerous, but reinventing really common wheels is a guaranteed time sink that will only make the time pressures worse. After all, it is hubris to think that one can do a better job in a week of two of hacking, then was done decades before with thorough analysis and experimentation. Most solutions for low hanging problems out there are far better than anything we can dream up in a rush. The time spent learning them is time saved from falling prey to well-known issues.

But alas, I think it took me about 15 years to figure that out ...

Layering, Architecture and Convergence

2011-10-26T14:13:00.000-04:00

The central problem in software development is managing complexity. This complexity comes from the sea of details and decisions, large and small, that need to converge into a stable, working system in order for it to be successfully used for its intended purpose. That not only includes the code, but also the analysis, management, documentation, packaging, distribution, training, support, configuration and operations that inevitably precede a user being able to utilize some functionality on a computer.

Unchecked, rampant complexity growth in any area can swamp a project, making it impossible to continue substantial progress. About the only way I know to contain complexity is by breaking it off into manageable chunks and getting it encapsulated into well-defined sub-problems. This works at a base level, but new problems emerge. Overlapping sub-problems create redundancies which add significantly to the complexity. This can be managed typing and categorizing the sub-problems carefully, so that overlap is rare or negligible, which gets one past the first hurdle. But as the number of sub-problems grows, their inherent complexity needs to checked as well. At this upper level, they just form a slightly higher, more abstract version of the overall problem, but all of the same issues apply.

Thus, at some point in order to manage the complexity another higher layer becomes necessary and the same issues repeat over again. As the complexity grows, the number of layers to manage it grows as well. Thus, in any attempt to manage growing complexity, a steadily increasing number of layers will need to be created. This is the inevitable consequence of our existence, we can’t handle unlimited sized problems. Smart people or well-running teams may have higher limits, but there is always some limit, and it is always reached earlier than expected.

Layering occurs in all aspects of software development, but for this post I’d like to just focus on how it effects the code, data and operations of software.

The first and most obvious layer in software is the underlying code itself. Programmers get presented with a number of domain or technical problems for which they assemble a large number of instructions for a computer to follow. It is well understood that failure to organize these instructions results in ‘spaghetti code’. That is code with a logical structure so intertwined it resembles the noodles in a plate of spaghetti. Of course this is bad, because any attempt to alter the code is hampered by a significant wall of complexity. From this, within the context of coding, we can draw the conclusion that dis-organization increases complexity.

As the code base begins to grow, it becomes necessary to break it off into more and more sub-problems to manage it. This layer is most often referred to as system’s architecture. It introduces conceptual groups like modules, sub-systems, components, etc. Each of these pieces encapsulates a larger subset of the functionality, and the interactions between them gets formalized. Although I’ve never seen it mentioned, dis-organization at this level can produce a spaghetti architecture, one that impedes both changes and overall stability. Experience coding in the base layer does not easily translate into understanding the organizational issues at this layer. They appear similar, but they are fundamentally different problems.

Once a system has grown out of its base components, it is often desirable to widen its functionality to a larger group of developers. This creates essentially a platform layer where the core abilities are exposed, and controlled, allowing for the existing work to be leveraged for more divergent purposes. Once again, failure to organize this layer results in more pasta, and of course understanding the dynamics of the necessary organization is not directly related to the layers below.

Continuing upwards we get to the product or system layer. This layer generally introduces a new set of problems because it is driven more often by the desires or needs of non-technical people. Organization at this level comes more as ‘paths’ that are followed serially or in parallel, but they still need the same generalized principles of organization. Experience at this level generally involves far more people and negotiating skills, than technical ones. A product manager for a commercial system for instance is balancing the technical issues against the direction and current of the market place.

The final layer that I’m going to discuss is the one that is completely ignored by most technical people. For users to access the system, it has to be running somewhere. In an effort to contain the complexity of design and work most systems form vertical silos. That is they handle a very specific problem and don't easily inter-operate with the other systems being run in their environment. There are some interconnections, but standards are weak because they interfere with individual goals or profit margins. As such, most modern large organizations have a large number of these silos. Like every other layer, redundancies cause significant problems. Thus, at an operational level, all large organizations need an organizational scheme to coordinate the building or purchasing of these silos into a manageable affair. This drives concepts such as the common enterprise software ‘categories’ like CRM and CM, which implicitly contain the primitive building blocks of the operational environment. Again, dis-organization at this level results in an enterprise wide plate of pasta, which diminishes that usefulness of the tools.

No doubt there are many more layers, since they are dynamic and unlimited in number. For any given layer, as it gets better organized we can say it is ‘converging’. By this we mean that in some overall way, its base complexity matches closely to the underlying ‘necessary’ complexity of the layer. If it’s not managed then it diverges, generally as a result of disorganization, over-engineering, over-simplification or any other amplifier that isn’t fundamentally inherent to solving the problem.

It should be noted that there is a subjective (human-based) component to minimal complexity. That is, for reasons based on the way individuals think, the lines between over, under and minimal complexity tend to move around. One person’s perfect trade-offs can appear overly complex to another person. Thus rather than a fixed point, a layer will generally converge within a range, particularly if there are many people involved.

Thus in software development, in order to manage complexity, we want to see the overall system converge on a reasonable number of layers, while each layer is converging on a reasonable system of organization. Development is an ongoing progress, and thus so is converging. We simply need to track all of these ‘paths towards organization’ and continually insure that they are headed for the correct goal. At the bottom layer that is accomplished by programmers pushing themselves for elegance, while at the upper layers is it often architects or managers trying to place some organizational constraints on the chaos below. Every layer needs to get close in order to win.

Contradictions

2011-09-13T18:00:00.001-04:00

Programming is easy. You just assemble large lists of instructions for a computer to follow. And these days there is an abundance of information out there about which combinations of instructions will produce the best effects. If you can’t figure it out, there are plenty of people to ask.

But programming is hard, because there is never enough time to assemble all of the instructions yourself so you end up relying on other people’s lists. And often, the behavior of their collections doesn’t match your expectations. They skipped functionality or built it weird, or just hacked it together without thinking.

But it is easy because there are plenty of the stable ‘components’ that have been around for years or even decades. There are loosely-followed standards and although some are quirky, once you understand how to use them properly they’ll do what you expect. It may take a while to find the ‘grain’, but once you know how to utilize something, you can make it work for you.

But then it’s hard because the more stuff you build on, the farther away you get from what is happening underneath. This detachment often causes people to over-simplify their understanding, leading to invalid assumptions about the way things really work.

But it’s easy again because there are plenty of conventions, knowledge and advice out there to follow. Sometimes you have to dig a bit, but it’s out there somewhere. At some point, someone has delved into the details and has spent time explaining them. It just might take a while to find and assemble that information, but if you take the time to learn how to do it properly, you can skip a lot of the frustration and pain.

But then programming is hard, because it often attracts people who want to utilize it, but they don’t have any real experience or patience to learn. They over-simplify the amount of effort involved, and think that short-cuts will get them there faster. Looking at a screen, many people think that the work involved is close to what they see. But it’s in what lies beneath that all the details and complexity brew. The screens are the easy part, particularly if you follow the existing user interface conventions.

But it can be easy. If you spend the time to learn it, then do it properly, and gradually work through all of the issues one-by-one until they are complete. Eventually you’ll get there. A good solid organized process with no cheap short-cuts will always result in something that is usable. It may just take a while.

But then we’re back to hard, because to overcome other’s lack of experience and impatience we need to point them to an authoritative reference. Somewhere that explains the basis for getting the work completed properly. There are little clues spread every where; some correct, some wrong. But no ‘building code’. Without that, new people coming in just ignore what was known for decades, without realizing that it’s been worked out already.

So is programming easy or hard? When the people around you accept it as hard, then although it may take a while, it doesn’t have to be painful or a mess. But when the people around you think it is easy, they get impatient and take really bad short-cuts to avoid the necessary work, so it becomes increasingly difficult to get anything finished properly within the chaos. Thus it all depends on whether the people around you get that in order for it to be easy, they really have approach it like it is hard. Or slightly restated: it is easy right up until you fall into the trap of thinking it is easy ...

Know your Limits

2011-07-28T15:29:00.000-04:00

I’ve taken this week off from my day job to work around the house. It’s a yearly ritual -- my house is over a 100 years old and in constant need of repairs. This time my main focus is the front deck. It’s 16 feet in length and the top is about 8 feet deep. There are four deep steps running the whole length of the deck. It was probably built at least 20 years ago, but I suspect that it might be way older, perhaps as much as fifty.

Four years ago I rebuilt the inside of the deck because it was starting to rot in places. I put back the original covering boards since most of them were in reasonable shape. This year we decided that we wanted to get it painted, but many of the original boards we’re starting to go at their ends, so I figured it was time to recover much of the deck and the stairs with new boards.

Sometimes, when I am working on a project like this, people look at me strangely. I’m a bit geeky looking and their initial reaction is to get concerned when they see me playing with power tools. I guess they expect me to end up injuring myself, but I’ve actually been doing this sort of work since I was young and have plenty of experience with big tools of all shapes and sizes. So far I haven’t managed to nail my feet to floor. I guess people think that software guys are supposed to stick to virtual bits. But for me “building is building” it is only the medium that has changed; the work remains the same.

Of course natural materials like wood take a bit of getting used to. Wood warps, bends, and comes in weird sizes (a 2x4 is actually 1.5x3.5). Sometimes it does what you want, other times it takes a bit of ingenuity to get it to fit properly. In that way, it resembles software when you are building on top of quirky libraries. Sort of standardish, but not enough to make the work easy and plenty of unexpected surprises.

So far my project has gone well. As usual I made up a plan for what I was going to do and ordered a huge amount of wood to get shipped to the house. Once I opened up the stairs I found a bunch of problems. The bottom stair for instance, essentially had its cross-beams completely rotted out. A number of other steps needed new, but rather awkwardly sized little support beams.

Pretty typical for this type of project. An initial plan is great, but you have to keep re-adjusting it as you find more issues along the way. The issues are never quite predictable and you learn to not get easily frustrated. Sometimes it’s smoother then you expect, but rarely.

The stairs were a fairly simple job, but because of a few technical setbacks -- my power cord died, although at first I though it was the mitre saw -- it took longer than expected. When I finally got to do the main part of the deck -- it’s essentially a separate unit from the stairs -- I found that my earlier reconstruction efforts hadn’t worked as well as I had hoped.

The main deck was sagging horribly in the middle, which turned out to be caused by my fixing the cross-beams that had rotted. I cut off the bad parts, then extended the remaining pieces. Those extensions weren’t the same height as the original cross-beams (which are 8 inches) and I foolishly didn’t brace them on anything. With time they started to shift downwards.

Not a huge problem, but it meant taking them all off again (there are twelve), then cutting some new pieces to prop them up underneath, and then finally putting each one back in place. It took around three hours, but at least this time it was braced properly.

During my enthusiasm to fix these initial deficiencies, I knew that the new height would no longer be sagging, and that it was now somewhat higher than the original height. Way nicer and a lot more solid, but it left me with a new problem. The very last board on the top of the deck, which is also the first board as one steps up from the stairs, sits on a pair of 6x2s. The inside 6x2 helps to hold up the cross-beams, while the outside one is mostly decorative (and had to be replaced due to carpenter ants, another minor setback). With the cross-beams now raised and fixed, these two boards were sitting nearly an inch below the main deck height. A gap large enough that I couldn’t leave it that way, but awkward enough that I would never be able to find a pre-cut board that would fill it properly.

I’ve got lots of tools, but nothing that could cut such a small piece for that long distance. I basically needed a 1x4x16. Sixteen feet is a hard length to keep straight with a hand-held saw.

As I was sitting there pondering my problem, my friend Tom walked by. He’s constantly building or fixing things and he’s dropped in a few times to take a look at how the work is going. It’s a hugely valuable contribution, since I’ve found that although I have experience with this type of work, there are plenty of situations were I hit a novel issue that I’ve never encountered before. To be able to stop for a bit, chat about what I’ve done and what I’m about to do really helps in both clarifying the issues and getting someone else’s perspective. And given that Tom has way more experience than I do, he often has really helpful suggestions or points out things that I have missed.

In this instance I was really happy to see Tom. His timing was perfect. He took a quick look at the problem and told me that he could cut the boards that I needed. He disappeared for a bit, and came back with a specialized circular saw. The cutting was a bit tricky because of the length (and because he was holding the boards in the air with only one end touching the ground), but he cut six strips of nearly straight 1x1.5. Layered side-by-side, across the length, they fit in perfectly and my problem was resolved. He popped off for somewhere else -- it was nearly dinner time -- and I continued to finish up the top of the deck, thankful that I had been saved from a rather tricky and no doubt, time consuming problem.

I can relate this back to software development because it is often when we are building complex systems that we hit upon a thorny problem. Over the years, I’ve seen programmer after programmer hide away in their cubicles attempting to solves these issues in isolation. Programming is about solving problems, but there are always times when the problem is beyond any one person’s ability or experience to fix it properly. Sometimes it is best to go out and seek advice, help or at least a sounding board. Just trying to explain a solution to someone else often shows that it is not particularly viable or just too fragile. Sometimes there is an overlooked trivial solution. However, some programmers go out of their way to avoid seeking advice. I think it’s a culture problem with programmers at the deepest levels. Perhaps the fear of sounding stupid because of a lack of solution. Or possibly just a need to horde the more interesting bits. It’s different for different people, but all too frequent.

From the outside however it’s usually appears as a case of someone taking on work that is too big for them to chew. They go beyond their limits and as a result their solution causes serious side-effects. A badly designed or poorly architected piece of software won’t ever fix itself and will always get worse over time. Stopping the problem early saves a huge amount of wasted effort. Admitting that the problem is beyond their limits may be difficult for a lot of programmers, but it is far better than the alternatives. Seeking help is a good thing.

Had I stubbornly refused to get help with my deck project, I might have labored for hours to find a solution. My initial instinct was to cut a huge number of 2x4 shims with my mitre saw. But as Tom pointed out that would have looked incredibly ugly. Although I have the skills do to most of the work necessary, this one sticky problem was just beyond my abilities. I neither had the equipment nor the experience to be able to cut the required piece (at the time I was seriously wishing that I had bought a huge bandsaw for the basement, it would have worked fine but I’m sure that my wife wouldn’t be too impressed with such a large machine sitting around idle most of the time).

For me, bouncing ideas off other people or letting someone else do a piece that they are better at is normal at work. When I am doing complex modeling or schemas for relational databases I usually spend some time discussing the design with Gavin, he’s got a lot more experience in this area then I do. Igor is a wiz at algorithmic stuff and in fiddling with complex technical bits. It’s always worth checking in with him first (he also is great at finding excellent parts for custom PCs; both of my home machines came from his recommendations). I tend to specialize in architectural or product related issues, which seems to be my latest focus (I used to prefer performance optimizations, but I guess I am getting older now...). I often consult with other programmers both internally and on the web whenever I need deeper advice. After twenty years, while I know lots of stuff, it’s just a drop in a rapidly increasing bucket. Getting things right the first time is far better than just guessing, and you’re just guessing if you don’t already have the knowledge.

A while ago I saw a blog post on how everyone should be doing code reviews. Formalizing that step is a good idea in places where the programmers are not communicating with each other, but certainly at my current work place it’s not necessary right now. As we are often bouncing ideas off each other and we do look at each other code (and point out problems), most of the code written by our rather small group was been reviewed at some point or another. That’s just a nice side-effect of our interactions. That, and we are able to gain indirect experience from each other’s efforts. If that culture doesn’t already exist naturally, then processes like code reviews are a great idea. Programming has gradually become a collaborative occupation where communication plays an important role in achieving success. The projects are too large and there are too many little details floating about for individuals to track properly.

Building software is always a complex endeavor and the external pressures such as time or management don’t make it any easier. You can learn by experience, but that often means doing the same work a few times before getting it right. That’s not a particularly efficient process. Software is complex enough that there are always a huge amount of things people don’t understand. Knowing where your strengths and experience really lie makes it easier to seek outside advice when the problems get beyond your abilities. Hiding in a cubicle, unwilling to talk to anyone, and struggling alone generally doesn’t turn out well (and is never the best solution).

Styles, Standards and Conventions

2011-07-14T13:14:00.000-04:00

Does it really matter how the code is formatted? Does it matter how you name the variables? Does it make a difference which variations of syntax you utilize for common tasks like loops?

For the most part any particular style, standard or convention is arbitrary. Sure, there are some that reduce or enhance the readability -- look at the obfuscated C contest for excellent reduction examples -- but overlooking the small trade-offs they all pretty much do the same thing. They set up some consistency in an otherwise unrestricted language environment. This doesn’t really matter to the computer, it has no sense of readability and it functions nearly identically no matter how the code is typed in.

So should programmers be free to create their code in any personalized or distinctive manner? Nope. We can not undervalue the utility in being able to easily read code, or in being able to spot obvious bugs because of how they violate consistency or naming conventions. A single file of code that is badly formatted, poorly named or bounces between different styles can easily hide bugs that would be obvious if the time and care were taken to add consistent formatting and naming. No one likes to waste time looking for problems that should be obvious, so that little bit of extra time and effort taken to make it clean easily pays off both in the short run, and over time.

Should a team of programmers all adopt the same style? Absolutely. Bugs are an indisputable fact of programing. Most systems are large enough, and complex enough, to require multiple people all working together. If one section of code is so eclectic that it prevents others from accessing it, it may work at the execution level, but it fails to be part of the system. And one is often blind to their own obvious coding flaws, so encouraging a second or third pair of eyes to help spot inconsistencies easily saves valuable time.

Job security is an often joked about excuse for excessive variations in styling, but it is far less stressful during vacations to know that any immediate problems can be solved by other people. It isn’t long before people notice that someone’s lack of presence means no progress, which generally builds up resentment. Pissing people off isn’t a very intelligent way of providing security (and never works in the long run).

An artistic desire for creative expression is another common motivation for group inconsistencies. That speaks to the idea that programming isn’t engineering and that the byproduct is a form of art. That may be true for some funky demo, but for serious projects one needs well thought-out industrial strength code. Failure to get this always means that the increasing acts of patching problems that were avoidable always grows large enough to consume any new efforts to extend the system. A mess begets a mess, and eventually eats up all of the time to do anything else. Thus ‘the art’ blocks out the ability to continue to build things. Creativity is nice, but getting something actually built is far better.

Styles, standards and conventions are a project/code-base specific issue. For each code-base (how ever large you want to define it) all of the programmers on the teams that are contributing to it, should follow the same rules as closely as possible. No excuses, no exceptions.

Prototype or System

2011-07-01T12:16:00.000-04:00

It is more than just intent. It’s really the application of knowledge in order to achieve expected results. That is, it is one thing to just play around with building something in the hopes that it might work someday. It is another to have gained the knowledge and experience necessary to build it with the certainty that it will do exactly what it is supposed to do.

Wikipedia’s engineering page references the ECPD which states that engineering is:

[T]he creative application of scientific principles to design or develop structures, machines, apparatus, or manufacturing processes, or works utilizing them singly or in combination; or to construct or operate the same with full cognizance of their design; or to forecast their behavior under specific operating conditions; all as respects an intended function, economics of operation and safety to life and property.

Effectively, if you are just “guessing” when you build something, you’re not applying any engineering. If you can accurately predicate how it will operate because you understand it, then it is engineering. It’s also worth noting that in the Wikipedia page, if you are applying knowledge and science, you can be considered an ‘engineer’ even if that work isn’t certified (I always thought that the designation meant both).

Sometimes when developing software we read a bit of the documentation, then we just start hacking together a bunch of code. If the code is relatively simple and what we are doing is straightforward there is a very good chance the code will work as expected. We don’t need any depth -- understanding of what happens below each line of code or in other parts of the system -- we can just work at a higher level and hope that there are no unintentional side-effects, or that the lower parts won’t change significantly over time.

That works for smaller systems, but it tends to accumulate technical debt without us being aware. It’s not what we know; it’s what we don’t know that is slowly and dangerously building up. Thus the main behavior of the system works, but a great deal of what could happen remains unknown. Generally I like to call this type of code a prototype. It’s the basic functionality, but in quickly hacking it together we skipped over the depth required to insure that it will always function correctly.

When I work like this, I consider it to be hacking. It can be fast and fun, but there are too many uncertainties to feel comfortable having people dependent on this type of construction.

For larger systems that have complex architectures, relying on ‘guessing’ is usually a fatal error. A small problem in a little prototype can become a massive headache in a big system. It can require months or years of cleanup to rectify it. Thus, hacking is no longer a reasonable way to work. Instead we need to get more depth in our understanding. We don’t need to understand it all the way down to the chips, but we do need to understand it down to any levels that were not well encapsulated. That is, if someone else solved the problem correctly, we don’t need to rethink it, but as often is the case in software, if they only partially solved the problem we need to understand the consequences.

To me, when I sit down to build something with enough understanding of the technology to know precisely what it can and will do, that is ‘programming’. There is no guessing, and there is no ambiguity. A programmer knows the steps needed to be taken-- both for the results but also for handling any errors -- and applies that knowledge to produce something that they are certain will work.

The key difference here is the knowledge and certainty. If you know how to write the code, you are programming. If you are just taking wild guesses then you are hacking. You may get lucky hacking, and for prototypes you may not have an alternative, but you are essentially leaving the end result to chance.

In that sense, it is very clear that programming can and should become a form of engineering. There are many programmers out there that have a huge depth of knowledge that allows them to build exactly what they want, in a rather predictable time frame. It is also clear that someone that has only lightly-learned a programming language, and can do a few simple things with it, is mostly hacking. Hacking is no doubt a lot more fun, but it is akin to learning some magic tricks without any real understand of how they work. They may be successful most of the time, but Murphy’s Law insures that when you need them most, they will fail.

Generally we can see this differentiation in action. In the more modern languages that purport to take care of the memory management -- because people no longer understand what is happening underneath -- the run-times have become massively bloated. Moore’s law saves us from this being a disaster, but it goes to show how many people are really hacking at their work. Another good example is the proliferation of threading bugs. A language like Java makes it easy to create threads and protects against many types of crashes, but does nothing to stop spaghetti logic from disrupting the application’s behavior. Again people hack at threads without a deep enough understand of how they work. To work with them in predictable ways, you have to understand how they work and what you can do with them.

To sum it up, there are always times when we have to resort to hacking to figure out the underlying behavior or limits, but there is no substitute for knowing how it really works. Hacking together a prototype is often necessary, but when it comes to building a real system, a huge amount of knowledge is mandatory.

The Customer is Always Right

2011-06-21T17:42:00.002-04:00

In many businesses the maxim “The Customer is Always Right” is often a reasonable way to retain a strong client base. It works best when the transactions are small and there is an expectation of repeat business. It also helps with ‘word of mouth’ situations, where many of the customers talk to one another. In the very worst case -- there are always people out there who feel compelled to take advantage of any situation -- you have to refund an item or two. Not a big loss in profit if it helps keeps business stable.

In software however, we are dealing with extremely long running projects. Most things have to be done in the correct order, and they have to be done well enough. Any earlier short-cuts manifest themselves in increasingly dangerous ways. Running a project by saying “Yes, yes, yes” to every and all requests is pretty much a death sentence. There are some things you just can’t do without negative consequences.

It’s always nice to please people and nobody wants to be the bad cop that says “No, it can not be done” or “If you do that, bad things are going to happen”, so it isn’t unusual to encounter projects in grave trouble because the people leading them cannot face up to their customers. They’ll give you plenty of excuses, but they are unaware of their own complicity.

If someone asks you to build them software then it is implicitly understood that a) the software shouldn’t suck, b) it shouldn’t take forever to build it and c) your expertise is required (or they could do it themselves). These are inherent universal requirements, they are true of each and every project no matter what is being built.

Few non-technical people are consistent or rigorous enough to be able to fully specify and understand software down to the depth necessary to write it. Thus change of direction, muddled thinking and indecision are frequent behaviors that are encountered with customers. But it is important that this turmoil at the customer level not filter down into the process of building the system. Software can’t be reactionary, it is a long slow process that involves grabbing each piece, one at a time from start to end, and getting it completed. Some design elements can be delayed in the process, and there is some flexibility in the long-term direction, but ultimately in order for the software to be released, it has to be written and tested first.

A project that is ping-ponging between arbitrary customer requests is one that is effectively putting too many balls into the air all at the same time. Not only is this increasing the inefficiency of the work by a significant margin, it is also increasing the risk that some, or many of these balls will get dropped or forgotten until it is too late. In the chaos of development getting these balls out of play as soon as possible by getting them completed is far more significant than how many of them there actually are. To be usable, the work needs to be done first.

So the most common failure in this regard is with the assumption that the customer is always right, and they should always get what they want. Reacting to all incoming requests is almost certainly going to cause a collision in the requirements. Enough poorly thought out “quickie” patches for instance will violate a) the software shouldn’t suck. Non-stop changes to the features or scope effectively violate b) it shouldn’t take forever. And the user demanding features, interfaces, etc. that are contrary to best practices or good working habits of professionals means that they don’t have any respect for c) your expertise.

The solution is that it is more than necessary to control the customers, and to control their expectations. This means having to have a deep understanding of their problems. This means some hand-holding, and often a lot of explanation about why some of their requests are not the right way to solve their problems. This means pointing out your expertise, many, many times if necessary. And sometimes this means having to say “No” to them, even if it is unpleasant and they are unhappy about it. On very rare occasions, this can even mean having to walk away -- there are some people that can never be pleased, so you’re going to lose no matter what you do. The alternative is to just do whatever they want, allowing the software to get into such a mess that they’re eventually going to blow up. Given that fate, it is always better to get this inevitable conflict out of the way as early as possible.

It’s nice to please people, but not at the expense of the project. Software developers are not professionals if the work they do does not live up to professional standards. Blaming poor workmanship on the customers or the environment is just a poor excuse for not mastering one of the most important skills in leading development projects. You can’t lead a project if you have no idea what needs to be done, or why it should be done one way or the other. You can’t lead a project if you defer control to other people. You can’t lead a project if you don’t know which direction to take. And you definitely can’t lead a project if you can’t control the customers long enough to get things out the door. Software projects are never reactionary.

Roles and Responsibilities

2011-06-11T12:17:00.002-04:00

One thing I’ve noticed about the software industry is that there is no unified base of definitions or best practices. Pretty much, everywhere you go, all of the programmers in different domains or markets have their own unique view of what the consider is standard, and proper. From a global perspective, this means that basically every programmer’s view of what is right, and what is wrong, is eclectic. With almost no commonality, this leads to a wide range of quality and interconnection issues within our industry. Basically we are dis-organized.

That is something that I suspect isn’t going to change for a very long time. Our industry isn’t mature enough yet to be able to standardize itself. Many people believe they know how to do it properly, but few even grasp the scope of the effort to align their ideas with others.

For this post I thought I would just list out the various roles that I’ve encountered while developing software. I’ve worked on in-house, consulting and commercial projects for 11 different companies, in five different domains. Over the course of that last twenty-five years -- including my co-op student days -- I’ve worked alongside or discussed development issues with a huge number of people. What and how to build systems has always been a keen interest of mine. From that experience, spread over a wide scope of differing opinions and viewpoints, I’ve built up a set of roles that I have found useful. Given the lack of consensus in our industry, I’m sure that these roles and definitions will not be in wide agreement, however I think it’s important to have some breakdown of the different skill-sets one needs to develop systems ranging from small to massive, even if they have different titles, or are partitioned differently. No matter what we call things, the same focus, decisions, analysis, work, etc. have to be done underneath.

So here is my list:

Business Analyst, Analyst, Domain Expert

Software is about solving problems, to get it working right requires a vast amount of minuet detail. So it’s important to have somebody in the project that deeply understands the workflow issues, domain issues and details involved. Most domains are complicated enough that keeping up with them or digging into the depths is a full time job, which needs as much focus and concentration as coding.

Data Modeler, Data Architect

For any system that is large, it will likely need to be connected with many other systems. To do this, there must be a ‘universal view’ of the data not just an application specific one. The most popular method of sharing large data-sets is via a common relational database, so the underlying schema for this needs careful thought and consideration or it will be unusable across the whole set of systems. Another growing trend is sharing via standards such as XML. Good modelling requires trading off correctness for convenience, which for one system is tough enough, but for multiple becomes a real problem that requires significant work in order to manage properly.

Computer Programmer, Programmer, Coder

Programming is the act of taking a description of some behavior on a computer -- in more or less detail -- and creating working code. There is inherent ambiguity in the description, which has to be solidified in order to make the system stable. Programmers focus on taking these descriptions and producing source code which can then be built and run. In smaller projects, with easier domains, the descriptions might be very vague thus allowing the programmers to dip into the analyses aspect, but since they often don’t have the depth of expertise, this leads to a huge number of changes driven by faulty initial assumptions.

Systems Programmer

Some types of programming requires strict discipline, deep understandings and very rigid logical thinking. The code is very algorithmic in its nature. This includes building operating systems, parsers, protocols, optimized calculation engines, and many other non-visual components. This also requires a significant amount of research in order to avoid spending time re-inventing the underlying knowledge from scratch.

GUI Programmer, Web Programmer

Some types of programming are almost entirely visual. The construction of the code -- its rigour -- is far less important than its presentation. By its very nature, this code is extremely repetitive, but generally does not involves very deep loops or concepts like recursion. Most really good GUI programmers don’t structure their code nicely, since they fiddle with it constantly in order to affect the way it looks when it runs. It’s all about appearance.

Applications Programmer

The bulk of code out there is basically Edit Loops. The system presents data, the user modifies it and then it is saved. This isn’t as glamorous as other types of programming, but it has its own challenges, most revolving around domain issues, quality and scheduling.

Database Programmer

Although it is changing, relational databases remain a fundamental technology in most development. Although they started from theory, over the decades they have become quirky collections of vendor specific behaviors. As such they require an unusually large amount of specific knowledge, that if ignored generally means that the utility of a database is mitigated by improper usage. Use them correctly and they help, do it poorly and it only makes the problems worse. Thus to get the most out of a database there is a need to get a programmer that has specialized in a specific vendor’s version.

Toolsmith

In a big project, the programmers are entirely focused on meeting deadlines. But they too rely on lots of complex software in order to complete their jobs. Most of this software needs special handling and significant configuration in order to help, rather than harm. Examples include build scripts, source code control, integrated development environments, document templates, etc. Any of the tools that are used frequently, and need to be sharpened in order to remain effective. Small projects generally distribute this work to the programmers (or don’t do it), while big ones need to have one or more people to fill this role.

Senior Programmer, Technical Lead

Any group of programmers will be naturally independent enough that left on their own, each programmer will choose their own unique process, design and conventions. Left unchecked, this quickly degenerates into a mess at the systems level. To avoid this, programmers are usually grouped into teams, and the teams are each lead by a senior programmer who has a lot of explicit experience with the specific type of development underway. If the lead doesn’t have experience, or can’t get the respect of the other programmers, then problems generally build up to a fatal level. The larger the project, the faster the build up, so sometimes in small efforts, while it is there, it is not noticed particularly by the participants.

Software Architect

As the number of people necessary to get the work done increases, the need for leadership and communication also increases. For a large project, with a number of teams, someone has to take all of the high-level decisions and focus them onto a coherent set of choices. Without that, the pieces themselves may work, but collectively they will be defective or unstable. The bigger the project, the more common choices that have to be made. Avoiding making these choices, just means duplicate effort that is unlikely to be synchronized, thus leading to massive (possibly unsolvable) problems when it is all brought together. Today's users have huge expectations for the quality and sophistication of their systems, which are most often beyond the ability of a single person, or even one small team to complete.

Software Developer

Building the software is still a significant problem, but it sits within a larger one. To meet its goals, software not only has to exist, but it has to be constructed within an environment, and sold often to another environment. There are a huge number of wetware issues implicitly involved in getting it out to the users, version after version. Someone who can take a project from a vague concept, all the way to multiple versions (including distribution, packaging and support issues) is a full software developer. That is, they understand not just the context of building the code, but also how it fits into the larger picture.

Project Manager

This often mis-understood role isn’t about issuing orders. A good project manager spends their days running around to all of the different people, making sure that they are not being blocked in their work. A project is running smoothly, when all of its resources are running smoothly. A project manager doesn’t need to have been a programmer, but they do need a deep understanding of the technical issues involved at all levels of the project, otherwise they won’t be able to properly asses the inter-dependencies. A big project is often a spaghetti network of smaller work items, that need to be done in the right order, to minimize the effort. Pre-planning, foresight, trade-offs, and technical understanding are important to make sure things progress smoothly.

Product Manager

Although a system is built to solve a set of concrete problems, there are often a large number of related problems floating around that could be solved as well. It seems like an easy issue, but particularly for commercial software, each choice comes with a series of very dire trade-offs. If one feature it built, it means that others are not. All projects required funding (budgets or revenue), and to keep this going they have to satisfy a large number of irrational conditions. Someone needs to have the fore-sight to seer the effort away from the dangers, and towards a constant series of wins that maintain the current momentum. Without direction or funding, the project dies.

Chief Information Officer (CIO)

In companies that don’t sell their code, but buy or build their own, the highest technical operations position is the CIO. Their job is to insure that the data needed for the business, is the data collected and organized by the IT department. It’s a difficult job given that the industry is never forth-right about the limitations of their technologies, and it is constantly moving. Sometimes this position is filled by a non-technical business person, in which case it is really a monitoring/reporting role for the people doing the actual work underneath.

Chief Technology Officer CTO

In a company that produces commercial software, or software services, someone must direct the overall vision of the development effort. Like the CIO, this role is necessary to perform long-term strategy. Often times, this position goes to a non-technical person as monitoring/reporting role, but for smaller companies it generally is held by the lead technical developer, thus it can be a leadership, working and visionary role. For someone interested in building massive systems, this is the best role in the company to have (although it can be very hectic and stressful).

Operator

This role has largely disappeared, but it was a very important one. Often the computer can’t make a valid intelligent choice, so the decision is passed back to a human. They choose to accept something, or escalate the problem. It is unfortunate that this role is disappearing, because it provides a way of injecting some dynamic intelligence into the system, reducing its instabilities.

Systems Administrator

In the operations department someone must look after each of the machines. Modern computers contain a vast array of low quality software that interacts very poorly. Modern practices have also focused on releasing software at a faster rate then it can be correctly written, so there is a never ending stream of new versions, some of which need to be ignored. As well, they are a plethora of security risks, and many ways in which people can accidentally damage their own machines.

Network Administrator

Interconnecting a large number of computers together requires an organized topology. In any large organization, there are always old machines dropping out, and new ones coming it. As well, because of lax security considerations on the part of almost all software vendors, there is an ever growing, massive collection of ways in which malicious programs can spread and cause very real, and expensive damage. Keeping up with this shifting landscape is a huge job.

Relational Database Manager

For massive databases, relational technologies need constant tweeks in order to prevent the performance from degrading. This means a lot of monitoring, collection of statistics, analysis and then implementing improvements that maintain the health of the database. Gradually, the technologies have gotten better at doing this for themselves, but for really large data-sets they are still insufficient, and the work requires careful though and deep thinking.

Operations Manager

Someone has to run the operations department. Sometimes they are ex-programmers, and sometimes they just moved up in the operations department. Operations is a reactive environment that is always shifting. Also, many people working in the lower roles are not in love with their jobs, so they don’t tend to be as prudent about their efforts. Someone has to keep an eye on the whole affair, and make sure it is running smoothly.

Pre-sales Support, Pre-sales Engineer

In hardware or software sales, it is unusual to find salespeople with more than a very shallow depth of the underlying products. However, they are extroverts and can sell it quite well. Technical people on the other hand tend towards being introverts and aren’t particularly successful as salespeople. In order to bridge this gap, most companies team up a gifted sales person with a technical one. This works well for both sides, since each can focus on what they do best. Pre-sales support is a great role for someone who loves technology but doesn’t want the rigors of hyper-focusing, day-after-day on a building effort. It’s a great paying career, and often a more comfortable choice then moving into operations.

Post-sales Support, Second-line Support

Software needs to be set up initially, and once the system is in place, someone has to deal with its inevitable problems. This is a problem solving role, that can involve deep debugging skills. While this position is often associated with commercial products, it is often an in-house equivalent.

Helpdesk Support, Front-line Support

Front-line support deals directly with the users for the commonly repeating problems. Most of them are user-based issues, but often they are just known software bugs that can go on for years and years. If a problem isn’t repeating (first or second instance) it is passed to second-line support. If the second-line support is organized, by the time the problem has shown up for the third time, there are instructions on how to correctly deal with it.

Data Administrator

Modern dynamic systems often involve a lot of customization that can be handled by the user interface. This creates a new role whereby one or more of the users or domain experts is nominated to track and modify these configurations as needed. This is not to be confused with database administrators who are working at a much lower technical level. This is essentially a privileged user role.

User

The user is any person that is using the system. Some users are actively adding or updating data, while others are management-based users that are monitoring activity or mining the data. It is worth separating the two groups, and also distinguishing between frequent users, and causal users, since the interface needs of the different sub-roles are completely different. The best systems, need very little or no training for the users to get value from them, but that differs depending on how they use the system.

Titles that I don’t like:

Stakeholder

This is a relatively modern term for all of the people who have some stake in the project. Sometimes this group includes the programmers, but often it doesn’t (although they do have a stake in the project if they are intending to work on it for the next five years of their lives). Aside from the image I always get of a waiter holding up a tray of T-bones, I think the term includes too many people to be useful. Many of the stakeholders want unnecessary control over the efforts, based on their providing funding. But once successful, they won’t be involved in being a user when it goes live. Software projects are rife with complex people issues that often center around who is controlling the process, and it doesn’t help to seed control to people whose depth is too shallow to make reasonable decisions. In order to be successful, the right people with the right knowledge have to make the right choices. Missing that is fatal.

Software Engineer

Our industry is so distributed and so chaotic that it is unlikely that we could even legitimately call ourselves a craft, let alone engineering. The first of those involves passing the knowledge on from master to apprentice, something that happens rarely right now, while the second involves writing it all down in an organized and usable manner so that it can be applied deterministically. We’d have to get through one, before we can arrive at the other. Currently we can’t even get a consensus on the basic roles involved.

Hacker

Hacking is what you do to get the prototype ready for a demo, or how you apply a temporary patch late at night. Once you get funding, or the next morning, you should take the work a little more seriously. Starting with a mess and hacking at it, only generates a bigger mess, one that will eventually exceed the abilities of all of the available resources, leading to a nasty downfall. Sometimes hacking is called for, but it should be used with the appropriate caution.

Following the Path

2011-06-01T09:00:00.017-04:00

There is some advice that is very well understood in a few commercial software development circles, but I don’t think it has made it out to the general industry.

Software capabilities are by their very nature interdependent. They come in collections of related functionality and features. One convenient way to view these groups is as different paths that can be followed by the development effort.

With that in mind, the sage advice is to “never step foot on a path, unless you’re prepared to follow it”. It’s something I’ve heard many times, and repeated often.

It’s actually a simple idea. Once you start some development along a new path, the users will push hard to get further along. Nothing is worse that a ‘lite’ feature that sort of works. It’s existence will drive the users to demand that it get fully implemented, properly.

A great is example of this is if you are building an editor. Someone might come along and suggest an easy, simple localized ‘source code control’ like mechanism. Maybe you just keep a backup of the last set of writes. Simple right?

But it isn’t long before that doesn’t satisfy anymore. They’ll need more versions, a way to revert and some access to the underlying historic information. Eventually, they’ll want a full version of source code control with all of the bells and whistles. That would be nice, but getting there organically probably means the design has been compromised by history. Tentatively stepping on that path is only going to cause future problems.

As well, the existing development is probably midway through another set of paths. Is it better to thin out the resources to work on more half-done features, or focus on a smaller number and get them closer to being well-rounded and complete? People get drawn in by features, but they stay because the tools work for them. A half-done job is only great for marketing.

Of course, there are times with new products where one has to stake out their direction. It would be nice to only focus on one path at a time, but products live and grow so it helps to gives the users some type of preview of the future. Working on several paths simultaneously is both a requirement, and a sound long-term strategy.

Still, it is not a choice to be made lightly. Just because someone suggests a nice sounding feature, doesn’t mean it is a good idea right now. An unfocused system that partially does everything is essentially useless. A focused one that does a few things right is considerably more practical.

The Boundary Between

2011-05-27T09:00:00.001-04:00

A continuous system is one where no matter how small a section you exam, there is an infinite amount of further depth. A discrete one is where there are indivisible atomic pieces, which can be broken down no farther.

Initially, as a by-product of how we naturally view the world, our perceptions were likely that the things around us were continuous. The Ancient Greeks were one of the earliest to propose that everything was composed of an underlying discrete set of atoms. That view seemed to prevail, more or less until the Industrial Revolution launched us on a deeper perspective implicitly showing us that the world was continuous once again. But the rise of our computer technology has been gradually reversing our perceptions back towards the discrete. We’ve been flip-flopping between these two alternate perspectives as our knowledge of the world around us changes, and grows.

The change in perspective is not what I find most interesting. It’s the behavior of the boundaries between these conflicting views when they are layered that is most fascinating. That is, moving from a continuous system to an underlying discrete one or vice versa.

A great number of interesting details about the human body can be learned from Bill Bryson’s “A Short History of Nearly Everything”, a book I highly recommend because it is both entertaining and also enlightening. By now we accept that humans are a large collection of millions (billions?) of co-operating cells. Bryson also points out that we are also composed of a nearly equal number of bacteria, and other symbiotic living organisms (which we need to function). He also addresses that from a matter perspective, we’re basically replacing ourselves continuously. New matter comes in, old matter goes out. New cells grow, old ones die. I think he speculates that we are completely re-composed of new matter approx. every seven years. If you’re interested, you can get the full details from his book.

With that in mind, from a perspective purely based on particles, we can view ourselves as a big collection -- a cloud -- of related matter. For a given human being -- we’ll pick a male and call him Bob -- as he interacts with the world on a daily basis, he is leaving behind some of this cloud. That is, Bob’s discarded skin cells add to the dust in the room, his bacteria and fingerprints get left on everything he touches, and his moisture evaporates into the area around him. If Bryson’s speculation about seven years is correct, then over that period half of the particles that have made up Bob have been dissipated to wherever Bob has been. From a cloud of particles perspective, ‘Bob’ is the densest part, but the whole cloud covers a much wider area based on his travels.

If we could build a machine to detect ‘Bob’ particles over a period of time, then some really interesting things occur. For instance, on any average week, Bob would go back and forth between his house and work. The cloud would include both locations, but also to a much sparser degree, the pathways in between. Setting the machine for a granularity of 7 days, would show that Bob was both at home and he was at work. Depending the mechanics of the detector, the bulk of Bob might appear in either location. From this viewpoint, we might conclude that Bob was in both states simultaneously, and that when actuated he would appear randomly in one or the other. We could compile a set of Bob particle-cloud states, that he might be simultaneously inhibiting: work, home, traveling, cottage, etc.

One can easily suspect that Bob would easily disagree with this world view. He is, after all, either at work or he is at home. He knows where he is. Although he is a discrete entity, he likely sees himself as continuous. The detector however takes a discrete measurement over a continuous time period, which at 7 days is not in line with Bob’s continuous view of his discrete underpinning. Bob does not see himself as a cloud, and certainly not as a cloud over a long period of time.

If we set the detector’s time frame to 1ms, then the resulting Bob cloud would be roughly what we see and think of as ‘Bob’. Shortening it some more would produce gradually more refined Bob versions, until we get down to some discrete time-block where our perception of Bob matches the detectors. In this case, the detector’s information would align with Bob’s understanding of himself and his world. There would be no randomness with regard to Bob’s state.

What this comes down to is that as we are flipping back and forth between continuous and discrete perspectives, if our discrete boundaries are not perfectly aligned (even though they are not adjacent), then an element of randomness appears to creep into the mechanics. Since our continuous/discrete perceptions of the world around us are not explicit (we use both of them without being aware of it), we are not always aware of the affects this bias in the way we model the world. Thus one might conclude that the randomness isn’t actually there, but that it is just an artifact of the boundary.