Software Industrialization

Software Abstractions

Mitch Barnett — Fri, 09 Oct 2009 06:57:34 GMT

When I use the term software engineering, the definition I am referring to is:

“Software Engineering is the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software, and the study of these approaches; that is, the application of engineering to software.”

So what then is a Software Abstraction? Take this paragraph from the awesome book, “Software Abstractions”, by Daniel Jackson:

“Software is built on abstractions. Pick the right ones, and programming will flow naturally from design, modules will have small and simple interfaces, and new functionality will more than likely fit in without extensive reorganization. Pick the wrong ones, and programming will be a series of nasty surprises: interfaces will become baroque and clumsy as they are forced to accommodate unanticipated interactions, and even the simplest of changes will be hard to make. No amount of refactoring, bar starting again from scratch, can rescue a system built on flawed concepts”.

Brilliant! It’s the first paragraph of the 1^st chapter into a world of real software design - i.e. formal specification, including the ability to model and automatically analyze your design:

“Unlike theorem proving, this analysis is not “complete”: it examines only a finite space of cases. But because of recent advances in constraint – solving technology, the space of cases examined is usually huge – billions of cases or more – and therefore offers a degree of coverage unattainable in testing.”

“Moreover, unlike testing, this analysis requires no test cases. The user instead provides a property to be checked, which can usually be expressed succinctly as a single test case. A kind of exploration therefore becomes possible that combines the incrementally and immediacy of extreme programming with the depth and clarity of formal specification.”

Check out Alloy Analyzer and the Community.

It’s slow going for me, but I am getting through the book. Until you read this and run Alloy Analyzer, you cannot imagine the number of “cases” Alloy Analyzer can run, like the text says, a billion or more. It picks apart your design like you have never seen before. Imagine what the number of cases would be when you combine dozens of models to formulate a software application.

This is a leap forward in software engineering design in my opinion. Being able to analyze your design against a billion cases without writing a single line of production or test code is unheard of in mainstream software development industry today. Using this design approach will produce software design's that will be more robust, perhaps by orders of magnitude, than any manual method we apply today. The software design itself, is a complete formal specification used by the “end” software programming language to transform the complete design into a runtime solution.

What about the software engineering definition comment in the opening? I am pointing out that we are following the definition of software engineering by using Alloy for designing software; it is one way to apply a systematic, disciplined and quantifiable approach to software design. In my opinion, we as an industry, and as professionals, should be applying more effort towards software design. Why? Read the “Software is built on abstractions…” paragraph again.

Daniel Jackson and his team’s work on Alloy Analyzer, to me, is the state of the art in software design today. When will it become mainstream? Good question, but you can help. As a Software Designer, have a look at Alloy. Maybe apply it to a hobby project. With enough practice, perhaps introduce formal design specification to the workplace on one small specific issue or algorithm you are working on. That's my approach.

I know change is hard and everyone resists, but look at it as an experiment to try out, see how it works, and if it makes sense to you. As a software engineering practitioner, I am trying to become a better software designer by learning new concepts and techniques. I believe Daniel Jackson’s Alloy makes me a better software designer.

Software Reliability in Web Application/Site Design

Mitch Barnett — Tue, 01 Sep 2009 06:06:22 GMT

"Software reliability engineering relies heavily on a disciplined software engineering process to anticipate and design against unintended consequences."

Wikipedia Software Reliability

The six paragraphs in this explanation on Software Reliability are bang on in my experience. In fact, the statement above best describes the difference between project success and failure. The more disciplined the software engineering process the better quality the output, in all measurable aspects, is my conclusion after 18 years as a software professional. Btw, disciplined != bureaucratic || inefficient.

I wonder if software reliability engineering had been applied to the following web sites, it would have been less likely for me to see these messages:

The irony here is that these two messagess occurred on two seperate dates, not just 20 minutes apart :-)

An Error Has Occured? Pics or it didn't happen! Oh... well how about directing me to another download site that is known to be working? I can't imagine what's in the "On-line Self Help."

No way man! ;-)

Aeroplan, a fairly large transactional site, was down for two days. I wonder if software reliability was on anyone’s minds when the site was designed…

You may have noticed that some of the screen shots show maintenance messages, not error messages (and other reliability messages). I find it hard to believe that there is (still) the practice of scheduling "planned outages" for web application/site maintenance on the main public domain web site. First business rule in the world wild web, you are always open for business.

Sure, some of these sites have staggering amounts of traffic and processing. However, it is 2009, where computer hardware has never been cheaper and the most powerful at the same time. Content Management Systems and Ecommerce systems, have never been more plentiful, more mature, and the most featured for the least amount of dollars.

That’s one of the points behind having a disciplined "software engineering process." One of the most important activities, in my experience, is designing for software reliability. No, I don’t mean life and death reliability, but given the budget constraints, what is the best or most acceptable "business risk" the web application/site is able to mitigate from a reliability point of view.

What do I mean by business risk in this context? Given that Aeroplan was not able to transact business for two days, let alone how it looked to "customers", I would be surprised if the executives signed off on that their main public web site could be down for a two day period. And no, I don’t mean "high availability" or five nines uptime, albeit related, I mean software reliability. Even if you paid for five nines up time, it is very likely your software will error out (or you planned a maintenance window!) well before you hit that uptime number. I would say that this is predetermined by how your software was designed and engineered.

I wonder when the "tipping point" will occur where the general public will stop accepting software that is the analog equivalent of missing a wheel on a car. Sure you can drive it, but not very well. That’s the ultimate problem with software, it is completely invisible to the majority of people, but if they saw it, and it looked like a car, then they would see that missing wheel, hear the engine misfiring, and see/smell the smoke bellowing out where canola oil was used instead of engine oil. The car is clearly a lemon and would not sell. Even if it did sell, there is a lemon law.

Yet software contunues to be built, and software products exist today, that are the equivalent of the lemon of a car, but there is no lemon law or even a guarantee of any sort for software. There is a reason for that. Some would say that software is too complex or at least no-one, other than NASA can afford the level of software reliability engineering that we comparatively see in our cars today. Do you think that is true?

Based on my experience, I don’t think so. In fact, over the mid term and long run of a software lifecycle, spending the engineering time to design the software correctly (or as best as possible given budget constraints) easily costs (considerably) less than just trying to bang the software out using the latest du jour software process methodology and then fixing the heck out of it for years to come. At least my own anecdotal experience supports that position.

These software quality issues were screen snapped over a couple week time frame while I was browsing the net. What does it mean? I feel it means a couple of things. One is that I am one person, the smallest possible sample size, yet look at the poor quality software I encountered during a short time frame. I can't imagine what the impact is on millions of users sampling (hundreds of ?) thousands of web applications/sites daily. The other is there seems to be far and few between companies that actually practice designing and engineering software in a disciplined manner...

Updated Next Day - Ha!

Gmail outage deprives millions of e-mail

Who's Your Daddy?

Mitch Barnett — Sat, 15 Aug 2009 07:32:09 GMT

+1 WolframAlpha

YouTwitter – Next Gen Twitter

Mitch Barnett — Tue, 11 Aug 2009 06:50:20 GMT

I must be getting old. I find twitter enables a curious social behavior, seemingly aimed at extroverted people, and some would say at its extreme, narcissistic.

While I don’t use twitter myself, or even Facebook, I do blog, ergo, I must be one of them too. Myself, I am simply interested in the technology as the human factor seems to be hurtling towards Singularity at a rate I no longer can understand.

I spend a lot of time in front of computers, I mean at least 8 hours a day for the last 18 years or so, mostly as a professional Software Engineer. As a Software Designer, I find twitter interesting, but why don’t we design the equivalent of twitter on video (with sound) instead of text? YouTwitter would allow you to record a video for 5 - 7 seconds (or whatever). We certainly have the computing horsepower, the bandwidth, and what seems to be a critical mass of cell phones with video/audio record functions and wifi capabilities. Not to mention any computer/laptop/netbook/media center/(Xbox?) with a microphone, web cam, and internet connection. Which essentially means the same size market as “regular” twitter is today, but rather than serving up text, now serving 5 second video (with audio).

As a Software Engineer, it is interesting to think of the design and implementation of such a system. Who knows, maybe it is already happening, and this shows how out of touch I am with social networking ;-)

If YouTwitter existed, I probably would use it – not only would it be fun, but change the way we communicate via computers. Take me for example, at 50 years old, and really been on one form of computer or another since 1980, I am (really) tired of writing text. It would be way cool not to have to type any more text. I write too much already and that’s why I would like to see YouTwitter. If I had the time and capital investment, I would design and build it myself.

YouTwitter is more than that. What about the capability to see people’s interaction without the text? Did I mention how our overall communication style would change if I could see you speak instead of just reading your text? Is this the next evolution in social networking?

I don’t know, but I would use it, even as an introvert.

The (Lost) Art of Software Engineering

Mitch Barnett — Tue, 04 Aug 2009 04:12:02 GMT

Quick, tell me what is common between, “Tragedy at Second Narrows – The Story of the Ironworkers Memorial Bridge ” and Software Engineering? Answer: They are both stories about engineering and one has tragedy in human lives.

The Tragedy at Second Narrows is an absolutely compelling story about engineering design. The tragedy is the number of lives lost due to what appears to be an engineering design oversight. Ironworkers Memorial is of very special meaning to me as I live near Vancouver and occasionally have the privilege to drive over the Ironworkers Memorial Bridge . I have a lot of respect for the Ironworkers, Engineers and the entire craftsman who worked on this piece of art. The art is that traffic flows across the bridge at the same speed as the #1 Highway in Canada – the user experience is seamless and that’s its beauty, it is also by design (i.e. it has been engineered that way).

The Ironworkers Memorial Bridge also has a special meaning to me as it relates to software engineering. I believe software design and development is an engineering activity. In fact, the more years I do this (since 1977 in electronics engineering and slowly moving to software engineering since 1991) the more I believe it. So do others, including educational institutions. You can get a 4 year degree in Software Engineering and the States have been doing it since 1996 when I saw it at RIT while working at Eastman Kodak in Rochester, NY. Damn man, you can even get a PhD in Software Engineering.

Software Engineering in the province of BC is recognized as a professional practice and since 1999, you can apply to APEGBC to become a licensed Professional Software Engineer. Aside from having an ABET recognized 4 year engineering degree, 4 years documented experience, 4 P. Eng.’s to stand up and vouch for your sorry ass, attend a 2 day law and ethics seminar or consume the equivalent in CD-ROM, and pass a 2 hour Professional Practice examine, there are additional requirements. You need to fill out a Software Engineering Checklist and you need to complete the Software Engineering Experience Guidelines.

It is very interesting to read over the last two documents, Software Engineering Checklist and Software Engineering Experience Guidelines. The point being, if you want or need a description of what Software Engineering is and how to document it, spend a bit of time reading these short documents. While a bit old, the first principles hold true, which is what I refer to the lost art in my title (more later).

As mentioned previously, Software Engineering is a recognized professional practice in British Columbia . Check out the benefits of becoming a licensed Professional Software Engineer. Meet the first licensed Software Engineer, Philippe Krutchen, who was licensed in 1999 in BC.

Software Engineering to many people have many different meanings. Some think/feel software development or programming is an artistic endeavor. Others think of it as craftsmanship. From my perspective, I agree with everyone. In fact, it is all those things and much more. Some people feel that the La Sagrada Famillia is art. Some would say the craftsmanship that went into it was second to none. But it was also engineered (i.e. designed). Check this out, Gaudi designed the weight-bearing of his buildings, upside-down with string and bearings. So when you looked in the mirror, of course it was right side up. I saw this first hand. Engineering at its finest, well before modern engineering with all our fancy calculators and our StarTrek Communicators.

So Software Engineering is design, art, craftsmanship and any other word that people want to use that is not engineering. Call it what you will, I agree with you. But it is also part engineering. That to me is the lost art, specifically the engineering design part.

In fact, I would say Software Engineering, at its very core, has four fundamental activities: requirements analysis, design, code, and test. Of course you have to deploy and operate, maintain, and a whole pile of other things (reports, UI enhancements, skinning, ad infinitum), but I just want examine these four core activities. Let’s put some definition around what I am getting at:

Requirements Analysis: I don’t care if you use story points, scenarios, use cases, flowcharts, or animated puppets to gather and package software requirements. The question is how accurate is your transformation and how good you are at repeating it. Understand what I mean? How do we as a team communicate “what” needs to be built? To answer the question, I am suggesting use whatever methods and tools you want as long as the result (i.e. the transformation) is accurate and you can repeat that accuracy on multiple projects consistently and eventually, predictable. That to me is the lost art of software engineering in requirements analysis. What is the percentage accuracy of our requirements gathering and packaging? i.e. relative to the intent of what the users actually require. And btw, if the users or community can’t tell you, in gory detail, what they want, then no amount of software is going to solve the problem. Capiche? What’s our percentage repeatability and predictability? That’s the theory and in practice it does works. But it is a lot of very hard work, so you really don’t see it too often in our field, hence the lost art.

Design: Ah, the crux of software engineering. Your choice of design methods and tools should be determined on which methods and tools will result in the least complex, but most (easily) extensible design possible. In short, at its essence is a mathematical formula for reducing complexity, but the art is your application of the formula at the problem at hand. Again how accurate is your design transformation relative to the requirements you used to produce that design? Can you do it consistently as new and old problems come your way? Whether you choose a formal design method like MIT’s Alloy, (if you have never used Alloy before, try it on for size, it really demonstrates software complexity through modeling. ) or a traditional OOD method, is really not the ultimate issue here – choose whatever method/tool yields the most accurate Requirements -> Design transformation.

Code – it seems everyone knows how to code in their favorite programming language. Again ultimately, from a software engineering perspective, use whatever programming language yields the least complex representation or transformation of the design. Oh, yeah, I know what about the human factor – where are we going to find <insert favorite language here> programmers? Or the million other factors that occur when selecting a programming language. Does it really matter? Not if you want to produce the least complex implementation.

Test – There seems to be a dazillion and one testing methodologies/tools. Does it really matter which one you use or what tool? I don’t think so. Again, what percentage accuracy is our testing against the 3 previous transformations? Oh sure, at what level of effort you ask? What was the requirement for testing accuracy and coverage? Don’t have one, better get one. More lost art.

Oh, I know I am greatly simplifying the above, but isn't that a lost art also? It seems to me that year after year we further complicate our art, not simplify it. One lost art of software engineering is trying for the most accurate transformations from requirements to test (and eventually deployment, operations, and maintenance.).

Another lost art it seems is designing for a least complexity solution or at least making it a priority. But that takes time, commitment and data. And that’s my point. It appears the trend I see in our industry, and if Jeff Atwood’s blog on Software Engineering: Dead is any indicator, reading the comments are an eye opening experience about people’s thoughts on Software Engineering. I have been there myself, on more than one occasion, and in fact, is normal for most companies that are trying to design software, but really don’t know how.

The only way to improve what we do as Software Engineers is to measure what we do and how we do it. Oh sure, that tired old saw. I have seen it work with my own eyes to know the truth and value behind it. Yet, we as an industry seem to have completely forgotten the value of measuring what we do and how we do it. Other industrialized industries would cease to function without this feedback control loop. Can you imagine pro sports without stats? It would never happen, but the trend in our industry seems to be moving away from measuring anything and just going for it. I wonder why that is?

My hope is that over time Software Engineering, while a Professional Practice in British Columbia , becomes more widespread, and what I used to take for granted as standard operating procedure (SOP) in Electronics Engineering, will become SOP in Software Engineering. But like anything, this is going to take time and education, particularly for folks that don’t know about the engineering part, or for others like me that have lost it from time to time.

Unprecedented – A Two Year Software Guarantee!

Mitch Barnett — Tue, 19 May 2009 05:11:53 GMT

As an 18 software development professional, I have been waiting for this day for a long time. Honestly, I thought this day would never happen in my lifetime, but it did and in a form I would have never guessed:

“The future of games development has been called into question after the EU Commission suggested developers provide a two year guarantee.”

Two year guarantee on… games? I would have thought that medical software or aviation/guidance software or any other software application/system that could put people in harms way would have been the first industries to have instituted guarantees on defective software.

Never did it cross my mind that it would be the gaming industry, but then again, after thinking about it, it makes perfect sense. Video games are a multi-billion dollar business targeted at mass consumers, a.k.a. the public. So when lots of people’s games don’t work, something has to give.

The key clause is, "two year guarantee on all games in the event that a bug or glitch is encountered preventing you from completing the game and/or ruining the experience."

As the articles states:

“At present, retailers are not obliged to give a refund on a video game that has a bug or glitch that prevents a user completing a game. If the proposals become law, this could change as users would have the right "to get a product that works with fair commercial conditions".

Note it says proposal, but I say great, it is about time! Why is it only in the software industry, people and corporations spend trillions of dollars on software and none of it is guaranteed? No, I don’t mean delivery guarantees, or warranty periods, I mean outright quality guarantees.

For me, as someone who develops software, I could not be happier. This marks the day that there is enough critical mass in the public to demand better quality software products. Really it means that software will need more engineering, as in software engineering.

“Software engineering is the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software, and the study of these approaches; that is, the application of engineering to software.”

At age 50, I have had two careers, the first in electronics engineering, where the application of systematic, disciplined, quantifiable approach… was never even questioned, talked about, it was simply expected, standard operating procedure, institutionalized, etc. But when I entered my software career in 1991, I was very surprised to find that in the software world, there were debates going on, still today, whether software development is engineering or art or science or…

Fortunately today, many universities are offering undergraduate degrees in software engineering and have defined the difference between computer science and software engineering.

This means that younger generations will be more exposed to software engineering rather than us “old guys” where there was no such thing called software engineering when we went to university.

I also believe that software engineers should be licensed as Professional Engineers (P.Eng. in Canada ). In fact, in British Columbia , Alberta and Ontario , you can be licensed as a professional software engineer.

You can read an excellent paper on, "Licensing Software Engineers" by Philippe Kruchten From the article, “The only purpose of licensing software engineers is to protect the public.” I totally agree. Now Philippe suggests that licensing software engineers who designs and develops software that can kill is protecting the public. I agree with that as well. But I would take bit of a broader definition that not only includes software that kills, but again protects the public from financial impact, which basically covers most business software, like ecommerce, banking, trading insurance and any applications that deal with money, but also even t the point of gaming software, which is what this article began with and the proposed two year warranty.

Perhaps a little reflection is required to let what I am saying fully sink in. For any software development project that has any impact on the public, financial or otherwise, the software engineer responsible for “sealing” the blueprints should be a licensed software engineer. A licensed professional software engineer is bound by professional practice and code of ethics. That means that a licensed professional software engineer is legally responsible for the design of the software. Think about that. Aside from the responsibility, the way software is designed and engineered is going to change in our world. I can only be grateful for that and like I said early on, it is about time. Today is a red letter day!

Real Software Engineering

Mitch Barnett — Tue, 12 May 2009 05:29:48 GMT

Over a half dozen years or so, I have noticed a trend in our software engineering world towards collecting less and less data about what we do and how well we are doing it. This would be contrary to lets say professional sports where data collection (i.e. stats) is a well know practice and pastime, plus on the rise, whereas in our professional software engineering world, it appears to be on the decline. I wonder why?

A history lesson first. A fantastic short paper called, “Lessons learned from 25 years of process improvement: The Rise and Fall of the NASA Software Engineering Laboratory” provides some real insight to a problem that now seems to be rippling through our industry en mass. I would say the Software Engineering Laboratory (SEL) was ahead of its time and has seen some possible future outcomes that may be coming true for some (most?) organizations in the corporate world.

The SEL collected software engineering metrics on process improvement from 1976 to 1991 and beyond. I am not going to summarize the lessons here, you can read the excellent paper for that, and if you were/are ever a part of a real software engineering organization, none of the lessons learned should come as a surprise. But I think that is the real lesson to be learned from this paper and the amazing contributions to software engineering that came out of the SEL and that is, very few organization are actually practicing real software engineering process improvement. And in fact, I would suggest that the latest trends in software development lifecycles, like Agile, XP, Scrum, et al are actually going the other way.

Allow me to explain where I am coming from; here is one of the lessons learned from the SEL: “Establishing a baseline of an organization’s products, processes, and goals is critical to any improvement program.” In most other industries, this is a given, including professional sports. Heck even in non professional sports, like your college track and field events, you establish your baseline for the 100 yard/meter sprint and constantly trying to improve on that as a simple example. Yet, in the software engineering world, where is our baseline? We have embarrassing baselines like spending millions and billions (note - how embarrasing, in my home country!) of software that is of poor quality or doesn’t even work. Maybe that is why we don’t measure ourselves?

We have “new” software development lifecycle methodologies that promote no collecting metrics on how well we are developing software. Its astounding and disturbing, not only to me but to businesses that are forking over millions of dollars for software development projects that seem to have nothing more than an ad hoc process of going wherever the project meanders, no matter how long or at what cost, let alone even validating if the right problem is even being solved in the first place.

How do I know this? I have been in the software engineering business for 18 years and had the privilege to have experienced many a software shop ranging from multinationals like Eastman Kodak, Motorola to midsize companies and to small start ups, including my own, in several different industry verticals and locations in North America . I had the good fortune of being taught Software Engineering Management, a 2 year post graduate program from the University of Calgary, taught by Motorola University, where each of the instructors had many a year experience and stories about software engineering projects, both failures and successes and most importantly, what the differences were/are. I have kept in touch with past employees/friends in the industry to hear interesting anecdotes over a virtual beer. And finally, I spend a lot of time researching and reading about software engineering in general.

And my conclusion is that we as an industry are (way) off track and seem to be heading further off track with the invention and adoption of new methodologies that are not based on (any) actual factual experience/metrics/historical data like the 25 years of solid software engineering experience that the SEL has produced.

Why do I believe that? I see it everyday and in several companies that I may come in contact with due to my job. I don’t need any further proof than that. Just even the simple metric of “establishing a baseline” of anything in your organization should be proof enough for you. What data have you collected for x software engineering projects you have undertaken, in any of the companies you have worked for? I bet it is a small percentage, if any at all.

Why is that? I think the SEL folks hit it on the head with one of their lessons learned, “It is difficult to make an engineering organization aware of the importance of software engineering to their mission.” I would substitute engineering organization with “any” organization. Software is everywhere and in everything. It is almost as ubiquitous as the telephone or power, yet we have no way to measure its quality, effectiveness or how productive we are at designing and constructing it. In fact, I would say that that any given orgranization is not even aware of how important the role of software plays in their very own organization.

And we software engineers, developers, programmers, coders, craftsman, or whatever we believe in (I believe that software design and construction is an engineering discipline) are our own worst enemies. Behold, I give you programming.reddit.com where almost 100,000 subscribers argue over which is the best is programming language, ever, every day, year after year. Or CodingHorror.com, with +100,000 subscribers, with its blog name, light hearted or not, sends the wrong message to the masses that coding results in a horror show. Or even Hacker News… How would any organization looking to solve a serious business problem that costs millions of dollars in software engineering take anyone of these seriously? Or just assume that the whole lot of us is represented by these blogs and news feeds? No wonder we have credibility problems. No wonder we don't collect stats!

So what we do about it? Just like anything, it takes a lot of hard work and some critical mass for an organization to realize that software may be a big part of their business and that maybe they should seriously consider looking at how software could be engineered for their business based on real software engineering principles and not on the latest and greatest fad of the month. I don’t know about you, but if I was to spend a million dollars on any software engineering project, I would want to be sure that it was on budget, on time, had some level of quality in it and actually solved the right problem. Establishing a baseline of what you are doing and how well you are doing it, is the start of real software engineering.

Programming Language Proliferation

Mitch Barnett — Wed, 15 Apr 2009 06:42:06 GMT

Ola Bini wrote an interesting article called, “Languages Should Die.” I was hoping that this was really the case, but actually it was the opposite. I believe it is this type of thinking (i.e. language proliferation) that has our software development industry in trouble:

“No existing language will die in a long time. But we need new ideas. New fresh ways of looking at things. And new languages that incorporate what other languages do right. That’s how we will finally get rid of the things we don’t like in our current languages.

You should be ready to abandon your favorite language for something better.”

Define better. Define do right. I feel we have already reached good enough long ago with C or Smalltalk. Heck, Smalltalk is one of the simplest languages there is - the entire language syntax can fit on a postcard. What more do we want? Oh sure, there are some small technical issues, like with all languages, but technical deficiencies is not what did in Smalltalk, it was marketing, but I digress.

Wikipedia says there are +560 programming languages. Sure they are not all in use today, but how many programming languages do we really need? How many written and/or spoken languages (i.e. English, Spanish, German, etc.) do you know well? Expert level? How many people do you know that are expert level in 2 or more spoken and/or written languages? How about 5 or more spoken and/or written languages? How long do you think it would take for you to learn 5 spoken/written languages at an expert level? Apply your CoCoMo model to that!

Yet, for any web developer, typically you need to know one or more core languages like C#, Java, Python, whatever, along with HTML, CSS, JavaScript and then usually your app is hooked to a database so you need to know some dialect of SQL. Ok that’s five languages right there. How many of us are expert in those 5 languages? I mean honestly. And of course there are variations for each language – example, look at the difference between C# 3.0 and 4.0. And for each language, there is one to many frameworks to learn as well – e.g. AJAX frameworks like JQuery, Mootools, Prototype, etc.

It is no wonder to me that we cannot get any real work done as we are all learning existing or new languages, frameworks, etc. It is no wonder to me that business is getting pissed off at developers in general because everything takes too long, too error prone and worse it does not meet their requirements.

Ok my point is not to be some old stodgy dinosaur, yeah I just turned 50, but I have a 6 year old daughter that can play Lego Star Wars better than I can. I am not trying to stifle innovation, but let’s be smart about what we are trying to innovate. Here is an excellent example in my neck of the woods, I spent a bunch of time learning C# since 2000 and kept up with the versions. Recently, I went to PDC and checked out all of the new C# 4.0 features. One of them was introducing a dynamic type to what is otherwise a very statically typed language. I used (Iron)Python for my dynamic needs and now some of that is in C#. Is that good, bad, better or worse or what? Damned if I know, it is yet another thing I have to figure out as a software engineer. Oh, and I really liked F#, a brand new functional language. So how many days, months or years do I need to invest in learning F# to become an expert? And most importantly, why would I do that? Other than the coolness factor, admittedly really cool to me personally, but what possible business value or real world application does it have that would cause me to use it? Answer = none.

See what I mean? As Ola says,

“So - you should go out and fork your favorite language. Or your least favorite language. Fix the things that you don’t like with it. Modify it for a specific domain. Remove some of the cruft. Create a new small language. It doesn’t have to be big. It doesn’t have to do everything - one of the really cool things about functional languages is that they are often have a very small core. Use a similar approach. Hack around. See what happens.”

Give me one sound business reason why this would be a good thing to do? You want a simpler language, well, we have that already, see Smalltalk example above. You want a better or a “right” programming language, you better have some real definitions as to what those mean, have identified real shortcomings (not just some syntactic sugar) in any other mainstream language, and your proposed improvement must be like 10X or why do it?

I feel that we already have (more than) enough programming languages to choose from. Let alone the frameworks and batteries that come with them. We software developers/programmers/engineers seem to be our own worst enemies as we are causing more and more complexity to a domain that is already complex enough. What are we really doing to reduce the complexity instead of adding more? Adding another language to my skill set like F# (which I would love to learn personally) absolutely has no business value for me or my customers in my world of ecommerce web applications.

In my software engineering world, I am looking at every angle to reduce complexity. It is simply a matter of numbers, the less is better. So if I can reduce the number of programming languages, frameworks, integration points, executables, assemblies, etc., the simpler the solution, the lower the cost, the less time to deliver, the easier it is to change and maintain and therefore representing the best business value to the purchaser of the custom developed software.

However, these days it feels like I am a minority in our software industry as we proliferate everything, including software development methodologies until it becomes insane. I am concerned that this is happening to programming languages (and everything else in our software development industry) as well. Just like the financial market situation we are in now, I am wondering when that will happen in our industry? For all of our brain power, we seem to be following the same path. What is it going to take to divert this path? When will the pendulum swing the other way to favor economic sense (i.e. proven software engineering principles) instead of the crazy proliferation of anything in the name of continuous improvement?

As Brad Cox says in, “Planning the Software Industrial Revolution, I only wish that I were as confident that the changes will come quickly or that we, the current software development community, will be the ones who make it happen. Or will we stay busy at our terminals, filing away at software like gunsmiths at iron bars, and leave it to our consumers to find a solution that leaves us sitting there? “

Software Development Gone Insane

Mitch Barnett — Thu, 09 Apr 2009 06:43:44 GMT

I may be getting old, but I don’t think I am out of touch. My sanity of being a software development professional seems to be tested daily by our industries predilection for Silver Bullets. The latest it seems is Scrumban. My wife can’t stop laughing when I say it to her.

I have not read the book, so I have no opinion on it. I have read some of the author’s blog material. Seriously, I mean no disrespect to Corey Ladas at all, but when I read the material, I can’t help to think that this is written by someone with a MBA. When I read this article, I think I get what he is saying, but I swear it is written in a different language than the software engineering world I live in.

Regardless of what marketing terms are used, the reality is that software development is always: understanding the requirements, creating a design, implementing the design, and testing the design and implementation to ensure it meets the requirements. Requirements, Design, Code and Test. Always have, always will, no matter what other fancy (marketing) words are used to describe it.

With respect to very large software projects, I understand the label Feature Crews, but to me, this is nothing more than classic functional decomposition at work, but with a new (fancy marketing) label.

Questions that the Agile, Scrum, Lean, give it a name, practitioners seem to never answer is the two most asked questions by people that pay for software development projects, which are, “how much is it going to cost and how long is it going to take?” It seems there is no best practice in any of these methodologies to answer these fundamental questions as these methodologies are focused on very limited scopes of the project as opposed to the entire project. And quite frankly, that is my main beef with these Silver Bullet’s, because as Fred Brooks postulated 23 years ago, there is no such thing as a Silver Bullet in software development. In my opinion and professional experience, he is right.

Software development is an incredibly complex, massively manual labor intensive effort, whose primary work product is lines of source code. For people that write code day in and day out, know this to be the truth. There is no hiding from it. We grind it out as we know how and love it. So when I am asked how much the software development project is going to cost and how long it is going to take, I apply a tried and trued approach to answer what is a very, very difficult question. This is why you see software estimation models like the Constructive Cost Model or CoCoMo for short. You can read the gory details of the model here. PDF alert! No surprise to see that counting lines of source code (or equivalent lines of source code) is the way to answer the tough question of how long and how much. Dr. Barry Boehm had it figured out years ago. When I took Software Engineering as a post graduate course at the University of Calgary , this is what we were taught and has been consistent with what I have found in the field. So yes, I use CoCoMo to answer the ugly, gnarly questions. And there are several automated tools that implement CoCoMo’s model, even ones online.

So what has happened to our software development industry that we need to keep reinventing the wheel in the name of continuous improvement? I think it is indicative of anything that is really tough to do. Everyone is looking for an easy answer or the next big thing. But as most things in life, the answer is already figured out. You just need to look in the right place, listen to wisdom and apply some common sense. Congratulations, you just made pro software engineer!

What’s my point? My point is two fold. Our software development industry is being run by marketing people and has gone insane ;-) Ok, I half jest. I know over time the pendulum will swing back to the basic fundamentals of what software development really is. My other point is to the young aspiring software engineers. Kids, look at some of the real software engineering techniques that hold up to the test of time. These are the gems. This is what is real in our industry. CoCoMo is real, it works, and is based on facts and historical proof. If you were to only read one book on software engineering, read the Mythical Man Month as it embodies what is really happening in our software development world, regardless of when the book was written. There is a reason why Fred Brooks earned the Turing Award. Your assignment is to understand why.

Brute Force Development (BFD)

Mitch Barnett — Tue, 10 Feb 2009 06:43:45 GMT

Man, it seems that every few years there is a new software development methodology that comes out and while not purported as the 2^nd coming, it certainly has all of the fanfare of the latest and greatest.

Let’s see there has been CMMI, TDD, Agile, Scrum, Lean, Waterfall, Continuous Integration, Spiral, Extreme Programming, RAD, MDD, YAGNI, Cowboy Programming, and the list goes on. I would like to add to the list an oldie but a goodie: Brute Force Development or BFD.

I would suggest that BFD is the most widely practiced software development methodology in the world. In fact, I would claim that the majority of organizations and people use this methodology daily and have been since the inception of software development.

How do I know this? In the real world of software development, where the size and complexity of even the smallest projects (e.g. >5000 lines of code) exceed the allocated budget and timeline, almost everyone resorts to brute force development in the end. Why? Because, we have to. How else can we do it? When was the last time, as a professional programmer, that you actually finished your project/product on time and on budget? Did you do it working 40 hours per week? Honestly?

Typically we start out with the best intentions, but as the schedule starts to slip and the budget is disappearing at the rate that a 426 Hemi goes through a tank of gas, we drop in BFD mode. We try and do the impossible. Extra hours are burnt, features are slashed, quality goes out the window, and we brute force our way to meet the impossible schedule.

Now, I am not complaining. This is just an observation having worked in many different shops, large and small, including my own start-up. We end up with BFD in the end.

Heck even the guru himself (who I have nothing but respect for), in his own post called, "Building a Fort – Lessons in Software Estimation" made some pretty interesting slip ups. My favorite was the, “I dropped a little piece of my laser level down the side of one of the footing holes, between the concrete form and the dirt, after I'd poured the concrete.” Oh Steve, think of all the things we have dropped in the software!

I will go out on a limb by saying I have yet to see any evidence that we, as software architects, developers, estimators, etc., are actually getting any better at this. I have been doing this for 18 years professionally and maybe I am dreaming, but it seemed simpler years ago. Not just that the requirements were simpler, but even from a technology standpoint. What I mean is that software vendors that produce tools, programming languages and applications have grown (seemingly) exponentially during this time frame as it seems any solution (and the tooling, languages, and apps) I am involved in has way more moving parts. A lot of these moving parts are new and unforeseen issues crop up well into the development cycle where one vendors library interfaces don’t seem to match what the documentation says, for example. And then we brute force it – to make it work.

Part of this is tongue in cheek as there is another meaning to BFD than can be applied to programming methodologies. You can get a hint by looking at the blog category this was filed under. I sincerely don’t mean any disrespect to the authors and believers of these software development methodologies, but sometimes the “marketing messages” can be a little much and even downright embarrassing. For example, try explaining to your significant other what Agile and Scrum mean. What do you think the “business folks” are thinking when you explain it to them? Do they even care? I would hazard a guess that all they care about is how much is it going to cost and when can we start using the software. Btw, they are also thinking, it better do what I want it to do for this amount of money... or else...

So the moment that things don’t go as planned, BFD kicks in. Whether you know it or not.

In 1994, before most of these methodologies and marketing names came into effect, I had the good fortune of taking a 2 year post grad course in Software Engineering Management at the University of Calgary in Alberta , Canada . It was taught by Motorola University and one of the instructors, with 30 years experience, had some awesome stories on how “yer doin it wrong.” The funny thing was, while we learned a great deal about software engineering (that’s the last time I write 17 exams in 2 years!), what we learned most was common sense and communication. In other words, how to tell your customer (ahem, the one paying your rate, salary, contract, or whatever) that we can’t write 100,000 lines of code in 2 weeks. The real methodology here folks is just called common sense.

I don’t think much has changed since then as we are always fighting that battle. Developing software for any decent sized project (>5,000 lines of code) is really, really hard, maximally labor intensive and fraught with… well, you name it.

I can hear the Agile folks saying that our methodology is the one that mitigates this risk. While that may be partially true, how do you answer the top two questions asked by the customer: how long will it take? And how much will that cost? And our requirements list is just that, a one pager with bulleted high level fatures items and some of the bulleted items have two words explaining the requirement. Oh yeah and at fixed price. Ready to sign up? In the end, in order to make that deadline or not burn through your fixed cost, it's BFD man. That’s the reality. And btw, could you not have come up with a better name. I mean did you not know that Agile is Dead?

So what’s my point? Well, aside from having some fun with the BFD acronym, as with most things, there is some truism there for sure. We have all done it, yes? I am sure that anyone that has written code for any length of time has done BFD. Which makes my newly minted, TLA marketing buzzword an instant leader in the world of software development methodologies!

All kidding aside, maybe it is time to step back and look at some of the basics for any software development project. The very first I would think is answering the two most basic questions of any software development project – how long and how much $’s. Do you have a predictable and repeatable way of doing that? How accurate is it? If you don’t, then you are likely to be doing BFD even before you write a single line of code and therefore none of those fancy software development methodologies won’t help you one bit. Know what I mean?

Remember, keep the rubber side down!