Shouting Distance

What Constitutes Sufficient Testing During the Building Phase?

eholman — Sat, 11 Jul 2009 15:52:15 -0400

Software development has two general phases, the building phase and the verification phase. In the building phase, the attitude towards change defaults to "allow". New features can be added, code may be significantly reworked, and designs might be altered, as the goal is to add new capabilities to the system at the expense of increasing the risk of introducing new flaws and problems. Once the building phase has substantially completed, the verification phase commences, where the attitude towards change flips to default "deny". The goal is to achieve a stable system that performs the desired features. Since every change has the potential to introduce errors, changes are only made to correct known flaws.

Obviously, a significant amount of testing is done during the verification phase to assure that the features are implemented as specified, and that no new problems have emerged. However, some amount of testing must be done in the building phase to convince the developer and the team that things are near enough to stable that it is worth switching the change policy to default "deny". The big question is, how much testing is enough? There are significant dangers to insufficient testing, but too much testing also has consequences.

Dangers of Insufficient Testing

Wasted time - Blocker bugs found during verification often mean that testers and developers sit around waiting for an issue to get resolved, and that wastes everyone's time.
False sense of security - There is a general feeling of excitement and progress when things move to verification, that the application is moving towards complete. If it turns out that major bugs exist and features are not implemented correctly, this can throw the whole schedule into disarray.
Blurring of phase line/Vacillation - This is the most dangerous result of all, a general blurring of the line between building and verification, or constantly switching back and forth between phases. The end result is an "anything goes" policy towards change, and a general march towards testing hell where no features can be verified because developers feel free to change anything at any time. It starts simply enough, where a couple of big bugs make it into the verification phase so things are moved back into building mode. Then it's back to verification after fixes are put in place but testing still wasn't done sufficiently and new bugs are found or the old bugs are rediscovered, but now time is getting tight and the team is encouraged to just get it working so changes start getting made wherever needed at all once. At this point, no more guarantees can be made about functionality. Essentially, this is permanent building mode. It is the sad truth that many applications ship directly from the building phase, without ever having a verification phase, and appear to do so successfully. IT WON'T WORK LONG TERM. Code can ship a few times under these conditions, but eventually this attitude will cause the codebase to crumble under the weight of a million changes with no external verification. It is accruing technical debt, and just because no one is watching the balance go up doesn't mean it won't eventually come due.

Given all these dangers, why not doing as much testing as possible during the building phase?

Dangers of Excessive Build-Phase Testing

False sense of quality - Developers are notoriously bad at testing their own code. It's too easy to fall into the trap of checking the few code paths you've spent the most time on and ignore some of the other paths that just happen to be the ones most frequently experienced by users. Developers also have a built-in disincentive, which is that testing finds bugs and that means more work for them, so they are more likely to turn a blind eye to a real problem or dismiss it as "unlikely". Like under-testing, over-testing can create a false sense of confidence in the code because so much testing was done, when in fact a small portion of the code was heavily tested, but in fact many new issues are likely to be found during verification.
Skill mismatch - Testing is a separate skill from development, and one that requires a very different mindset than development. Some developers are also good testers because they are good at thinking like users, or are good at thinking up diabolical ways to break their own code. Most are not. They can produce quality code but haven't developed that additional skill set. In short, developers are generally not very efficient testers, and asking them to do testing beyond what is necessary to move to verification is inefficient.
Perfectionism - If the management and team come down too harshly on developers for letting even small or unlikely bugs through, developers will tend to respond by becoming perfectionists, never letting their code be called "done" until they have gone over every line, every usage scenario, and every potential input. Developers become loathe to make changes because it means their code might have a bug and they will get an earful from the team. This result is nearly as bad as the "Blurring of Phase Lines" case, but in a different way. Rather than live in the build phase forever, the verification mindset takes over the entire process. Instead of shipping code of questionable stability because there was never a verification phase, code never gets shipped at all as every developer is too worried about protecting their reputation.

Unfortunately, trying to require too much testing can be just as bad as requiring too little, so then returning to the question of how much testing is enough, suppose that during the verification phase, two different bugs are found:

If more than 10000 transactions are rolled back simultaneously, a system crash results.
The application will not launch.

Which of these ought to have been found and fixed in the building phase? Most observers would likely agree that item 1) is probably not the kind of thing that could or should be found in the building phase, and item 2) is most definitely the kind of thing that should have been found. Somewhere in the middle is a great gray area of bugs and issues that some believe should be rooted out during building, and others might consider a verification phase find. Even if the opinion on a particular issue may vary from developer to developer, it is useful to establish some criteria to determine what bugs might indicate that the building phase code has or has not been sufficiently tested. If a bug is found during verification, consider the following:

Recency - Might this bug have existed for weeks, months, or even years and the new work has somehow uncovered it, or was this bug clearly introduced by the latest work? Developers can easily break parts of the code that they do not use frequently, or that has been around for a long time because they assume that it has gone through many rounds of testing and is hardened. It can be very hard, and it is probably inefficient in the build phase to throughly test enough of the system to find bugs that have stubbornly hidden through many rounds of testing. On the other hand, if a developer adds a feature and it is found to be immediately broken, that's a red flag.
Obviousness - How likely is it that a developer performing a reasonable testing regimen would have come across this bug? In the examples given, the application failing to start is horribly obvious, while a failure that occurs after a large number of concurrently rolled back transactions is much less so (unless this is a known common condition in this application). Obviousness is a value judgment though; what is obvious to one developer might not have been to another, especially those with more rigorous testing practices. Still, a developer whose code frequently has obvious bugs on the main code paths is generally not doing enough testing in the building phase.
Frequency - How often does this bug occur? This goes along with obviousness in that an infrequent bug might not show up in the kind of basic testing done in the build phase, while frequently occurring bugs should be found and rooted out. There are frequently occurring bugs that might not show up in any of the obvious code paths though, for instance, a failure that occurs after the user clicks "Back" on their browser and resubmits a form. This might be something that happens all the time with a web application, but which a developer might not think to test.
Severity - Less important than the other three, but still something to consider. A developer can easily overlook something important to users but which doesn't result in errors or crashes, for example, the tab order of a set of fields, because it's simply harder to test for that kind of thing. On the flip side, a very severe issue such a system crash that only happens under very infrequent or even apparently non-replicable conditions can go equally unnoticed. However, serious errors that are easily detectable just by walking through the application should be caught during the building phase.

To summarize, bugs that would have been added during the most recent set of changes, are obvious (according to one's own definition of obvious), frequently occurring, and severe should all be stamped out during the build phase. After that, it's up to the manager and the team to decide what level of testing to do before moving to verification, and that will be a factor of both the experience and skillset of the team, and the type of project being implemented. By choosing this level wisely and frequently adjusting it through feedback, both developers and testers can work efficiently to create a stable, feature-rich product.

Fake Book Management

eholman — Thu, 11 Jun 2009 23:20:16 -0400

Most of the Jazz repertoire that I know, I learned from "Fake Books". For the many non-musicians who are unfamiliar (and musicians in most other genres who do not make use of them), a Fake Book is a heavily simplified version of a song, generally only containing the melody and simplified chord symbols. For an experienced player, this is enough information to "fake" their way through an unfamiliar song. However, learning to play from a Fake Book is a complex skill. As aptly summarized in the Wikipedia article:

Fake books are not intended for novices: the reader must follow and interpret the scant notation, and is expected to have thorough familiarity with chords and sheet music. However, fake books can be an avenue to playing songs quickly; a few chords and a one-note melody line can allow even an amateur to play a passable version of any song with relative ease.

When I first began trying to learn from fake books, I found them opaque and frustrating. How can I be expected to create some semblance of a particular song with only a melody and a few chord suggestions? After playing through dozens and dozens of songs, I did steadily improve my ability to play decent versions of the songs I encountered, eventually becoming able to play passable versions of many songs on the first read through. As I developed these skills, a couple of key points jumped out at me:

Knowing what the song is supposed to sound like reduces the difficulty by an order of magnitude. Since I listen to a lot of Jazz, I had heard many of the songs even if I didn't know what they were called or had any previous sense of the chords and melody. Familiarity with the song helps you know if you are on the right track with the feel and how the chords should lead to one another.
Faking it is mostly pattern recognition. The same harmonic patterns (sequences of chord changes) repeat over and over and over again. The vast majority of chord sequences are drawn from a small set like ii-V7-I (for you music theorists playing at home), or slight variations on those sequences (like a tritone substitution for V7). The rest are either less common but still notably recurring patterns, or a tiny number of novel sequences that are what make a song unique and interesting . If you can recognize the sequences from the very common set and 50% of the less common ones, you can easily play 95-100% of the song already.
A little music theory goes a long way. I'm lucky to have received a standard course of instruction in music theory as part of my undergraduate music degree, but even knowing a little about voice leading, harmony, and scales allows you to fake it much more effectively. The theory helps fill in the blanks left by the fake book with the best possible guesses.

An interesting thing happened after I had been playing from fake books for a while: I stopped being able to play standards from "real" sheet music effectively. It's not that I couldn't do it; I could easily sight-read the notes and faithfully reproduce what was written much as I would for a piece of classical music. I just became frustrated with the way the music was represented on the page. It was way, way too much information. Instead of having a nice set of guide posts from the melody and simplified chords meant to be embellished, I found myself doing this awkward process of looking at the notes, parsing the elaborate and often misleading chord symbols, and then attempting to reverse engineer and extract a fake book version of the song so that I could produce music that sounded like "me" and not like the version written on the page. It was incredibly tedious. It was also very unexpected that the return of the safety net of fully specified music could prove to be so unpleasant.

As I reflected on this experience, I though about how much harm we do as technical managers by overspecifying the job to be done, giving developers sheet music instead of the fake book version. Part of this stems from insecurity, not wanting to let developers do the wrong thing, and certainly many newer programmers need that level of instruction, but the tendency should always be to remove and cut until just a suggested route to completion (the melody) and the critical requirements to be met (the chords) are left. This gives the developer the creative freedom to complete the task in a way that makes sense to them and helps them establish ownership while still providing a framework for execution.

Not every developer can work with a manager who asks them to "fake it", but I prefer to hire and work with developers who can. By reducing task specificity, I can spend less time giving instruction and leads to a more efficient style of management. More importantly, it also means that I often get better solutions to technical problems than I would had I more fully specified the task. It can be scary giving that much leeway to the development team, but in my experience, reducing task specificity leads to greater independence and confidence on the part of developers, and that means a more flexible and capable workforce.

Technology Focus: XBRL, Part II - What it can do, and One Major Caveat

eholman — Sun, 21 Dec 2008 22:42:15 -0500

In part 1, I described XBRL and a bit of its background. In Part II, I'll talk more about why it is useful, and one major caveat to keep in mind.

As mentioned in Part 1, XBRL has now been mandated for use by companies filing disclosures with the SEC, starting with the biggest companies and eventually requiring compliance from all submitters. Clearly the SEC believes that the XBRL filings will be superior to the previous ASCII and HTML text submissions, and I think they are right. While I don't have a lot of visibility into the SEC's thinking process, I can speculate on the advantages XBRL brings to them:

Validation - Built-in to XBRL is a notion of calculation linkbases, which describe how the values of certain taxonomy concepts must balance. This is as much for the submitter as for the SEC, as it can easily eliminate bookkeeping or data entry mistakes since it will catch errors such as two values not summing to a third value, or two values not summing to zero. These checks are not intended to catch a malicious document submitter who could easily manipulate multiple values to avoid the calculation checks, but it should help cut down on errors such as dropping a zero or missing a negative sign. More of these validation capabilities are being added over time.
Reporting - Because XBRL documents are easily processed by computers, and must conform to the US GAAP Taxonomy, reporting is greatly simplified. You can take the underlying data encoded in the document and present it in different formats, compare and contrast different companies using different criteria, and even write queries against the data like you would in a standard database. To see some of this in action, check out the SEC Interactive Financial Report Viewer, which allows you to drill down into XBRL submissions to see a glimpse at how having data in a consistent format can unlock serious reporting capabilities. More importantly, the old EDGAR system is being replaced with the new IDEA system, which is built around the notion of businesses providing data, which can be queried and analyzed, instead of providing documents, which can only be searched.
Communication - Similar to reporting, the ability to communicate financial reporting data to interested parties is greatly simplified due to the standard format of XBRL and the UGT. Right now, I can go to the SEC website and pull down the 10K for Microsoft corp and I get a huge HTML or PDF file with dozens of paragraphs and numbers. This information is in a relatively standard format, but even still, this is not an ideal communications format for anyone who cares about particular aspects of their business, or desires to know particular pieces of information. SEC forms are great for the SEC purposes, but since the data is stuck in document form, it is difficult to make it serve any other purpose. Once the data has been converted to XBRL, any tool that supports the format can use it for a novel application. Mark Cuban hinted at this in his recent blog post, in which he envisions XBRL as the solution to the problem of regulating new financial instruments, and as a way for government watchdogs to more easily monitor government spending.

So the move to XBRL is going to be a major step forward, that much should be clear. However, one thing I have noted on occasion is that people who first encounter XBRL look at it and see many of the same goals and capabilities as the Semantic Web, particularly RDF/XML, RDFS, and OWL, but XBRL is a syntactic standard, and not a semantic one. It defines how things should be said, and not what they mean. Looking at a taxonomy concept, it is easy to see a term called "Cash" and assume that anywhere you see a taxonomy concept called "Cash" in any taxonomy, it must mean the same thing. However, not only is there no guarantee that all "Cash" terms are equivalently defined by the users, but information about what defines "Cash" is only lightly covered in XBRL via the reference info that names an external resource where a concept definition can be found.

We can hopefully assume that as long as different instance documents use the same element from the same schema, they have the same intent, but across taxonomies, all bets are off. Unlike with RDF, where particular concepts are defined as resources and can be referred to by any other resource, XBRL is rooted in its taxonomies and not in a set of core concepts. There are some efforts underway to bridge this gap and create more formal definitions of concepts, but as you might imagine, defining concepts in a consistent way across accounting systems is a daunting task. Someday though, perhaps individual taxonomies will begin to reference certain core, shared concept ontologies.

I hope you found this brief introduction to XBRL clear and useful. Questions? Clarifications? Criticisms? Looking for XBRL development? Drop us a line!

Technology Focus: XBRL

eholman — Sat, 20 Dec 2008 22:37:27 -0500

Distance Software takes on a very wide variety of projects. That's part of the fun of being a consulting company, and a major reason why I decided to start a business. In our work, we often become very knowledgeable about certain applications, tools, libraries, and specifications. As many of these are niche items, do not receive the same level of discussion and review that bigger technologies enjoy, I am starting up a new feature here on Shouting Distance called "Technology Focus" to help broaden the coverage of these items.

The technology in focus today is XBRL, the eXtensible Business Reporting Language. XBRL is at its core, a set of specifications for the encoding of enterprise data in XML format. XBRL can used to describe two main things:

Concept Taxonomies - An encoding of a set of business concepts along with significant additional metadata for describing the presentation, validation, and source of the concepts. For instance, the taxonomy might contain a concept like "Cash and Cash Equivalents" with metadata indicating that it is an instantaneous measurement, i.e. it represents a value at a point in time, not a change over a period of time, and that it must be represented by a number.
Instance Documents - An encoding of a certain set of data for a particular business that refers to a particular taxonomy. For instance, The instance document using the taxonomy described above might have an entry for the "Cash and Cash Equivalents" concept with a value of 100, with unit information indicating that the number is in millions of USD, and context information indicating that the number represents the period ending March 31, 2008. The document is an "instantiation" of the concepts described in the taxonomy, hence the name Instance Document.

XBRL got a major boost recently when the SEC mandated that all companies with a market capitalization of more than $5B would be required to file their financial disclosures using XBRL starting in 2009, with the remaining companies required to comply by 2011 (See this Gartner article for more information). The SEC spent a long time creating their own taxonomy explicitly for this purpose, the US GAAP Taxonomy (UGT). The SEC filing system has been updated to accept XBRL instance document submissions that are built using the UGT, and under a voluntary XBRL program, filers could attach an additional XBRL document (not considered part of the "real" filing) to their standard filing to test the new system. However, starting very shortly, XBRL submissions will be the standard.

The XBRL specification itself is fairly complex. While it makes use of standard XML concepts, such as that each taxonomy is a valid XML Schema Document, it also uses other standards such as XLink and XPointer to allow a taxonomy to be dynamically composed. Using an initial taxonomy document as a jumping off point, a valid XBRL schema processor must follow the various linkages and pointers to grab all of the directly and indirectly referenced documents and concepts to produce what is referred to as the Discoverable Taxonomy Set (DTS). The specification and the rules for generating the DTS can be found at xbrl.org, the international XBRL organization.

This can all seem overly complex at first glance, but ultimately, it needed to equal the complexity of the representation problem, which is only all possible financial data. Consider the following challenges, XBRL needs to:

Allow for the representation of data across many types of reports, from simple balance sheets and cash flow statements to the risk/return information in a mutual fund prospectus.
Support taxonomies that are generally useful, such that a large number of companies can all report on a similar set of concepts in a consistent way, but at the same time, be flexible enough to allow an individual company to represent additional data that is generally acknowledge but optional, down to concepts that are totally idiosyncratic to a single company's operations.
Work for financial reporting concepts around the world. It can't just work for GAAP-based accounting, but it must work for IFRS reporting, the British system, the Australian system, the Dutch system, the Japanese system, and so on. While these different accounting systems share many common features, they each have a unique set of standards and concepts that must be supported.
Be able to adapt as the standards change. Not only must it function across all financial reporting systems, but those standards change over time, often extremely rapidly.

As much as I appreciate simple, elegant standards, the world of financial reporting is neither simple nor elegant and it requires a solution that can meet these and many other challenges.

Tomorrow: Part II - What XBRL can do, and a few Caveats

Why do I still use Java?

eholman — Fri, 14 Nov 2008 22:18:24 -0500

I occasionally get asked why I prefer to develop in Java versus some of the powerful or flexible languages that are currently in vogue such as Ruby or PHP. My response is that I tend to build a lot of enterprise software, for which Java is better suited. Enterprise generally refers to software built for business purposes, but it goes far beyond the intended audience. Enterprise software is generally:

Very large in scope, with features in the thousands or tens of thousands.
Intended to be used by a large set of users over a long period of time, sometimes several decades, with new, possibly complex and unexpected features being added over that time
Developed by a large (relatively) and often frequently changing team, as required by a complex system needing a variety of specialists that is developed and maintained over decades.
Built to meet certain standards of uptime, performance, and overall robustness so that users can rely on it to perform their jobs

This is not to say that Enterprise software is the better than other software, or more useful. Most of the software that we use, even at our places of business, does not meet these criteria. Consider the Windows Operating System versus the Google search engine. Both are ubiquitous for many computer users, and both are large, complex systems meant to be used over long periods of time, and were developed by large, changing teams. However, if google.com goes down (which I am not aware of happening in recent memory, but it's possible) we are frustrated and annoyed, but we can wait for it to return, or temporarily use a different search engine. If Windows crashes or otherwise fails to operate, we are essentially blocked from doing anything productive (short of installing Linux). As much as we all enjoy a few jokes about the robustness of Windows, in the long run you have to admit that it is held to an extremely high standard. The Google search engine is not Enterprise software, and the Windows Operating System is, for just this reason. Google does make enterprise software in the form of GMail and Google Docs, and they are quickly learning what a different world it is as they scramble to fix outages that leave their enterprise customers flailing and furious.

Java is especially suited for enterprise development because the complexity of the development process scales well as complexity of the application increases. Here is a simple graph that illustrates that concept:

This graph is my rough perception of how the complexity of the development process increases as the complexity of the application increases. At the low-end of the application complexity scale, the things that Java does for you to help you manage a large amount of code mostly get in your way. However, as the application increases in complexity, those things eventually start working in your favor and prevent things from getting out of control on the high end. Meanwhile, languages that are much better suited to that low to middle region shoot quickly upward.

Java specifically addresses, as part of its core feature set, the enterprise requirements called out above:

Large feature set - Java expects that you will have a large code base supporting a large number of features. With core language and toolchain capabilities such as packages and jar files, Java expects that you will be decomposing your system into namespaces and reusable groupings. This helps manage a large code base, and provides a natural structuring of the code into named functional units. Very few other languages have such a heavy emphasis on helping you manage how your code is organized and released.
Application used over long period of time with many modifications - With the near perfect platform independence of the JVM and libraries and dynamic linking, Java makes it easy to migrate to new platforms and to integrate with new tools.
Developed by a large and changing team - Java is a great maintenance programming language. Pedantic features like the requirement that the package name match the folder name and the class name match the file name seem annoying at first (and nowadays if you are using an IDE, which I'm sure you are, you probably don't even think about it) are a godsend when you need to step into a new code base. Just look at the fully-qualified name of a class and you can more or less know where you ought to be able to find the .java file. Ever tried to track down a C++ class in one of the two dozen header files that are #included? It can be a nightmare, even with an IDE. Java is often criticized for its lack of features like multiple inheritance, mixins, operator overloading, dynamic method binding, or a preprocessor, but these things, while powerful in the right circumstances, can add a huge amount of complexity to a codebase while offering few if any capabilities that cannot be provided in another way. When you have a lot of developers coming and going on a project, its nice to know that they will spend their time writing new code and fixing bugs instead of trying to track down where some_random_ruby_method() is defined.
Robustness - With compile-time type checking, automatic garbage collection, no ability for users to manipulate memory directly, simple stack trace facilities, and descriptive errors for most programmer error conditions, it is very easy to write a robust program in Java. Like the features in Java that help with maintenance, many of these features are often seen as drawbacks to those who are used to relying on them, but they often do more harm than good. There is a reason so many server applications are Java based nowadays.

So that's a long-winded way of answering the why Java question. I feel that it is the best tool for the kinds of systems I tend to build because it was created by people who understand the challenges of developing applications that meet enterprise software requirements.

Public Policy for Entrepreneurship

eholman — Mon, 10 Nov 2008 22:42:27 -0500

In a recent post on Blog Maverick, Mark Cuban gives P-E Obama a low mark for failing to include any entrepreneurs in his economic recovery team. Leaving aside the question of whether or not the current economic situation could be helped by entrepreneurship or if there are low-level economic issues that need to be resolved first, it got me thinking about public policy as it relates to entrepreneurship. If the government decided to pick establishing new ventures as one of its key policy goals, what would that legislation look like?

I have owned and operated Distance Software for over a year now. Before that I worked at a small research company for seven years funded mostly by government contracts, and before that, I worked for a B2B startup that collapsed in the dotcom boom. I have never worked for a company that more than 50 people. I have never worked in an office that had more than the office I work in right now (8, when both companies in the space have all their staff in). With these experiences, I feel fairly qualified to talk about what might encourage and support entrepreneurship from a public policy standpoint. Here are my top five initiatives:

Fix the #%@$&ing heath care system - The debate over health policy is hopelessly framed as a choice between a "socialist" single-payer system and the current employer-paid system. The Obama campaign ran ads here in Wisconsin in which it described his health policy as somewhere in the middle of the extremes of an "expensive single-payer system", and the current system, described as "Insurance companies in control, denying coverage", or something to that effect. With this kind of view, I have to agree with Cuban that they are blindly failing to include the entrepreneur's perspective on what would make good policy.

Personally, the number one thing that prevented me from starting my own business years ago was the knowledge that I didn't know where or how I would get health coverage. I can't be the only business owner who has felt this way. Now that I'm in business, with every hire I make I have to consider how I will offer them some health benefit if I hope to be competitive. Because of the employer-paid system, every single business in America is also in the business of navigating the health insurance industry, and has to spend some amount of tie and energy picking a plan, administering that plan, and accounting for the plan costs in their books. Sounds pretty inefficient doesn't it?

The current health care system is rabidly anti-entrepreneur. I don't claim that a single-payer system will solve all the problems, but pretty much anything would be better than what we've got. I can guarantee you that if Americans had universal coverage that would cover them regardless of their employment situation, you'd see the market flooded with 30-something entrepreneurs who suddenly have no reason not to take the plunge.
Create Something like SCORE, but More Useful - Once I had gotten a lot of the basic elements of my business in place that were easy to figure out with a little research, like contracts, invoicing, employment issues, payroll, and an accounting system, I had a lot of questions that were more esoteric and, in my mind, required the input of a business veteran. I had questions like:
"I'm starting to hire, but a lot of my clients imagined that they would be getting me and only me on their projects. How do you transition from one to many as a consultant without making clients feel like they've gotten a bait-and-switch?"

"How soon should I hire an administrative person? Can it be done too soon?"

"Are there any short or long term drawbacks to incorporating as an LLC versus a standard corporation given my corporate goals?"

A few of the resources I had used recommended contacting the Service Corp of Retired Executives (SCORE), as a way to speak with a real live veteran of the business. I contacted my local branch and arranged to meet with someone there. The guy I met with meant perfectly well, and did answer a few of my questions, but he was just a volunteer trained by the organization and had kind of a stock routine he followed. I listened to his info and took their pamphlets, but I kept waiting for him to say, "and then we'll set you up with one of our Executive members", which never happened. Perhaps it's different in other parts of the country, but overall I felt like it was a huge missed opportunity to match up old with new and answer some of those esoteric questions. I believe that an organization, maybe even SCORE with some tweaking, could be federally created and funded and to fill that role.
Qualified Tax Holiday - This is a general policy idea that makes sense for entrepreneurs. It's a simple concept: anyone can request that their payroll taxes be waived for one year. There are some strict rules that try to disincentivize gaming the system:
- Note that it only applies to payroll taxes, so people can't go selling off stock and homes during that year and avoid capital gains.
- You can only do it once (obviously).
- It only applies to the first $100K of income, so there is also not much point in trying to do much other than work real hard that year.
- It can only be used for personal taxes, so corporations cannot use it to avoid their tax liability.
- It must be declared in advance, not retroactively. You file a form with the IRS saying I'm not going to pay taxes next year, and then you are exempt. That makes sense for entrepreneurs, and others who expect they might have a dicey financial year.

In general, I despise the notion that taxes are what make it hard to keep a business open, or what keep people from starting businesses. Mark Cuban had a post on this same point a few weeks back: entrepreneurs simply don't care about what the marginal tax rate is, they are are worried about their big idea. On the other hand, during the first year of a business, you will probably have a cash flow problem one or (many) more times. You need to budget for and regularly pay your own estimated taxes, and you probably are cash poor since you may have money tied up in inventory or waiting on payment from clients. If that burden can be eased on entrepreneurs, they can use that money to establish a nest egg to get through tough times, invest in a piece of equipment, or make a payroll that wouldn't have been made, all keys to keeping a business running. The goal is simply to increase the success rate during that critical first year. If non-entrepreneurs choose to utilize it for a little extra pocket cash, so be it.
SBA Payment Gap Fund - This is a fairly simple idea that might make a huge difference to a lot of small businesses. A core problem with a new business is that while you both bill clients and pay staff for work completed, your staff has to be paid immediately and your clients generally have some delay in their payment, usually 20 or 30 days. Once you have been in business for a while, you create a cushion that allows you to pay staff without issue and then backfill when invoices are paid. Up until that point though, you generally have to either delay paying staff, borrow to cover your costs, or avoid paying other bills to create the cushion, none of which are good options.
To address this problem, the SBA could create a "Small Business Payment Gap Fund" that works as follows:

- Small businesses wishing to use the fund would submit their outgoing invoice and along with a letter of agreement from the payee indicating that the amount would be paid within a certain number of days.
- The Payment Gap Fund would issue an immediate payment (minus 1% for using the service or something along those lines) for the amount due, and send a bill to the original payee.
- The original payee would then pay the Fund instead of paying the entrepreneur.
- As long as the clients are paying back the Fund, the entrepreneur can continue to use the service. Once the overdue amounts exceed some level, the service is frozen and cannot be used by that business until the balance is paid.
- If the client does not pay, the entrepreneur eventually becomes liable for the money.

The idea is that good businesses with paying customers who simply need better cash flow can benefit, but it avoids the hassle and risk of a debt system since the loans end up being very short term.
Require Inclusion of Small Businesses in Contract Bidding - This one is specific to the tech industry. State and Federal Governments frequently contract with private industry to design and develop custom computer applications. Generally, they get bids from the same set of clients over and over again, large consulting firms who specialize in this kind of work. Since almost without exception, the work goes to the lowest bidder, they hire a lot of entry-level staff and pile on a few layers of management to get things built. This makes it appear as though they have a large team ready to go that can still work on the cheap. Of course, this flies in the face of the consensus opinion of how good systems get built: small high-quality teams, little or nothing separating the users from the developers, frequent releases with immediate feedback from the users, and so on. The end result is highly predictable: big, unwieldy systems that don't meet user needs, take far longer than agreed upon to finish, overrun on cost, and generally fail to meet the stated objective of the contract.
I remember a recent example: the Milwaukee Public Schools contracted to have a system built to help school administrators manage their students' standardized test results so that they can better do their work of reporting current performance levels and creating plans to address shortcomings. The system was built by some big software consulting shop and delivered to the vast majority of users in a single release. On the day that it was most needed, when many of the administrators began running their reports to get key data, the system ground to a halt, making it basically unusable and creating a major crisis. Many users resorted to staying up until the wee hours so that they could log on with then load was lower and get access to their data.

I was so frustrated when I heard about this because I knew that a small team doing frequent releases and good unit and load testing practices could have easily delivered this system with none of the issues. However, if a small company like Distance Software had submitted a bid, they would have looked at the size of the staff and tossed it. The people doing the buying don't understand that building software is not like building a house. If you want to build a big house, you need more people because there is more work to do. If you want to build a big piece of software, you want to stick with a small team because communication overhead is what kills you.

So what to do? Force government agencies to consider small business proposals on their merit and cost and require them to ignore the size of the company. Require a certain percentage of final bidders to be small businesses (and don't use the Federal Government definition of 500 people; that's insane). Once a few agencies get really good, really cheap, on-time software built by a couple of developers and a designer, they will hopefully realize what they have been missing.

So there are my five initiatives that I would propose if I were put in charge of encouraging new ventures and supporting existing ones. What would you add to the list?

Am I Right to be Skeptical of non-IDE Developers?

eholman — Thu, 06 Nov 2008 22:04:23 -0500

I've noticed myself becoming biased against developers who claim to not use any IDE as part of their normal course of development. Am I right to think less of their abilities and productivity because they choose to develop in a text editor, even a powerful one meant for coding? This bias isn't uniform, as there are cases where an IDE makes more or less sense. I would say that:

My bias is much stronger in the case that the development community for a particular language or technology tends to develop in an IDE, and there are several very powerful and well-thought-of IDEs, see for instance, the Java development community.
My bias is stronger in the case that the developer is/has been doing Enterprise-level development. I have a forthcoming post discussing that concept in detail, but in a nutshell, I am biased against non-IDE developers who are building something where users are paying for the service and expect a high level of robustness and stability over a long period of time.
My bias is much weaker in the case that a user is a recent grad and may not have been exposed to IDE-based work, or is doing research-y type work that might be hindered by an IDE.

The development world seems split on the issue of whether or not IDEs are a Good Thing or Bad Thing for developers. For every "10 reasons to use an IDE" article out there, there is a corresponding rebuttal. This article from a few years back goes into great detail explaining the core difference between Language Mavens and Tool Mavens, where the former tend not to use IDEs since they lack support for the latest and greatest languages and features, while Tool Mavens favor the power of tools over language capabilities to magnify their productivity. However, in my experience, being a language maven , i.e. using cutting-edge languages and features, is something of luxury in The Real World, where you are interfacing with legacy systems and often have to work with what is already installed somewhere, so your choices are limited. In that sense, I discount the Language Maven perspective.

In a broader sense, I don't actually understand how developers create systems of any significant scale without using an IDE. It just seems like you lack so much context that performing a task of any complexity becomes unwieldly. Speaking from my own experience, I have built systems that would have been basically impossible to create if I had attempted to write them outside of an IDE. My experience may be unique, since I do primarily Java development, but here are a smattering of things that Eclipse handles for me that emacs (my old "IDE") could not:

Build Automatically - Every time I save a file, the entire source tree is rebuilt. I see any errors or warnings I've created instantaneously. I actually can't remember what development was like before Build Automatically. Oh wait, yes I can: I also use Visual Studio regularly.
Refactor Change Method/Class/Variable name - Without this capability, names have inertia. It's a pain to find every reference and update it, so things have names that are out-of-date. They don't accurately describe what something does because the nature of a class or method has changed over time but no one wants to fix it. Misspellings are never corrected. It leads to subtle, but painful, bit rot. In Eclipse, I can change the name of a class or method or variable as fast as I can type it, and all references are automatically corrected, so I have no barrier to keeping names up-to-date.
Run Tests - I can one click run my JUnit test suite, so I do it all the time. You can do this without an IDE, but it's more difficult. When I used emacs, I would keep a shell open and rerun the tests, but it was much harder to process and understand the results, and to remember to keep running the tests because they were out of sight and out of mind when I was writing code.
Ctrl-Click to Jump - I'm guessing emacs could be retrofitted with this one, but I don't think I've ever seen it. In Eclipse I can hold Ctrl and click a method to jump to the method definition. When I'm done, I can use the "back" button to go back to where I was editing. In the newest versions of Eclipse, I can hover on a method to see a some or all of the source code in a tooltip without leaving the current file, while frequently eliminates the need to ctrl-click at all, and helps me avoid breaking my current train of thought.
So if I talk to a Java developer who doesn't use an IDE, they are
Find References - It's nice to be able to get a caller graph of a method to see exactly where it is being called, and who is calling the calling methods, and so on. Again, this could be retrofitted, but it involves a deep parse of the code base, something that non-IDE editors tend not to do.

So if I talk to a Java developer who doesn't use an IDE, they are basically telling me that they:

Do a lot of development without recompiling their code, so there can be a big delay between when they introduce a compile error and when it is resolved.
They generally (unless they are extremely disciplined) avoid altering class, method, and variables to match changes in behavior or implementation.
They are unlikely to have a large, frequently run test suite.
They probably spend a lot of time scrolling through files looking for methods instead of writing code.

When I write it all out that like, my bias certainly feels justified. While those are broad pronouncements about someone's capabilities just because they don't use an IDE, I'd love to hear the counter arguments. Are there developers out there who shun an IDE and feel like they are gaining other advantages that outweigh the above, or that they have similar capabilities that I'm just not aware of?

More Interview Ideas - The "How Did They Build It?" Game

eholman — Mon, 03 Nov 2008 23:59:06 -0500

I'm always looking for ways to improve the interview process, and I hit upon another interesting technique when reflecting on the qualities I am looking for in a good hire. To quickly review, these generally include:

Enthusiasm
Flexibility and Open-mindedness in regards to technical approaches and solutions
General technical skill
An intuitive sense of how things are built

Enthusiasm and Open-mindedness are easy to figure in the interview process. I have my own thoughts on how to learn about general technical skill, but on the last one, I think the easiest thing is just to ask someone to do what I tend to do on a daily basis: look at things are try to guess how they were probably built. I do this sort of reflexively; whenever I see an application, website, or any other piece of technology that does something clever or unusual, I immediately start trying to deconstruct it in the same way a kid might try to figure out how a magic trick works. Here are two quick examples that came to mind:

jacksonpollock.org - The technology is simple enough. It's clearly a flash application (if you could do this with javascript and CSS alone, that would be 100 times the hack). However, what is the algorithm that generates the paint strokes? It seems to be some combination of mouse speed and motion, as if you have a paint can from which paint is leaking at a constant rate and if you move it quickly you produce a small amount of paint per square inch, but if you leave it somewhere it will pile up. There's slightly more too it though, as it must be doing some simple physics modeling or just randomization to produce splatter.

DHTML Lemmings - Speaking of 100 times the hack, how about a full arcade game using only DHTML? I haven't really dug down into the code at all, but I can guess at how it is implemented although there are probably dozens of good ways. I would assume that the screen is composed of images rendered into divs or other containers, just like any tile-based game would be. The sprites are probably just images that are moved or altered by javascript by the game logic, and animation is a well-known javascript technique.

There are both small, fun applications that don't have any serious tricks that need to be unraveled, but the exercise is still useful. A lot of what I and others like me do really boils down to reverse engineering. Reverse engineering not in the DMCA violating sense, but in the sense of being given an application and a huge mass of code and needing to quickly understand the code by guessing at how the application is probably built and then looking for evidence that you were right.

Asking a potential hire to demonstrate their skill in this area seems very valuable, geared toward their area of expertise. A good web designer should be able to tell you how a webpage was probably constructed without looking at very much of the HTML and CSS. A good web application developer should be able to guess at the database structure, the types of objects in the system, and the types of actions being taken without looking at very much of the code. If the candidate makes wildly improbable guesses, or simply refuses to play the game, I would look elsewhere.

Five Useful Android Apps

eholman — Sun, 02 Nov 2008 21:19:33 -0500

There are a few reviews of android apps post G1 launch, so I thought I would add my thoughts on my five favorites that you may not have seen elsewhere.

ConnectBot

What's ConnectBot? Only a full-featured terminal emulation and ssh client that runs on your phone, that's all. As a consultant and small business owner who hates to ever be out of touch with servers, this is the ultimate security blanket for me. It looks and feels just like a command-line ssh client on Linux so there is almost no learning curve. Not something a typical phone consumer is probably looking for, but a critical app for me.

K9Mail

K9Mail is a fork of the build-in Android email code base. While it currently only provides a minimal set of improvements (Delete actually deleting mail on the IMAP server, allowing keyboard shortcuts for some actions), I am hopeful that it will be a springboard for some slick email features. It also shows the power of an open platform, so the choice is partially philosophical.

Twidroid

I am not a huge Twitter user (yet?) but this simple, attractive Twitter client is a great improvement over accessing the website via the Browser app.

Coloroid

Maybe this is a game that is well-known to others on other platforms, but I have not seen it before. The premise is simple: you start from a single block of homogeneous color and select a sequence of color changes. Whatever color you choose will cause your current captured set to expand into any adjacent color block that matches your selected color. Your goal is to keep the number of changes beneath some maximum while attempting to capture every color block on the board. It sounds complicated, but it is really not. Through judicious selection of color sequences, you can grab large sets of blocks at one time and keep the number of color switches to a minimums. It's the perfect thing to kill 5 or 10 minutes with that can be picked up and put down without worrying about timers counting down, remember where you were in the game, etc.

pTerminal

pTerminal is a shell for the Android OS. I haven't played around with it much, but it is useful to check out what services are running, and to run some simple commands such as ping, which can be critically useful when needed. There are apparently some good lists of commands out there that I should track down to really get the most value out of it.

Are you a G1 user with a favorite app? Let me know in the comments.

Will lack of background processes be the iPhone's undoing?

eholman — Sun, 02 Nov 2008 00:57:44 -0400

I ordered a T-mobile G1 prerelease and was one of the lucky ones who received theirs on October 21st. I've been playing around with, and installing many different applications such as Locale, which allows you to set system configuration based on your physical location, for instance if you want your phone to go on vibrate when you get to work or after 10PM. This application relies on the ability to run in the background, monitoring your current location and then taking actions as appropriate. Because of that requirement, it could never be ported to the iPhone, which does not allow background processes. It got me thinking about whether this restriction will prevent any great applications from being developed for the iPhone, and relegate it to a high-quality consumer device that is unfit for enterprise use in the same way that macs once developed that stigma.

[On a side note: remember that up until Mac OS X, the Macintosh had a non-preemptive process scheduler, meaning that the foreground process got basically all of the CPU and could more-or-less refuse to cede it; this was great for low-latency applications like digital music, but annoying to horrendous for day-to-day computing. The parallels with the current iPhone decisions are striking.]

Here is the problem in a nutshell: the refusal to allow background processes imposes an application development philosophy that is incompatible with building something of depth and sophistication. An application must be designed to focus solely on interactions with the user and to do no offline, out-of-band processing. That works fine for web browser, email, and other interactive applications, but you only have to Google "iPhone instant messenger" to see the problem. IM is something that you naturally start and stop, and that needs to sit in the background waiting for more information. There are IM applications for the iPhone, but you can't do anything else while you use one.

As an application developer, this is a deal-breaker for me. I literally cannot conceive of a useful application that doesn't have some sort of background processing capability, and while there are certainly many well-designed and useful applications available through the app store, the current software selection has been widely criticized for its shallowness. I think that those two things go hand-and-hand. The foreground-only restriction encourages the development of simple tools with flashy user interfaces and games, both of which are in abundance for the iPhone. It prevents the development of applications that passively observe context, process information asynchronously, take actions on the users behalf, and push information to them at appropriate times, all key pieces of software that I and many others like to develop, and find personally valuable.

Apple has argued that these background processes consume an inordinate amount of battery life, although that seems like an odd argument given that it would be easy enough for a user to turn on and off as needed. To address the issue, Apple stated a deadline of early October for integration of a new push-notification feature into the SDK and the OS that would allow users to send notifications to Apple servers, which would be queried by individual iPhones. This attempts to get the best of both worlds in that applications can send a message to an external server to do processing in the background, and then push notifications to the user so that they can switch back to that application to get the new information while still preventing background processes from chewing up battery. This has two problems in my mind, the fact that they missed the deadline and even this level of functionality is still unavailable notwithstanding:

1) Application developers must now structure all their applications as client-server, where the iPhone is just a pretty interface and all processing is done on application servers. For instance, (assuming I am understanding how push notifications are supposed to work) for an IM solution, the server software would have to proxy for the user to check for incoming messages and push notifications to the apple servers for that user, which then show up on the phone. Since IM clients currently don't work that way, except for web-based tools like Meebo who are well-positioned for this kind of thing, they would have to be totally rewritten to take advantage of this capability, and the application developers would need to stand up a bunch of servers to do the proxying.

2) It still doesn't let an application developer monitor the phone's context and take action. Locale would still not be possible since when it was not in the foreground, it wouldn't have access to the user's state. iPhone-style Push notifications would be just as useless as the current setup for these types of applications.

There is no doubting that the iPhone was disruptive to the wireless industry, is a marvel of engineering, and has sold a zillion units to satisfied customers. I wonder though if it will all be undermined by this simple problem, or even if it is a continued consumer success, it will always feel like a toy compared to where a phone like the G1 may evolve to. Certainly it will be a long time before we know the answer, but I find one thing very telling: whenever I describe Locale to an iPhone user, I always get the same wistful and slightly concerned look.