Pragmatic Software Development

Trends of 2011

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Fri, 30 Dec 2011 16:13:00 +0000

December 30 feels like a good date to reflect on the trends of the past year. Looking back to major trends of 2010, I realize that 2011 was more of an evolution and consolidation of technologies, rather than introduction of new ideas.

Mobile apps are rapidly becoming the most popular type software. Mobile space now resembles Web of late 90’s when everybody realized that they need a web-site. Furthermore, like e-commerce of late 90’s, mobile-specific business models are also beginning to emerge. At the same time there are clear signs of market consolidation with iOS and Android being clear winners, and Blackberry, WebOS, Windows phone and others being clear losers.

While Tablet can be seen as just another mobile device, it certainly has unique characteristics and distinct usage patterns. 2011 saw great explosion in tablets’ popularity and adoption. Initially thought of as keyboard-less laptop, the tables are now perceived as completely different type of consumer device. Table market-space is still in its infancy and the jury is still out on what future tablets will look like – a computer, a toy, a media consumption device, etc.

Cloud computing begins to lose its hype because it’s becoming a standard part of software development and operation. This technology is entering its maturity phase. By and large, Amazon remains the dominant cloud provider. A noticeable trend of 2011 is movement from IaaS to Paas, where infrastructure cloud providers begin to offer more hosted services like ElastiCache, RDS for Oracle, Beanstalk, etc.

NoSQL space also seems to enter a consolidation phase which followed Darwinian explosion of database types and approaches. At the moment, it seems that MongoDB and Cassandra capture most of attention. As with any new technology, people begin to realize its sweet spots and its limitations. Also, a better understanding of the umbrella “NoSQL” term leads to separation to document-based (Mongo), key/value based (Reddis), or table-based (Cassandra) solutions.

Social Platforms are now considered an important part of any consumer-oriented products. A new entrant into the social space is Google+ that competes head-to-head with Facebook. Time will tell whether it has a future or whether it ends up like other failed social products from Google. The interest in Social Platforms is further driven by a slew of high profile IPOs (LinkedIn, Groupon, Zynga).

While future predictions are futile, I have a strong feeling that 2012 will also be year of elaboration of existing trends rather than introduction of something radically different, but only time will tell.

Happy New Year!

Productive Time

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Mon, 28 Nov 2011 23:15:00 +0000

As trivial as it seems, I have discovered that leveraging periods of high-energy during the day for most strategically important tasks greatly enhances effectiveness and performance without consuming additional resources of time or effort.

Everybody has their own high-times and low-times during the day. Some people consider themselves morning persons, some people consider themselves evening/night persons. These categorizations usually serve for identifying periods of high-energy and low-energy. Naturally, people like the high points and not so much the low points.

What I have noticed, however, is that there is a certain disconnect between daily biological clock and scheduling of important tasks. Consider this example: I know many “morning” persons that start their day with coffee, emails and their favorite news. The most productive time of their day (7am-10am) is therefore used for routine activities that do not require too much energy or concentration. If, instead, this most productive time of the day was leveraged for strategic tasks such as design, analysis, planning, then the same amount of time would give a quantum leap in terms of productivity.

I've tried it myself. Now when I come to the office, I don’t process emails, I don’t look at the latest tech news, I don’t finish off leftovers from yesterday. I intensely focus on high-priority items. The emails can get processed pretty much any time during the day, so can the news be read later. After just few weeks I have realized that I made more progress on the important tasks during these few weeks than I did in many months.

Additional benefit of using morning time for “thinking”, is that you are getting less distracted by others. Some people are not in the office yet, while others are busy reading the news:)

I don’t want to leave an impression that I necessarily advocate using morning time for strategic activities. If you’re an evening person, then by all means – leverage evenings. My point is that you need to identify your high-times and your high-importance tasks and make sure that they are in agreement.

P.S. if you're reading this in the morning, you might use this advice to improve your productivity:)

Golden Mean

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Mon, 31 Oct 2011 15:04:00 +0000

Even if you’re not into Aristotelian philosophy, you must have heard about the doctrine of the Golden Mean. It advocates finding a middle solution between two opposite extremes.

Yet, it is surprising how often in Software Engineering people forget about the Golden Mean and violently advocate an obviously extreme solution or view.

Here are few examples:

· 100% TDD vs No TDD

· BDUF vs No Design

· Strong typing vs Weak Typing

· Management Theory X vs Theory Y

In each of those pairs of opposites, each option has some merits. Yet, taking it to the extreme and asserting its universality, turns each of them into an absurd. The goal of a successful architecture or design effort should be finding the right tradeoff between competing forces and therefore finding the Golden Mean.

One important note is that the Golden Mean in each of those pairs is heavily dependent on the context and is not something absolute and unchanging. That’s why it’s so funny to read people blindly advocating a particular option with nearly complete disregard to circumstances and context.

Using Maven Profiles for Managing Dependencies in SOA Application

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Fri, 23 Sep 2011 13:09:00 +0000

Recently we have enhanced our Continuous Deployment process by creating a bootstrap application that initiates DBs with tables and initial data. We use JPA/Hibernate to auto-create the tables from model objects. Since our application is composed from numerous services, each service carries its own part of the model, and there is a shared model used by all services. The bootstrap module depends on all the services through Maven's pom.xml, and populates the DB with tables from all dependent services.

The problem is that different clients use different combinations of services. Obviously, we don’t want to deploy services not used by a particular client, nor do we want to create tables needed by those unused services. After some research, we settled on Maven Profiles that allow activating and deactivating different dependencies within the POM based on active profile.

Consider the following example:
The app contains three services (I wish only 3:)): S1, S2, S3, and a shared model M. Client A needs M, S1, S2, while Client B needs M, S2, S3 components. Originally, we would make bootstrap program depend on all 4 components and create all 4 database objects. So, in order not to pollute Client A’s environment with S3's DB objects that are not used, we introduced two Maven Profiles.

Default dependency (profile): M, S2
Profile-Client-A: S1 dependency
Profile-Client-B: S3 dependency

<id>Profile-Client-A</id>

                  </dependency>
        </dependencies>
      </profile>

<id>Profile-Client-B</id>

            </dependency>
        </dependencies>
      </profile>
<profiles>

      </dependency>
      <dependency>
            <artifactId>M</artifactId>
      </dependency>
</dependencies>

Each client has its own eclipse workspace that contains only components relevant to that client. In each workspace, the same bootstrap project is configured with a corresponding profile as active. The same is done in corresponding Hudson build jobs: build for Client A uses maven –P Profile-Client-A setting which “pulls in” only relevant dependencies.

By leveraging Maven Profiles we’re now able to remove all irrelevant dependencies from each of our clients. The workspaces look much cleaner, especially since in reality we have much more than just 3 services…

Caffeinated Coder – Short Circuiting Your Body

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Tue, 09 Aug 2011 12:59:00 +0000

In 16 years of my professional career, if there is one thing that’s common between all companies where I worked, then it’s certainly the high amounts of coffee consumed by programmers and other engineers. Understandably, developers need frequent energy boosts to stay focused and productive in this demanding and challenging line of work.

I was doing the same thing – drinking 3-4 cups of coffee per day, and was considered “low” coffee drinker compared to other coffee-addicts that consumed 6+ cups a day. Then, with the age, I developed IBS, which, in my case, prevented me from drinking coffee. After stopping drinking coffee, I felt extremely low energy and had serious difficulties concentrating and working with any degree of intensity. It was much like any other addiction: when you stop it, it hurts.

Then I discovered the mid-day exercise. The building where I work has an excellent gym, and I decided to try it at lunchtime. I wanted to go there to work on my back pain, but, to my greatest surprise, I discovered that doing intense exercise during the day, gave me more energy than any amounts of coffee that I used to consume. After coming back from gym, I’m full of energy, don’t need any coffee or tea, can concentrate, and generally feel significantly better.

It almost feels like by consuming coffee we short-circuit our brain into feeling energy. Of course, it’s unscientific and speculative, but, based on my own anecdotal evidence, I can recommend at least to try substituting some of the caffeine with a more healthy alternatives like mid-day exercise. The same, by the way, applies to using caffeine as a substitute to decent sleep. Yes, it’s possible to get a temporary boost, but I can’t imagine that in the long run this is sustainable or healthy.

To shard, or not to shard: that is the question

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Fri, 15 Jul 2011 13:32:00 +0000

When dealing with very high volumes of data, one usually needs to decide how the system is going to scale when the data grows beyond a reasonable size. In my current project we’re working with tables that reach 1B records. I’m talking about MySQL 5.5.

After initial load-testing and profiling we had our first “oh, shit” moment. When tables are in hundreds of millions of records, regular “laws of physics” don’t always apply. So, we desperately started looking at ways to alleviate these problems.

Clearly, the most scalable method is sharding. Sharding is usually implemented at the application level in such a way that the application knows on which node a particular piece of data resides. A common example is placing users 1-1M on shard 1, users 1M-2M on shard 2, etc. This method allows for theoretically unlimited amounts of data.

Now, the bad part. Implementing sharding is pretty complex and time consuming. Most importantly, sharding forces breaking up the data by certain criteria (like users_id). All queries that specify the user are OK, because we know where to go. But, queries that don’t specify user IDs must be executed on ALL shards. That is a major point of pain. Of course, you can come up with a smart parallel execution strategy, but then you’re effectively entering a realm of programming of distributed databases.

There are some ready alternatives: ScaleDB and Gizzard are examples of NewSQL, and are layers that reside between the application and mulitiple MySQL nodes, and know where to execute the queries. Then, of course, there is a plethora of NoSQL solutions that are capable of distributing the data: Cassandra, MongoDB, etc. In our case, increased complexity associated with introducing new systems into the project, would outweigh benefits of these systems.

So, what’s a desperate developer to do in such a case? Well, in some cases it’s possible to fall back on time-tested ways of reducing table size by means of archival. Archival does not mean that the archived data is not accessible. It might just be accessed from another table. Also, sometimes it helps to partition the table on MySQL level. While it’s far short of the benefits that complete sharding would yield, it does help with performance of queries with partitioned column.

The conclusion is that while sharding is very powerful and scalable technique for dealing with large data volumes, it involves a lot of complexity and side-effects. A careful examination of alternatives is in order. Be it NewSQL, NoSQL, or plain OldSQL (archival, DB-level partitioning), it’s worthwhile to carefully analyze tradeoffs of each solution.

Agile Change and Rework Waste

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Tue, 14 Jun 2011 13:00:00 +0000

One the main benefits of Agile methodologies is adaptability to change. Changes occur in all projects, Agile or not, but Agile projects are better equipped to handle the changes through short iterations and ongoing adaptation to new requirements. The customer does not need to specify everything upfront. As the system is being developed, he can refine the system with the knowledge that was obtained from previous iterations.

Now, there is a difference between refining the product, re-prioritizing certain features, redefining the unimplemented features on the one hand, and discarding an implementation of already developed features on the other hand. Discarding often occurs because “now i see it differently” or because “that’s not what I wanted”. Granted, as people discover more about the product, their needs and requirements change. However, the proper way to address it is to implement the minimal and agreed agreed functionality first, and then “grow” it according to evolving understanding of the system.

Unfortunately, people often confuse iterative and incremental development. Iterative development means that a team develops bare bones of a feature and then grows it. Incremental development means that one whole feature is implemented after another whole feature. In such case, if the implementation is wrong, then most of the effort is wasted, and the feature must be reworked. This is not the Agile change management – this is waste, and it should be rigorously eliminated (think Lean).

An alternative to wasting effort on implementation of misunderstood, or incorrectly specified functionality, is not to request the customers to sign the requirements in blood. It is to structure the implementation in such a way that least debatable or more clearly understood and agreed upon functionality is implemented first. If there are areas that are likely to undergo significant amount of change, it's worthwhile to prototype them before committing to a full-fledged implementation. Changing prototypes is a lot cheaper than changing actual implementation.

For example, say you need to implement user management functionality in a back office part of your application. You could try to specify all fields that are required, you could make guesses, or you could torture your customer to divulge the secrets of user’s information. Or, alternatively, you could just implement a very basic set of properties (first/last name, address, birth date) in initial iteration, and later “grow” the user data as requested by the customer.

Furthermore, it is in customer’s and provider’s best interests to think the requirements through before starting their implementation. Incomplete and incorrect specs cannot be miraculously cured by Agile methodologies. What’s incorrectly specified will be incorrectly implemented. So spend the time analyzing and planning what and how things will be implemented during the upcoming iteration or release.

Agile Business Discovery

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Tue, 03 May 2011 13:36:00 +0000

One of the most important advantages of Agile methodologies is the rapid feedback loop. This feedback loop is intended for refining and optimizing the development process. It also helps to ensure that final product matches customer’s expectations and solves the actual business problem.

In all this, it is assumed that the business problem is known. But what if it’s not? What if the business model itself is being discovered? Shouldn’t we apply the same Agile principles in discovering the business model?

You might ask: “What kind of business is that, if it does not understand its own business model?” Many examples come to mind: start-ups, big organizations trying to expand, etc. In all such cases, the business model or at least the required feature-set needs to be discovered through a rapid feedback loop with potential customers.

Describing Innovation At Google, Patrick Copeland put it extremely concise: “Before building it right, build the right IT”. He recommends what he calls prEtotyping (as opposed to prOtotyping) to get a real-life customer feedback about a potentially great business idea. Whereas prototyping focuses on proving that the product can be build and refining its technical structure, prEtotyping is aimed at proving the business model itself, and helps screening ideas before any real investments are made into the development.

Many people believe that their ideas or business models worth millions. However, from the experience we know that not every company becomes the next Google, Facebook, Tweeter, etc. The only way to get any confidence in a product is to elicit early feedback from potential customers. Essentially, it means applying empirical, scientific technique to discovery of viable business ideas. As with any empirical studies, the method requires actual data, preferably in a numeric form. This data can include page hits, returning customers, etc. Often, even the most primitive pseudo-implementation (CRUD?) is enough to gather such data. Sometimes, just a study of customer preferences and simulated sessions allow vetting of ideas without any implementation or prototypes.

In conclusion, if you’re trying to implement a new business idea or to augment an existing one, use the same Agile feedback methods to validate or screen out the idea before investing in it.

Economics of Amazon Outage

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Thu, 28 Apr 2011 13:41:00 +0000

As if the world needs another rant about Amazon EC2 outage, I’m going to risk adding my 2 cents to the debate.

On the one hand you have apocalyptic visions on the cloud future focusing on inherent unreliability of public clouds. On the other hand you have people explaining, reasonably enough, that there is nothing new about hardware failures, and talking about necessity of preparation for it by means of redundancy and other fail-safe methods.

I, obviously, see logic in the latter view, and understand that “outage happens:)”, and you need to prepare for it. However, often the missing ingredient in this line of reasoning is the cost of the preparation. In the end, everything boils down to economics. There is a very good description of estimating the cost of the preparation in this article: “The most straightforward approach is to estimate the cost of a failure and then multiply by the probability it will occur.”

Now, if the probability of having such failure is higher in the cloud than in private data-center, and cost of insuring against such failure is also higher, then overall costs, compared to private data center, rise significantly. Many cloud deployments do not include the cost of redundant infrastructure in their economic models. Adding that cost, which often doubles the numbers, can make it more difficult to justify the cloud strategy.

In addition to the cost of insurance against widespread outages, there is the cost of bullet-proofing your software against local “mishaps”. People working with the clouds surely experienced disappearing instances, diminishing processing resources resulting from heavy processing loads of other tenants, etc. Dealing with all this takes time and effort, i.e. money, to address. That money also goes into financial models used to estimate cloud worthiness.

Of course, elasticity of the cloud and its on-demand availability, have virtually no replacement, and will therefore be always needed by small or growing businesses. But for stable and predictable IT needs, it remains to be seen how the economics play out and what makes more sense from the financial perspective.

How much code coverage do you really need?

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Thu, 14 Apr 2011 13:27:00 +0000

This post was prompted by reading a number of categorical tweets from @unclebobmartin. In case you’re not familiar with Uncle Bob – he’s one of the most prominent Software Industry experts, author of Clean Code, signatory to Agile Manifesto. In late nineties he did a profound work on documenting best OO practices (SRP, open/closed, interface segregation, etc). So when he speaks – it’s worth at least consideration.

He takes a maximalistic approach to TDD and unit testing in general. It can be clearly seen from his tweets:

“Two things. Repeatability and cost. Manual tests are horrifically expensive compared to automated tests.”

“Manual tests aren't tests; they are trials. And since there are humans involved, they are suspect.”

“What you are telling me is that I should be open to the possibility that some code shouldn't be tested. Hmmm..”
“100% code coverage isn't an achievement, it's a minimum requirement. If you write a line of code, you'd better test it.”

He goes on to compare software testing with other mundane but critical activities that are considered mandatory in other fields:

“A surgeon on the battlefield may not have time to wash thoroughly, but the risk of death and cost of treatment will be high.”

“Do accountants cover only 80% of their spreadsheets with double entry bookkeeping?”

“How many times have you seen major outages that were due to some silly code that some silly programmer thought wasn't worth testing?”

While all these points certainly have merits, they show only one side of the picture. The reality is that not all applications require such a meticulous testing. Not all application are of the same importance as surgeries on a battlefield or accounting of big $$$. (not to mention the “creative” accounting employed in many cases:).

An even more important point is that thorough code coverage does not guarantee absence of bugs. Even Uncle Bob admits that:

“Tests cannot prove the absence of bugs. But tests can prove that code behaves as expected.”

This is obvious considering that same misconceptions and logical mistakes that were put in the code by the developer, are not likely to be discovered by the same developer when testing his own code.

In the end it all boils down to ROI and pragmatism. Some apps need more testing than others. Some modules need more testing than others. Some bugs need more fixing than others. There will always be a judgment call about whether additional time and money spent on automated testing and coverage are justified or are just a premature optimization.

Layers Are Not Tiers

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Mon, 21 Mar 2011 15:44:00 +0000

I know what you're thinking: well duh! And yet often the distinction gets lost somewhere between design of the software and deployment topology.

Layering is an important technique that allows to manage complexity, increase cohesion of the components and to isolate inevitable changes from impacting rest of the system.

Tiers usually refer to the deployment structure of the systems. Tiers are separated from each other for various operational and security reasons. My point is that multiple layers can often be deployed into the same application tier and should not be equated with them.

A lot of the misconception, at least in Java world, comes from the traditional JEE stack: presentation layer, business layer, DB layer. Those are valid distinctions that should exist at the code level, but not necessarily in deployment. For example, it often makes sense to deploy presentation layer and business layer together for the sake of performance.

The standard deployment topology looks something like this:

But sometimes there is not need to separate Presentation Layer from Business Layer into different deployment tiers. Combining them allows us to avoid overhead of serializing and transporting data between the tiers. So the resulting topology would look like this:

Furthermore, sometimes it even makes sense to access DB directly from presentation layer, entirely skipping the business layer for fast reads of DB data. This does not mean having SQL code mixed in the pages, it just means that data extraction does not need to be done in fat EJB-like components. It can come straight from DB views and be wrapped by a light data access layer used by the MVC controllers. This might sound like a heresy but do we really need this useless layer of empty services that just pass data from DB to presentation?

Of course “writes” need to be processed by the business components and have all business rules applied before landing in DB. Updates are usually done on a per-object level and benefit from rich domain model. But “reads” often just need highly optimized and joined data from DB, especially if it’s a read-only data.

For all these reasons it’s important to distinguish layers from tiers. Multiple layers can be combined into the same deployment tier and sometimes even have some of the layers skipped.

Sonar: feel the pulse of your code

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Tue, 08 Feb 2011 14:14:00 +0000

Regular code reviews are like physical exercise: everybody agrees how important it is but only few have time and energy for it. Yet, code reviews are one of the best ways to reduce defects and associated waste in debugging time, not to mention the overall product reliability.

There is a distinction between low-level tedious review of syntax of programs and review of their business functions. It is the low-level syntax and structure that are the ones that take most time and are also least interesting.

Reviews for technical components of the program have been automated for quite some time. Tools like PMD, FindBugs and Checkstyle are very popular Eclipse plugins and represent the first line of defense against simple but dangerous bugs. The problem with those tools is that they are intended for separate individuals and their reporting is done at a relatively low-level of detail.

Enter Sonar (see screenshots below).

Sonar is a web-based tool that analyzes your code using countless different metrics. It checks violations of coding rules similar to PMD/FindBugs/CheckStyle. It also presents an extremely useful dashboard of aggregated stats such as LOC, classes, duplications %. It even keeps track of how those metrics were changing in time. Best of all it’s easily integratable with Maven and Hudson and works seamlessly behind the scenes on each build.

Having high-level overview of code IMO is extremely important especially when the code base is large. Stats like coding-rules violations, test-coverage, complexity-per-method or per-class, cohesion are conveniently presented in numerical form. And these are just few of dozens (if not more) relevant stats.

If you want to see thorniest places in the code – go to Hotspots view. It ranks parts of your code using various metrics of “badness”. These include most violated rules, most untested classes, most complex classes, etc. Hotspots view is linked with Violation Drilldown views where you can expand areas of interest. In addition to that, Design view can be used to find cyclical dependencies between packages (a big no-no in proper OO design).

Ability to observe how these metrics change in time is very important because it allows to trace meaningful degradation in code quality down to individual code modification (i.e. when someone submits a bunch of spaghetti).

Overall Sonar is a great tool that exposes previously hidden data about code and presents it in a very convenient, interactive and intuitive format.

If you want to keep you code clean (no pun intended on Uncle Bob’s book) – Sonar is your tool.

Dashboard

Hotspots

Timeline

Managing Interruptions – Part Deux

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Mon, 17 Jan 2011 14:20:00 +0000

In the previous installment of Managing Interruptions I introduced to strategies that worked for me and allowed increasing focus and attention span when working in a highly multitasking and frequently interrupted environment while trying to do focused development at the same time.

The strategies are:

Minimize interruptions: for example discourage excessive use of instant messaging.
Combine interruptions and handle them in batches: instead of handling an interruption when it occurs, queue it and handle it together with other interruptions during allocated time.

Recently I successfully introduced third strategy that can be viewed as an extension of the second strategy:

Handle interruptions in first half of the day, while trying to avoid and postpone most of them in the second half of the day. (Of course which part of the day and it’s duration is entirely up to you)

The reason why this works for me is that I don’t have to switch between interruptions and focused work at all. While second strategy allows “islands” of focused work, those islands are relatively small. In order to do real programming you need at least a good few hours in order to do something productive. By allocating second half of the day for programming, I was able to significantly improve my concentration and amount of produced code.

The approach also makes it easier to handle interruptions in the first half of the day because I’m entirely allocated to them and my mind is in “interruption handling mode: which is very different from “focus and concentration mode". Also, I’m not worrying that yet another interruption is preventing me from programming because I’m not doing any programming at that time. Taken together it significantly minimizes context switching between interruptions and focused work.

Now to the technicalities. Even if the idea itself sounds reasonable, it might be hard to implement in reality. To achieve it I did the following:

I communicated to my subordinates and my boss that I will be trying to handle most of external requests and interruptions during first half of the day, while handling only critical requests during second half.
When an interruption arrives during second half, unless it’s absolutely critical, I flag it for handling the next day.
I put a printed sign “Janitor is out! Janitor reception hours are 8am-1pm”. (I think architect/team lead’s work has a lot in common with the janitor:))

After employing this strategy for about a month I’m positively surprised by the results. I am able to efficiently handle all requests and at the same time be more productive member of the team.

Trends of 2010

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Thu, 23 Dec 2010 14:13:00 +0000

The year is almost over and it’s hard to refrain from nostalgic retrospective of major trends than emerged during the past 12 month. None of these trends started in 2010 but each of them seems to reach an important inflection point.

Clouds are now pretty much at the pinnacle of their hype. Everybody is doing something cloudy now. There is a good reason for it – they do seem to provide a real value in certain situations. Their sweet spot is powering development and testing environments, offloading temporary load spikes and low-investment test-bed for startup ideas. The key to their success is their on-demand elasticity and provisioning. So far it seems that Amazon is by far the clear leader in cloud providers. Other PaaS and SaaS providers like Google App Engine and SalesForce are much more nichy and restrictive.

There are still big unanswered questions with the clouds which include security, regulatory compliance, vendor lock-in, etc. Probably next few years will see rapid evolution of cloud computing that would address some of those problems.

Mobile Development is capturing huge attention and interest. Recent success of iPhone Apps and Smart Mobile devices in general look like an inflection point of a broader shift in computation devices away from the traditional desktops. It is true that many Smart Phones today pack in more processing power than desktop computers 10-15 years ago. If this trend continues then new tools and methods will need to be developed and adjusted for mobile applications that are still in their infancy.

NoSQL databases are becoming ever more widespread. Their ability to scale for today’s information and communication needs often outweighs compromises that are made when moving away from traditional RDBMS. I wrote a short blog about various popular NoSQL options. An interesting and very recent trend is support that major frameworks like Spring begin to provide for NoSQL DBs. Grails GORM now carries support for a number of NoSQL databases.

While many people read NoSQL as “NO SQL”, a better approach would probably be “Not Only SQL” because relational database are not going to go away any time soon and still provide very valuable services.

Functional Programming is getting more and more publicity. It seems that Functional Renaissance started from necessity to write programs targeted at multi-core computers. Functional Programming paradigms lend to better distribution of computations among the CPU “heads”. A proliferation of functional languages is now emerging: from grand-daddy of all functional languages Lisp, to new kids on the block - Scala, Closure, Erlang, Haskell and F#. Some languages such as Scala attempt to bridge Object Oriented and Functional programming by allowing the developer to choose the most suitable tools within the language.

There are many more interesting developments in such areas as Social programming and Agile methodologies but they are somewhat less pronounced than the trends that I have mentioned.

Certainly year 2010 feels like an important stage in development of computing and software in many important areas. Major paradigm shifts are occurring in the whole information processing universe. It will be interesting to see how these trends evolve and what new trends are in store for 2011 and beyond.

Vertical silos vs. Horizontal affinity

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Wed, 08 Dec 2010 15:26:00 +0000

It is safe to say that Agile methodologies are considered mainstream this day and age. One of the main precepts of the Agile methodologies is using of cross-functional teams and delivering applications in vertical slices for rapid feedback and early exercise of integration.

There can be no debate that the feedback loop and early integration is critical for the success of the project. However, what happens in the systems where infrastructure is significant, perhaps biggest, part of the system? It is hard to create slices of the infrastructure based on vertical slices of functionality being developed for the current iteration/sprint/user-story.

Such a low-level and application specific infrastructure has much higher affinity within itself than with any of the vertical silos where it might be used. It makes sense therefore to develop this infrastructure separately from the vertical slices and leverage it later with the usual Agile iterations.

An important point in this is that by infrastructure I don’t mean DAO layer or service layer. In typical web-apps such layers are nothing but an amalgamation of parts of vertical pieces of functionality. By infrastructure here I mean something like low-level networking, threading models and abstractions, core messaging abstractions, or low-level application-specific services.

Sometimes parts of an infrastructure can be harvested through an evolutionary architecture and emergent design. But in any big and especially non-plain web-systems there is a significant amount of upfront infrastructure that needs to be in place in order to start developing the “end-user” functionality.

Focusing on a single level of abstraction (SLAP), in this case the low-level infrastructural services increases productivity and helps creating a consistent system.

It is therefore important to carefully examine which parts of the system exhibit highest levels of affinity and develop them together as a fundamental block upon which rest of the system will be built later.

Is Java suitable for today’s web-development?

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Fri, 03 Dec 2010 13:44:00 +0000

I know it’s a provocative subject and I might get flamed for it but I have to share my thoughts on the suitability of Java for today’s web development.

As always, the applications are developed to solve business or other human problems. Therefore the closer the application implementation is to the domain of the problem – the faster and better the application development is. If you’re programming bank application you want to operate with bank accounts, not object arrays. If you’re developing gaming application you want to develop players profiles and game rules, not low-level 3d rendering.

If you’re developing a web-application, for example facebook app, you want to deal with RESTful Facebook Graph API, with social features, and at least with somewhat interactive pages. What you don’t want to do is to deal with low-level plumbing like mechanics of reading the URL content. Such details are nothing but an accidencial complexity within the system.

A simple example:

Let’s say you need to read content of RESTful URL something like https://graph.facebook.com/me. All you really want is to get a string with the URL's content.

In Groovy it would look like this: “https://graph.facebook.com/me”.toURL().getText()

After spending few hours trying to find something similar in Java I have reached a conclusion that the best I can do is either:

Use apache HttpClient and deal with hierarchy of its objects. This is a bit more than few lines of code.
Try to find another obscure library.
Write custom code like this:

public static String getText(String url) throws Exception {
        URL website = new URL(url);
        URLConnection connection = website.openConnection();
        BufferedReader in = new BufferedReader(
                                new InputStreamReader(
                                    connection.getInputStream()));

        StringBuilder response = new StringBuilder();
        String inputLine;
        while ((inputLine = in.readLine()) != null) 
            response.append(inputLine);
        in.close();
        return response.toString();
    }

Now, of course, I can write it and encapsulate it in some helper utility object etc.

But the problem is that this code has nothing to do with the problem that I’m trying to address in my application. It represents a much lower level of abstraction. At yet I’m forced to resort to spending time on researching, implementing, testing, maintaining such a basic functionality.

For comparison, Ruby/Groovy/Python developer does not need to know anything about BufferedReaders, InputStreams, using StringBuilder for loops etc. All he really needs to know is the URL, and the string where to put the content.

It might seem that I’m blowing this issue out of proportion but I think it exemplifies a much deeper problem. Java was invented and evolved in a different era. Today’s web applications use different and higher abstractions. This is normal cycle of development of computer languages. Each new generation abstracts low-level details of the previous generation allowing developers to focus more on their problem domain. 15 years ago java simplified development by allowing programmers not to care about pointer arithmetic, destructors and other C++ plumbing.

I know that Java is a generic language and it will continue to be used for a very long time. But guess what: C++, and shockingly even COBOL are still in use, but in only very narrow domains (embedded or low-level programming for C++ and ancient legacy systems for COBOL).

I suspect that Java is going in the same direction. Today I would be hard pressed to start development of a decent web-application in pure Java.

By the way – I had exact same argument about C++ and Java about 10 years ago with my former colleagues.

Groovy and Grails for prototyping

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Mon, 22 Nov 2010 13:50:00 +0000

There is much debate about dynamic typing vs. static typing. There is much debate about new languages (Ruby/Groovy/Erlang/Scala) vs. old and generic languages (Java, C++).

Some people hold very strong opinions on these subjects while others are more open to different approaches.

Surprisingly, I found relatively little disagreement on the usefulness of dynamic languages such as Ruby or Groovy for prototyping. Ease of use and speed of development are the most important benefits of these languages. Their potential weaknesses such as performance and maintainability of the code are not that important in prototypes. (more on these potential weaknesses below).

These rapid development tools are perfect for flashing out the vision and model of the product without spending too much time on the underlying technological “plumbing”. Business people are impressed to see a mocked prototype of the product in few days after the initial discussion. Only by seeing it they can come up with a useful feedback and refine the product in their minds. Implementing these changes is as easy as doing the initial prototyping. This rapid feedback and adjustment loop is not only agile but also very cost-efficient because only minimal amount of effort and time is spent.

Grails really shines in creation of domain models because often a model can be described through a set of CRUD screens and their relationships. This representation is a lot friendlier than dry and “tech-y” entity diagrams. It is much more understandable to business folks.

As for possible problems that arise when developing with Groovy and Grails – I believe they can be solved and the cost of solving them is far outweighed by benefits of speed and agility. But I am not trying to sell or pitch yet another technology. I believe that it is extremely well suited for certain types of applications while it’s probably not the best choice for others. Like anything else – this is just another tool in your professional toolbox.

Of course, it all applies not only to Grails but also to Ruby/Rails, Python/Django and other dynamic languages.

SQL now, NoSQL later

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Thu, 11 Nov 2010 14:22:00 +0000

This post is about trade-offs of choosing a NoSQL database vs. a traditional relational database specifically in the context of a new business or a start-up.

In the recent years there is more and more interest in NoSQL databases. One of the main motivations for using such databases is their ability to provide massive scale needed in large applications. Naturally biggest internet sites such as Facebook, Linked-in and the like are heavily using and even developing NoSQL solutions.

As with everything else in life there are always trade-offs. A famous triangle helps to visualize main competing forces when implementing a distributed system. People going for NoSQL solution are usually aware of the trade-offs and adjust their systems accordingly.

Less often mentioned is the lack of NoSQL tooling that has been developed for more traditional relational databases. Such tools include IDE support, code libraries, ORMs, entire CRUD generators and the reporting solutions. These tools significantly expedite development of new software. Just think of Rails' Active Record or Java's Hibernate.

This bring us to the main point: a new, possibly start-up company, is usually more concerned with getting the product out of the door as soon as possible and validating the business model than preparing for “inevitable” Facebook-like success.

Many successful companies tell the same story about iterating between various ideas and models before finally hitting a successful one. In such rapid iteration and evaluation - tools are crucial in providing the necessary productivity and speed. Using ready-made tools for relational databases, ideas can be prototyped, deployed and evaluated much faster than with custom made solutions.

Developers from Facebook and Linked-in emphasize that their initial solutions were the simplest ones possible. Linked-in used to run a relational DB for most of its data. Facebook used to host photos as separate files on a simple NFS. Only when they became successful they invested in scalable NoSQL solutions.

It is not an unreasonable approach therefore to start with the simplest relational DB (mysql?) and rapidly create a working product that can validate the business model. If the model is right, it will generate enough interest and money to evolve the solution with more scalable NoSQL DBs. If the idea is not successful, however, a lot less resources were spend on it and more resources and time can be spent on trying the next idea.

Main objection to this approach is that it requires throwing away a solution and re-creating it with NoSQL. While it’s true to some extent, it is not necessary to throw away all the code. In addition to that, usually after first release, there is enough new knowledge about the system and the business that a significant part of the code will change anyways.

In conclusion: while NoSQL solutions provide huge benefits for massively scalable applications, developing with NoSQL might not be the best option when trying a new business idea. Relational databases provide easier and faster route for experimentation and evolution of a business model.

Some NoSQL alternatives

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Mon, 01 Nov 2010 01:10:00 +0000

From a very superficial view - this is my understanding of some NoSQL alternatives.

CouchDB – stores JS-documents. Light, good replication/synchronization, sometime used for offline storage and mobile devices. Can be used in low-write intensity web-sites. Implemented in Erlang.

MongoDB – stores JS-documents. Fast DB implemented in C++. Can be used in relatively high write-intensity web-sites. Supports multi-master mode. Supports basic sql-like queries. Recently added auto-sharding.

Cassandra – full blown implementation of Amazon’s Dynamo combined with column-based storage. Relatively new product designed for massive scale. Of course, complexity and “raw”-ness are potential downsides. Used by Facebook.

Voldemort. Massively scalabled, replicated, redundant DB with pluggable storage engines. Also based on Amazon’s Dynamo. Provides most sophisticated features and scale at expense of complexity. Used by Linked-in.

Hadoop – can store huge amounts of data that is accessed through map/reduce functions. Frequently used in analytics and data-mining.

Managing interruptions

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Tue, 26 Oct 2010 14:07:00 +0000

There is no debate that interruptions and frequent context switches significantly affect productivity. This is especially true of the programming activities that require high levels of concentration.

On the other hand interruptions are unavoidable fact of modern working environments. Emails, instant messages, phone and old-fashioned knocks on the door are present everywhere. Furthermore, people in managerial positions are interrupted constantly (I read somewhere that statistically it’s something like every 5 minutes). Some people with high frequency of interruptions simply resign from the usual focused effort and work on “interrupt-basis” giving up on any serious technical work such as code development or even code-review (which also requires high level of concentration).

So what’s a tech-lead or low-level manager to do in order to optimize his time while still being responsive to the needs and requests of others?

So far I’ve discovered two ways of dealing with this major problem:

Minimize interruptions where possible (what a revolutionary idea…)
Combine interruptions and handle them in batches

Minimize interruptions where possible.

Sometimes people get “too connected”. They allow other people to constantly come to them with questions, send instant messages, etc. In such cases it’s important to encourage people to interrupt you only when they can’t find an answer by themselves. If they do otherwise, point out the way by which they could handle the problem themselves.

As for the Instant Messaging, I got completely off it (call me crazy). The problem with instant messaging is that it’s way too easy to communicate and interrupt. People don’t even have to leave their comfortable chairs. That tricks people into abusing the instant messaging with unreasonably simple and trivial matters.

If you’re not available on instant messenger, having to walk to your office or write you an email will force people to think twice whether they really need your help. If there are some people that absolutely require your presence on instant messenger (like your boss), create another account and don’t share it with everybody. I tried it and after initial shock people adjusted and even got used to the new mode of communication.

Combine interruptions and handle them in batches

The idea is to create slices of time when you simply ignore the interruptions. This means not responding to emails, telling people to come later, etc. One technique for that is Pomodoro. Basically you are focused for 25 minutes followed by 5-10 minutes used for handle the accumulated interrupts.

Personally, I even shut down the email client (outlook) so that I don’t see any indication of emails or alerts. I know that you can configure it but having it not run at all provides much stronger isolation from interruptions.

I installed a scheduler program that automatically launches the Outlook each half an hour. Then I spend some time dealing with the accumulated emails.

Before I implemented this technique I used to get interrupted with the emails every 3-5 minutes. I was receiving 100-130 emails per day, and these were not just dummy notifications, most of them required real attention. After I implemented this – I deal with the same amount of emails but much more efficiently and with less effort. Most importantly - it does not break my concentration.

Now using combination of these techniques I managed to significantly increase attention span and focus and therefore improve the overall productivity

In Conclusion:

Try to avoid interruptions whenever you can. Sometimes it means changing your habits and habits of people around you.

When you cannot avoid interruptions at least try to combine them and handle them in batches.

Good luck.

Bad Attitudes of Agile

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Tue, 21 Sep 2010 14:42:00 +0000

Article:Bad Attitudes of Agile: "Christopher Goldsbury explores some 'bad attitudes' of Agility - assertions about management, documentation, testing, teams, and schedules that are commonly encountered, but contrary to reality. These bad attitudes find refuge and justification in Agile despite the fact they are false. Addressing these viewpoints before they, potentially, darken a good movement is essential. By Christopher Goldsbury"

Big Ball of Mud, Still the Most Popular Software Design

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Wed, 15 Sep 2010 13:15:00 +0000

Big Ball of Mud, Still the Most Popular Software Design:
"Big Ball of Mud, is a code jungle which is haphazardly structured, sprawling, sloppy and connected by duct-tape. Over the years we have been introduced to various guidelines such as SOLID, GRASP and KISS amongst age old, high cohesion and low coupling to deal with this Mud. However, the situation still remains bleak and Big Ball of Mud seems to be a popular way to design and architect software.
...

Interestingly, as per FJ, Yoder felt that many aspects of Agile directly lead to mud. These included,
1) Lack of upfront design
2) Late changes to the requirements
3) Late changes to the architecture
4) Piecemeal growth

By Vikas Hazrati"

Optimize by measuring, not guessing.

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Mon, 13 Sep 2010 13:30:00 +0000

Although it sounds obvious, it’s amazing how many times people try to deal with performance/scalability problems by guessing where the problem is coming from. I have seen it countless times. People come up with something like “it must be the XYZ module that’s taking so long, let me go and tune it”. Or “the serialization is slowing everything down – we need to switch to a binary format”.

I understand that people that developed the application justly believe that they know its good spots and its bad spots. But the problem is that performance degradation can come from so many other areas that it’s very difficult to accurately guess about the root cause of the problem even if you know the application inside and out.

In my experience the best way to improve the performance is to treat the application like a black box. This means abandoning all preconceptions about internal workings of the system. Just put the application under a good profiler (well, duh!) and observe. Many times I was surprised how unrelated the real bottleneck was to the suspected bottleneck. Often the bottlenecks are found in very low-level areas of the code that are not even considered when thinking about potential hot-spots. I remember how once after exhaustive attempts to improve performance of certain areas of application logic, the problem turned out to be in the low-level message routing loop.

Other times the problem can come from 3^rd party components such as jdbc drivers and DBs that are usually not even on the radar.

Even if people insist on “knowing” where the problem is coming from – what they actually have is a hypothesis, not an established fact. To prove the hypothesis they need to collect empirical data supporting it before proceeding on the assumption that the hypothesis is correct.

In conclusion: resist the temptation to guess about root causes of performance or scalability problems. Be scientific about it. Be methodical about it. It will save you time and frustration of chasing the wrong problem.

Importance of independent testing

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Mon, 23 Aug 2010 22:33:00 +0000

In this age of TDD, people are getting an impression that TDD or other forms of developer-testing virtually guarantee absence of bugs. They think that 100% code coverage means that bugs have nowhere to hide.

The problem is that developer-tests are made with the same assumptions and logical errors as the code itself. If you don’t think about a particular scenario that would cause your code to malfunction, you are not going to write a test for it.

Of course, modules/classes/services can and should be unit and integration-tested. That provides guarantee against basic errors and allows conducting safe code-refactoring by verifying that existing functionality is not broken. Personally I’m a big fan of integration-testing and intend to blog on it later.

People should also keep in mind that despite wide-spread beliefs in extereme effectiveness of developer-testing, empirical studies show moderate rate of bugs detection though developer-tests. In Code Complete 2 Steve McConnel reports that “Individual testing steps (unit test, component test, and integration test) typically find less than 50 percent of the errors present each. The combination of testing steps often finds less than 60 percent of the errors present (Jones 1998).”[McConnell, 2004]

That compares with about 80% detection rate achieved by modeling and prototyping, and 75% detection rate achieved by formal code reviews. You might not agree with the conclusions but the numbers are accumulated over a large number of projects and chances are that they are at least somewhat reflective of the reality.

I truly believe that in addition to all developer testing (include unit-testing, reviews and prototyping), independent QA/Functional testing is indispensable in finding real bugs that would otherwise make it into the production.

First of all, a person or a team of testers are free from the assumptions that led to the bugs in the first place. Second, a fresh pair of eyes and an experience (and talent) in finding faults of the system are what is needed to uncover the dark corners of the software.

Third, I’m a firm believer that a division of labor leads to a higher productivity. Without going into long economic theories, it is known that people that specialize in a particular skill tend to be more productive in it. From this stems the overall increased productivity of the team and the company. A tester is a specialist, well…in testing. That’s what he knows to do best. Just as a developer, he accumulated a bag of techniques and tricks that help him solve testing puzzles. Because of this he can find more bugs, more boundary-conditions, more unexpected inputs than the developer.

For these reasons I believe that rigorous testing by an independent team of testers is absolutely necessary in addition to all forms of developer-testing.

Steve McConnell rules!

noreply@blogger.com (Philopator Ptolemy (pseudonym)) — Sun, 15 Aug 2010 20:14:00 +0000

As person with over 15 years in software business I must admit that the biggest mistake I made is that I assumed that general programming and later management knowledge just comes to you with the experience.

Even if it is partially so, you’re inevitably exposed to a relatively small sample of problems and solutions. Furthermore, you’re bound to learn only from your own mistakes. In many cases people see those mistakes and the mistakes made by other people around them (including management) and assume that these mistakes are somehow unique to their project or their company.

But as the saying goes: “There is nothing new under the sun” (Ecclesiastes 1:9). Most of the mistakes that you or others around you made had already been made countless times before. It pays to read serious literature about avoiding such mistakes through use of best software development practices. Further more, this literatures should preferably be read before engaging is big development projects.

After reading quite a few books on general subjects of software development (besides technology and language-specific books) I can say that I found most valuable information in the books of Steve McConnell.

When I started reading “Code Complete 2” – it was love at first sight. The book made me completely rethink my approach to writing code. The book includes both low level advice such as recommended naming conventions and sizes of modules as well as high level concepts such as managing complexity though consistent levels of abstraction that is essential to intelligibility of systems. I believe that any serious software professional ought to read this book, preferably before he spends years on making and correcting his own mistakes.

The next book by McConnell that is more intended to team-leads/architects is “Rapid Development”. Although the book was published in 1996, it’s still very relevant today. Concepts and ideas in this book apply to today’s “agile” world as much as they did to “waterfall” world of past decades. It contains an invaluable list of practices and classic mistakes that every developer/manager encountered at some point. Same as with Code Complete 2, this book is essential reading for any professional software practitioner.

A more specialized book by McConnell is “Software Estimation”. The book contains exhaustive description of various estimation methodologies along with advice on applicability of each methodology within a given context. It also contains information on estimation of “iterative” (aka agile) projects. I believe that serious software estimation is impossible without basic knowledge of techniques described in this book.

A common feature of McConnell’s books is reliance on large studies of projects and code-bases. This is particularly valuable because it takes the information from realm of personal experience and opinion into realm of scientific and empirical knowledge that is so much missing in many books on software. I don’t want to start a flame war or invite angry feedback but books such as Uncle Bob’s “Clean Code” lack this kind of empirical proof and are mostly based on author’s opinion (with which I mostly agree, btw). I believe that software profession has evolved for long enough time that we can use information accumulated in countless successful and failed projects to better plan and execute new projects.

Lastly, I regret that very few colleges and universities include McConnell’s books as part of mandatory learning material. IMHO they contain infinitely more useful information to future software engineers than many abstract and outdated subjects in computer science.