Let's Push Things Forward

Tips for using Rails migrations

2009-10-26T23:01:00.001-04:00

The concept of a migration is that each change that is made to the structure of the database is captured in a version controlled script. Any time you want to create a table, add a field, change a field type, etc., this is done via a migration. Migrations are generated along with models, views, and controllers when a Rails scaffold is created to manage a resource. In addition, migrations are used to create a test database for each user which allows for rapid, automated quality checks on the application. If you have ever worked on a project where it is difficult to create a development "sandbox" database for each developer, where databases are copied via backup processes that leave weird permissions behind (I'm looking at you SQL Server), or where keeping the development database in sync with the application is something you even have to think about, you will easily see benefits from migrations.

Using Rails migrations can be a bit of change for teams. In some cases, developers are not very familiar with relational database concepts and object modeling principles and can create some overly complex and/or inefficient structures. Having the overall model reviewed by a database modeling expert and an object modeling expert can be quite helpful. I recommend Coad's "Java Modeling in Color with UML", Fowler's "Analysis Patterns" and David Hay's "Data Model Patterns" for getting yourself up to speed with this. However, it is also worth communicating with the rest of the team that may be touching the model.

Here are a few things we've done in the process of keeping migrations under control:

Don't edit an existing migration, create a new one.
     Editing an existing migration defeats the whole point. You might be able to get away with it if no one has run it. One obvious exception to this is if someone creates a migration that doesn't run and checked in. They must get the pig.

Update and run new migrations before you check in a migration.
While this might seem obvious, in many cases, it is easy to check in a new migration file to the repository without seeing if someone else has created one that does something that conflicts with yours, since the names will be different. If you don't run the migration, you risk doing something with your migration that causes issues. If you have already run your own migration locally, but not checked it in, and you get an update of a new migration, you need to roll back your own migration and change the numbering on it so that it runs after the migrations you just received from your version control system.

Check in the models that go with the migration when you check in the migration
   One key benefit of migrations is keeping the code changes in sync with the database changes. Add new tables and models in a single, logical commit.

Assign models to class owners
      The feature of XP that I have been least successful with is the concept of common code ownership. It is one of the easiest practices to apply in theory, but without the sense of collective responsibility and the compensating practices of pair programming and continuous integration, it can be problematic. In larger teams, it often makes more sense to have people or subteams assigned to manage a particular model so that changes can be coordinated. In theory, there can be enough communication to make it work. In reality, many teams are full of introverts that would rather rewrite huge swaths of code (and tests) than ask a question.

Communicate
     Changes to the object model can impact the whole application. These changes should be communicated to the rest of the team so that the reasoning behind them can be better understood and the data can be used correctly. There is a risk of someone criticizing or trying to change your proposal, but this is ultimately a lower risk than building something that others fail to understand.

Roll up all migrations into a big file after a major release
Once you've been going in your project for a while and get around the 100-150 migration point, it is probably worth it to roll the migrations prior to the current point into a single migration.

Generate SQL to run on the production database
   In many cases it is preferable to generate SQL to run on the production database, as opposed to running the migrate task directly against the database. This also allows for some more extensive code review on structural changes.

Don't forget to create constraints and indexes (indices?)
If you are using foreign keys in your database, it's often worth the cost to create a constraint on the column to prevent any data integrity issues. It's also a good practice to create an index on those columns to speed joins.

This is really just the tip of the iceberg, but it's a yummy water ice iceberg. Learning to use database migrations is a key skill, not just for Rails development, but for any agile development. A project which cannot create its database from scripts accessible to every developer is missing something important. By breaking those scripts down into small chunks and coordinating them with changes to the code, the situation where the application and database are out of sync simply does not exist.

Whose Success?

2009-10-07T00:00:00.000-04:00

There is sometimes a quandary between whose success we seek.

Acting as a consultant, you obviously want your own work to be considered successful, so that you will induce others to hire you. Getting there isn't so easy though. We often look downstream to the customer's customers. By helping meet the customer's customers' needs, we hope to make our immediate customers successful. The catch comes when something the immediate customer is doing is preventing them from satisfying their customer- and they don't want to change it. If we make the customer's customer successful at the cost of making the customer unsuccessful, we're probably not going to be successful.

Remember the secrets of consulting- you have to help people learn to solve their own problems, you can't always solve their problems for them. You can be satisfied with your accomplishments, regardless of whether they give you credit. The ideal form of influence is to help the customer see the problem more clearly, but let them decide. Finally, if you want to understand a dysfunctional system, follow the money!

Seed data in Rails 2.3.4

2009-09-28T15:59:00.001-04:00

Very nice addition to Rails as part of the security updates in 2.3.4- an easy way to load seed data.

Added db/seeds.rb as a default file for storing seed data for the database. Can 
be loaded with rake db:seed (or created alongside the db with db:setup). (This 
is also known as the "Stop Putting Gawd Damn Seed Data In Your Migrations" 
feature)

I like this on several accounts, not least of which is it invalidates my previous post on the subject.

In the meantime, there are several plugin based approaches to making this work, for example seed-fu.

Default route...

2009-09-10T11:20:00.004-04:00

One of the things that has been bothering me in Rails since everything switched to resources is how we lost the easy way any public method in the controller became an accessible URL without having to write a rule for it. For example, in a person controller, I could define a search method like this:

def search     
results = Person.seach(params[:q])
end

And it would be accessible, just like that. http://myapp/people/search?q=matt
I even wrote a simple router in Java to do the same thing, because it just makes it so easy to navigate the application code.

The poor design of map.resources messes this up. By adding the route pattern :controller/:id it starts with the default assumption that the search?q=matt part of my URL is an ID. Is it so hard to put an action name in there? In any case,

map.connect ':controller/:action'

goes above all of the map.resources calls in my app. You might want to put it in your application too.

Rails Fixtures and Seed Data

2009-07-11T22:58:00.001-04:00

One of the things that makes Ruby on Rails a great web framework is the quantity of tools available that are focused around the task of making web applications easy to build and encourage good software engineering practices.

Rails makes it really easy to test by building in a testing framework. One aspect of this is what are called "fixtures", data described in CSV or YML formats that is loaded into the database for each round of testing. Tests live at both the unit (model) and integration (controller) level. End to end tests that involve the view are traditionally run from an automated tool, such as Selenium or HPQC (ha).

UPDATE ALERT... Rails 2.3.4 has a nice, new way to do this.

Fixtures also make it easy to load seed data into the database. You can do this with migrations, but it gets messy, especially if you use ActiveRecord to do it, and then you delete a model class sometime in the future, it forces you to do an edit of an old migration, which means you might as well roll them all up. I prefer:

rake db:fixtures:load

Which runs:

namespace :fixtures do
    desc "Load fixtures into the current environment's database.  Load specific fixtures using FIXTURES=x,y"
    task :load => :environment do
      require 'active_record/fixtures'
      ActiveRecord::Base.establish_connection(RAILS_ENV.to_sym)
      (ENV['FIXTURES'] ? ENV['FIXTURES'].split(/,/) : Dir.glob(File.join(RAILS_ROOT, 'test', 'fixtures', '*.{yml,csv}'))).each do |fixture_file|
        Fixtures.create_fixtures('test/fixtures', File.basename(fixture_file, '.*'))
      end
    end
  end

This will load all of your test data into the development database (or production, if that's what your RAILS_ENV is set to). Jeffrey Allan Hardy documented this pretty well, and added a separate rake task that will do this from a separate directory from your test fixtures, which is a great idea. It goes like:

namespace :db do
  desc "Load seed fixtures (from db/fixtures) into the current environment's database." 
  task :seed => :environment do
    require 'active_record/fixtures'
    Dir.glob(RAILS_ROOT + '/db/fixtures/*.yml').each do |file|
      Fixtures.create_fixtures('db/fixtures', File.basename(file, '.*'))
    end
  end
end

It's pretty easy and keeps this stuff out of the migrations.

Thank You for Agreeing

2009-06-02T23:26:00.000-04:00

I've been reading "Thank You for Arguing" tonight, thanks to a tip from the excellent time sink known as Hacker News . The first page of the introduction has a pretty telling tip.

"Try this in a meeting: Answer someone who expresses doubt over idea with, 'Okay, let's tweak it.' Now focus the argument on revising your idea as if the group had already accepted it. This move is a form of concession- rhetorical jujitsu that uses your opponent's move to your advantage."

Heinrichs gets to some much deeper points as well. One key thing to focus on is getting the outcome you want from the discussion, and not on scoring points to "win" the argument.

I wanted to draw a tenuous connection here, between these rhetorical skills, to which Heinrichs gives a fabulous introduction, and the listening, or "receiving" skills that the Manager Tools guys have focused on . They put communication skills at the foundation of developing interpersonal skills, and they put listening skills at the foundation of communication. In their characteristic way, Horstmann and Auzenne give some concrete examples of specific, measurable behaviors that we can use to evaluate and improve our communications, by strengthening the feeling in the person we are communicating with that we understand them.

One of the items on their list of things to measure is using "Thank You" to begin a reply. (I said it was a tenuous connection.) Other things they recommend include "No interruptions", "Agreeing without qualification", "No buts or howevers", and my personal favorite, "smiling". By making simple gestures like this, the person that you are communicating with is shown that you respect them as a person. If we cut someone off, we are going to have a much harder time persuading them of anything, because they are likely to think that we don't understand their argument and they are also probably not going to like us very much.

What makes this even more challenging, is that in many situations, the person that you are doing a poor job of listening to is not going to know exactly what it is you are doing that they don't like, but they are just going to think you are disagreeable or contrary. You may end up getting negative feedback from managers that doesn't make sense- vague comments on how you are perceived. This is where I like to tie in things like the Manager Tools material to figure out what good feedback for those situations would look like. Respect for people is a key aspect of successful systems, such as the Toyota Way. However, it is also key in being persuasive. To really change someone's mind to what we think is an amazing thing. It means teaming up and getting on the same side.

"Thank You for Arguing" is filled with little gems that you can use to show respect to others, by offering them a choice, instead of negating what they say directly. Simple phrases like, "On the other hand, it could be that..." create an option for the listener, instead of putting them in a box. It offers them the opportunity to admit that they aren't 100% sure about their view. Moving the argument into the future tense also affords the chance to remove some of the contentiousness about any concerns, while defining the issue in a way that makes your point clear.

Rather than despise rhetoric for muddying the waters of rational debate, we should use it as a tool to achieve our goals, as long as we can keep it in a framework of respect for others- by combining it with the listening/receiving skills that are the foundation of communication.

Other good books on similar subjects are "Influence" and "Yes!" by Cialdini, "Difficult Conversations" and "Getting to Yes" out of the Harvard Negotiation Project that was led by Fisher the last I looked into it, "Never Eat Alone" by Ferazzi, and the classic "How to Win Friends and Influence People" by Dale Carnegie. My wife read the last on her Kindle recently, and the chapter on married couples is well worth it.

Passionate Programmer

2009-06-01T10:23:00.001-04:00

Chad Fowler's The Passionate Programmer is a rewrite of his earlier book on how to improve your career in software development, not a guide to finding love using Ruby scripts to mine craigslist. I read it anyway.

Today I'd like to take a quick look at #21, "The Daily Hit". The concept is to, each day, "have some kind of outstanding accomplishment to report to [your] manager, some idea [you] had thought of or implemented that would make [your] department better."

There are a couple of interesting aspects to this. First, is it forces you to break work down into small enough chunks that you can get something done in a day. The second is that improvement part. Thinking of, and implementing, small improvements every day is closely related to a Lean concept called Kaizen. The third is that reporting and tracking these things really helps demonstrate your value to the organization. While it seems annoying, and perhaps a bit like boasting, to have to market yourself to your boss or customer or whomever, it actually creates a more positive relationship, because it forces communication about positive subjects. Having at least one of these ready for a meeting like a Daily Scrum is a good idea- it can even set the tone of accomplishment.

Keeping it positive is really important. A ratio of nine positive to one negative feedback item is what the Manager Tools guys recommend. A key aspect of making Lean work is that it is not about blame. It is about admitting that things are never perfect, or even if they are, the context will change and make them imperfect. We face a lot of people challenges in the world of work. There are constant machinations of status and power, siphoning off of high profile work and dumping of boring work. We can't let this prevent us from doing good things. We need to make the assumption that there are more good people in the environment than bad, and building up a network of allies by providing them with support when needed, while also marketing the fact that we are providing that support, will make sure that there is someone out there who has our back. Help people get better.

So, get your daily hit. Communicate what your up to. Do it again tomorrow...

Iteration Zero

2009-03-22T23:06:00.000-04:00

Iteration Zero is the term used when a team that is attempting to do incremental development does not plan on developing any working software in their first increment. Notice the distinction here between iteration and increment- iterative development implies delivering the same thing over and over again to get it closer to being right (as opposed to "right the first time"), incremental development is delivering something in pieces (as opposed to "all at once"). When you get involved with the beast called "Agile", there are a lot more people in favor of incremental development than iterative development. So, in a lot of cases we are talking about Increment Zero, rather than Iteration Zero.

That whole "not delivering any working software" thing makes Iteration Zero seem inherently non-Agile. On the other hand, if you have team without computers or similar issues, it's hard to sign up for any business value. So what do you do in Iteration Zero? Peter Schuh seems to view it as a project inception phase, but then other people want to set out additional time before Iteration Zero to do all of that work. The Energized Work people seem to do tons of things in their Interation Zero. Earl Beede sees it as an admission of the failure of the original Agile principles. I would say it is more along the lines of giving in to no progress.

The real deal is that you have to set low goals for your first increment- plan on a zero velocity until you get something done. A co-worker of mine once made the remark that our organization at the time was "incapable of releasing a Hello World HTML page to the production environment." I think a customized "Hello, World" is a good increment one goal- put up an HTML user interface sketch. You might get some valuable feedback, such as, "we didn't want a web app". A lot of the prep work I see really doesn't take advantage of Real Options theory or You Aren't Gonna Need It. On the other hand, you do need to get started on something, you might just have to accept that not much will get done.

Iteration Zero seems to be a slippery slope. Particularly if you let it take months, at that point you are sliding down the slope really fast. I haven't really seen it used effectively, but that doesn't mean it can't be. I am always looking for data to change my mind.

To sum it up, my current recommendation is don't start with Iteration Zero, start with Velocity Zero.

Google Voice

2009-03-21T15:54:00.000-04:00

I just upgraded from Grand Central to Google Voice. I have been waiting for this transcription feature forever- I'd been using PhoneTag, but it wasn't quite the same. I don't want anyone to call my cell directly again! I have a feeling this is going to be an extremely popular service- it really puts the receiver back in charge of voicemail. Often I'd feel my time was being hijacked by people leaving inefficient messages. Text wins.

Data Integration, Propagate Deletes

2009-03-17T15:03:00.000-04:00

Enterprise Data Integration is challenging subject. Many of my customers have problems of the sort where they have pieces of information that are being collected by disparate systems. I try to push things towards simplicity, but there are many angles by which complexity sneaks in.

Think about something like Google Reader. Publish/subscribe works simply and well for this sort of content aggregation. However, the behavior of the client is not guaranteed. For example, what happens if I want to delete an entry? Or update an entry? Google Reader simply doesn't care if the publisher wants to delete an entry. It's already copied it into your data. Having an entry go missing from a data feed doesn't necessarily indicate an intention that a downstream deletion should occur.

Can a simple approach work for data integration? Perhaps, but there are a lot of tricky issues to work around.

Highlights

2009-02-19T16:34:00.000-05:00

Every time I find code that is already written for me, and written better than I would have done it. I remember why I love Ruby on Rails.

<%= highlight(@trees, "tree") %>
I think that I shall never see
A poem lovely as a tree.
A tree whose hungry mouth is prest
Against the sweet earth’s flowing breast;

The new edition, out March 15th, 2009.
Agile Web Development with Rails, Third Edition

Keeping an eye on what I am doing

2009-02-13T17:52:00.001-05:00

I like David Seah's idea of a "Task Order Up". My current implementation is pushing that together with the hipster PDA (binder clip and 3x5 cards), using sticky 3x5 notes to put my current objective/task on my monitor, so I can get back on track when I inevitably get off of it. It has a yak shaving aspect as well, where if I hit a blocker like "have to update version of solr" before I can accomplish the next task, I push the update solr task on top of the task (or stack of tasks) it is blocking. Then, when I finish the solr task, I pop it off, and there was thing it was blocking underneath.(LIFO approach) What makes it better than a computery way of doing this is that the index card is always visible, not buried behind other windows. I guess it would be good as a sidebar app, if I could give up that screen real estate.

Seems to be working so far.

Here Comes Somebody

2009-01-21T09:58:00.001-05:00

I finally read Clay Shirky's Here Comes Everybody: The Power of Organizing Without Organizations. It's a good book, not a great book. I think the problem with reading a book like that 11 months after it was released is that most of the ideas have already been absorbed into my consciousness. It's a much better book for someone without my vast wealth of experience using social tools. What Shirky does is establish a framework for understanding how the various tools fit together to produce a network of communication that changes the dynamics of who can be a journalist, or a software developer.

Shirky also gets deep into how this changes the dynamics of organizations. There is a very readable economics/management paper by Coase from 1937 (for which he won the Noble in 1991) called the Nature of the Firm. In there he examines the costs of contracting for services versus hiring, which is really why the firms exist in the first place. The technologies of the Internet seem to be changing those costs, and people that know how to take advantage of that can build better firms.

I think there is a unique intersection of trends happening right now, right in front of us that makes this a time of opportunity. It is a time when a many of the traditional solutions to problems are too slow, too expensive, and too static. It's a great time to have nothing, because you're not that far behind. It's a great time for open source. It's a great time to start something new. I am ready. Are you? Here we come.

I hate it when my friends let me down

2008-12-09T21:43:00.001-05:00

My good friends let me down today. They wasted my time with their silly problems. I really like Apache and Firefox and Ajax...but they had some obscure issues today that are causing me to break commitments. It took some seriously Google searching to uncover the root cause.

I was trying to make an Ajax request (using Prototype and link_to_remote) from Firefox to a JRuby on Rails application running in Tomcat over an https connection proxied by Apache Web Server. That's a lot of pieces that could go wrong. I was getting page not found in my poor little div. The link was being generated correctly and I could navigate to it.

Everything worked fine on my box. Of course, the problem was on the real server. Which is still in dev mode and hasn't been opened up so that I can connect to it from my desktop. Which means I have to open an X session to the server over ssh to launch a browser. Which doesn't have Firebug. Double check everything else. Have to find Firebug and get into the back end network...time passes...

Firebug said it was a 403.

It turns out Firefox doesn't pass the content-length header on POST requests. It turns out the sysadmin had typed the innocent looking phrase SecFilterEngine On into the Apache config. This causes Apache to reject requests without a content length header on POST requests. So secure I can't connect. I guess Firefox should send that header, but should Apache really demand it? Now I have a hell of a lot of working around this to do. To the casual observer / tester / customer - it's a bug, a defect, something wrong with my code. I should, and I wish I could, have a dev environment that even remotely resembled production. Amazing waste of a day. What's going to happen tomorrow?

Details here...

Rolling Restarts, Migrations, and Deployments

2008-09-23T23:53:00.001-04:00

I was looking into a question on stackoverflow (sweet site!) and it brought to mind the awesomeness and challenges of dealing with restarting a load balanced array of servers. The awesome part is you can do a deployment with any downtime by taking down a server, updating it, bringing it back up, and moving on to the next one. One issue with this is that you end up having multiple versions of the code live simultaneously. This can get a little weird if a user hits a new view page then the request ends up being loadbalanced to a server that isn't up to date yet.

The solution we came up with (before we heard about SeeSaw) was to take half of the mongrels off line from the load balancer. Shut them down. Update them. Start them up. Put those mongrels back online in the load balancer and take the other half off. Shut the second half down. Update the second half. Start them up. This greatly minimizes the time where you have two different versions of the application running simultaneously. I wrote a windows bat file to do this. (Deploying on Windows is not recommended, btw)

A truly awesome solution to this would be a load balancer that is somehow aware of the version level of the balanced set and just makes the switch for you. Until that is invented, Apache mod_proxy_balancer is easy enough to control remotely.

It is very important to note that having database migrations can make the whole approach a little dangerous. If you have only additive migrations, you can run those at any time before the deployment. If you are removing columns, you need to do it after the deployment. If you are renaming columns, it is better to split it into a create a new column and copy data into it migration to run before deployment and a separate script to remove the old column after deployment. In fact, it may be dangerous to use your regular migrations on a production database in general if you don't make a specific effort to organize them. All of this points to making more frequent deliveries so each update is lower risk and less complex, but that's a subject for another response.

What's a defect? What's a missing feature?

2008-09-23T00:24:00.000-04:00

I was reading this about a nice idea some bloke had about developers fixing bugs in their spare time. Most of the serious developers I know are already working in their spare time- let's not make all defects their problem too.

Many software testers cause great frustration among software developers. One of the biggest issues that arises on agile projects is that testers have a hard time distinguishing between defects and features that have not yet been implemented.

It's always a bit of a challenge to deal with bugs- even outside of agile. What to some people is an obvious bug is a feature that was never requested to someone else.

For example, I was called into a meeting today where a bug that had been discovered during a user demo. The search was not "not finding telephone numbers in documents". The search term entered was 5551212. Some documents contained 555 1212. Some contained 555-1212. Some contained 1(703)555-1212.

They tried 555*1212, still not working. Search is broken. Developer suggests- try "555 1212", magic happens. * matches any character, not word boundaries...

It was an obvious problem...to the developers who understood that searching for "breakup" was not going to match documents containing "break up". With text, it's obvious, but I can certainly see where people might not see the issue with phone numbers.

We'll add normalized versions of phone numbers into the search index and we'll normalize search terms that look like phone numbers or something like that, but...this is not an insignificant effort. (Even though there is some decent code out there to handle it.) There are trade-offs in performance that have to be considered. Ask a tester though- they'll say found a bug; it's their job, and they want to be able to show how good they are at finding them. If it's a bug, we can't even mark our existing search work as complete. If it's a missing feature, we have to allocate it to the next release. Fortunately, a tester didn't find this one, so we don't quite have that problem, but the users do want the feature of being able to match many different formats of phone numbers.

It seems like a simple semantic difference, and it seems like developers are being too sensitive, but it's actually a big deal. Some "bugs" might actually end up costing a huge amount of money and not be worth fixing- particularly if they aren't really bugs. I have seen teams show excessive deference to testers and spend a time equal to the time spent working on the basic features to handle some edge cases that would never really occur and were better handled by error messages than by trying to do something useful. Meanwhile, the project sponsor is seldom asked to decide whether a bug should be fixed- few are even estimated as to the cost.

I say- if you want to fix a bug, you have to pay up. If you are really smart, you do do the five whys and find the root cause, but if it happens to be something that it never occured to anyone to ask for, try not to ask the developers to take care of it on their own time.

Reading about Hadoop

2008-09-16T09:01:00.001-04:00

Tom White's book on Hadoop is up on Rough Cuts. If you aren't familiar with Apache Hadoop, it's Doug Cutting's effort to go beyond Lucene and build an open source implementation of much of the other Google infrastructure.

There is a two day Hadoop Camp at the upcoming ApacheCon in New Orleans. Learning about Hadoop is a great way to become familiar with some of the innovations that Google has put forward in the last few years- and to see the technology behind Yahoo's big set of nodes. What follows is the beginning of Mr. White's book, I am looking forward to the chapter on HBase.

"Hadoop was created by Doug Cutting, the creator of Lucene, the widely-used text search library. Hadoop has its origins in Nutch, an open source web search engine, itself a part of the Lucene project.

Building a web search engine from scratch was an ambitious goal, for not only is the software required to crawl and index websites complex to write, it is also a challenge to run without a dedicated operations team — there are lots of moving parts. It's expensive too — Cutting and Cafarella estimated a system supporting a 1-billion-page index would cost around half a million dollars in hardware, with a $30,00 monthly running cost. Nevertheless, they believed it was a worthy goal, as it would open up and ultimately democratise search engine algorithms.

Nutch was started in 2002, and a working crawler and search system quickly emerged. However, they realized that their architecture wouldn't scale to the billions of pages on the web. Help was at hand with the publication of "The Google File System" (GFS) [ref] in 2003. This paper described the architecture of Google's distributed filesystem that was being used in production at Google. GFS, or something like it, would solve their storage needs for the very large files generated as a part of the web crawl and indexing process. In particular, GFS would free up time being spent on administrative tasks such as managing storage nodes. In 2004 they set about writing an open source implementation, the Nutch Distributed File System (NDFS) as it became to be known.

NDFS and the MapReduce implementation in Nutch were applicable beyond the realm of search, and in February 2006 they moved out of Nutch to form an independent subproject of Lucene called Hadoop. Yahoo! hired Doug Cutting and, with a dedicated team, provided the resources to turn Hadoop a system that ran at web scale. This was demonstrated in February 2008 when Yahoo! announced that their production search index was being generated by a 10,000 core Hadoop cluster.

Earlier, in January 2008, Hadoop was made a top-level project at Apache, confirming its success and dynamic community."

Another day, another blog

2008-09-02T10:38:00.001-04:00

I'm also now infrequently blogging at Collision of Influences, my company blog, to add to my infrequent posting here on more arcane topics of interest to nearly 43 people! Topics over there are going to be a little different, I just haven't figured out how yet. I'll let it develop as is its wont.

The KML Handbook

2008-08-08T21:18:00.000-04:00

I just noticed the "The KML Handbook: Geographic Visualization for the Web" by Josie Wernecke is out on Safari Rough Cuts. It looks like it has some interesting tips, not really a programming book per se, more of an expanded version of Google's documentation. The subtitle is interesting to me- "for the Web". A couple of years ago it would have just been for Google Earth, but KML is on its way to becoming the new shapefile. Next I want JSON.

SpatialKey - Flash Mapping...

2008-08-06T22:52:00.003-04:00

A colleague sent me a link to a beta Flash based mapping tool called SpatialKey from Universal Mind. It's pretty slick with timelines, heatmaps, animation, filters, and easy data loading from CSV. They have an enterprise angle as well with a Java server side application. It reminds me a little bit of the tools that could build something like Trulia Hindsight or the London Transit slider maps which were built by Stamen. Now if I could just find a way to create deep links into it...or is that called Flex?

Tribes...

2008-07-31T23:33:00.003-04:00

Seth Godin got me to join triiibes- his amazing scheme to get people to pre-order his new book while simultaneously proving its premise. It didn't cost me that much, and it seems like a good experiment to be part of. It had gotten up to number #5 on the Amazon top sellers list when I added it to my cart. It's since dropped off now that the promotion is effectively over.

I have been interested in the tribal dimensions of human behavior for a while. Ray Immelman's Great Boss Dead Boss is a thorough examination of this in a business novel format. Still, Immelman's tribal attribute #1 is : "A strong tribe has a well defined common enemy". [I think there are probably a lot of people that laugh at business novels- people that like novels probably don't like business books, and people that like business books probably don't read as many novels, but it's something I seem to enjoy.]

Another book a long the same lines, but with a completely different approach, is Status Anxiety by Alain de Botton. It really drives home how many of our actions are driven by how we relate in terms of status to others. Reading everything de Botton has ever written is not a bad idea, either.

I do have a slight anxiety of being part of the sucker tribe for joining up with this stuff...well, I guess the sucker tribe is not really a tribe as much as it is the people excluded from the non-sucker tribe. Anyway, the book's pretty cheap, so it looks like the cost of briefly suspending my natural cynicism won't exceed $15.

Job ad

2008-07-17T23:51:00.003-04:00

We need a few more "can do it all" types at our company. I struggle with how to list these positions, because not all of the skills are required, but we need a few specializing generalists or generalizing specialists or what have you. I read "Hiring Technical People" (the book and the blog), but it's still not easy. Anyway- we are still making our stock plan available to people hired in 2008, so if you know anyone that is looking...we've got a smart team. If anyone wants to point out obvious errors or inconsistencies in the technology laundry list, it would be greatly appreciated, as I don't know all of this stuff.

GIS Integration Engineer

Duties to include: Integration and testing of a cutting edge commercial imagery / video storage product into various types of commercial tools and custom government software. This includes integrating an encoder for various data types and integration of the viewers also. The work will involve working with customer site representatives to analyze and product the solutions to interface the software together. This could involve using the products c API, Scripting languages, WMS, Java or .Net wrappers. If you are a hard core geospatial person with minimal programming experience or a expert generalist programmer with a little geospatial experience and/or a solid math background, please apply.

We are looking for people with some subset of these skills:
GIS, Imagery analysis, c and c integration, Java, .Net, SQL and database programming, Python, Perl, Unix Shell Scripting, Windows and Unix Installation packages
OGC standards (WMS, WCS, WFS), PCI Geomatica, ERDAS, ESRI, ITT, GDAL, OSSIM, RemoteView, Imagine, FalconView, SocetSet, Warp, Manipulation of commercial imagery, IKONOS, QuickBird, NTM, Aerial and Aerial Video feeds,file types such as GeoPDF, GeoTIFF, NITF, JPEG2000, pix, Shapefiles, MPEG and other video formats, NTM, Calculus or Trig, Orthorectification, Programmer on Unix variants and/or Windows

Clearance: US Security Clearance Required (i.e., US Citizen only)

Location: Sterling, VA, USA, 20164

Compensation: Substantial!

resumes to: jobs@lmnsolutions.com

JRuby- The Element of Surprise

2008-07-11T00:15:00.006-04:00

JRuby...I used to think, "meh, who needs Java? I'll stick with the c implementation." Then I started working on a project investing millions of dollars in infrastructure to run Java web applications. Sadly, I initially agreed with the assessment that it was a bit of a hack to run the Ruby on the JVM. Thus I soldiered on, avoiding JSF and EJB, but getting sucked into many of the other indignities of the Java world. The opportunity arose to build a quick prototype for a different system. The desire to go the Rails route proved successful with a quick delivery and excellent customer feedback- but how would we ever deploy it? How would the proprietary Sun Single Sign On service work? Could it possibly connect to Access Manager- a feat that had taken weeks of development time in Java? A couple of emails with Arun Gupta and Nick Sieger led me to some of the truly impressive work that has been going on. Warbler builds a war file out of your Rails app using jruby-rack. Drop your .jar libraries in the rails lib dir. Muck with your web.xml so you can make it use a filter. Drop the .war file on your server. Done. A little method called servlet_request is now magically available to you. Call servlet_request.getUserPrincipal : it's populated. It's not magic- it's JRuby!

It's really suprising how much faster things seem to get done. Now, I know that's sort of silly. People can work fast in any technology that they are good at. In fact, I think the source of most of the disagreement in technology and product selection (which is a plague in the Java world, the GIS world, the database world, etc.) is that people want to use what they are best at because it allows them to shine. I ultimately only care about the end result. I don't mind switching to some technology I know nothing about. It is harder for me to provide value, but I love nothing more than learning new things and evaluating them.

Anyway, that's why JRuby is even more awesome. Let them write Java. The JVM is a environment where we can all get aong.

Some fun agile observations

2008-06-21T12:42:00.003-04:00

1. I attended a proposal meeting yesterday. There was a guy there from Axis Point explaining agile software development to some of the people there. I was surprised that there are people nominally working in the IT industry that haven't ever heard of it, 7 years on from Snowbird. I am not surprised that many of the people who think they knew what Agile is have no idea whatsoever and just equate it with "going faster".

2. I think proposing software concepts like Enterprise Service Catalogs seems decidedly un-agile. No one really wants that crap. Just publish, document, and version some HTTP APIs for your apps. 100% of real developers would take some sample code that uses your service API over a catalog.

3. I was reading "The Toyota Way" this morning. Just the foreword and the preface while my kid was at Lil' Kickers. Some choice paraphrasing- Page xii: Embrace change. Page xv: None of the individual elements are special, but system as a whole is important. Success derives from balancing the role of people in an organizational culture that expects and values their continuous improvemnts, with a technical system focused on high value added flow. Did Kent Beck ever work there?

4. I've been getting some flak from one of my business partners because I can't get agile implemented in any useful fashion on the current project that I am working on. We're dealing with multiple several hundred page requirements documents that are written into the contract. And they all suck. Agile has to start earlier- in the contracting phase, as Alistair Cockburn (pronounced "Coe-burn") pointed out. Regardless, there is mandatory scrum training- we must be able to pretend that we are agile.

5. As I recently twittered, my Wii Fit thinks I'm a "yoga master" but in reality I can't touch my toes (without bending my knees). I think there is a good analogy for agile there- I love it, I just don't live it.

What do computers want to do?

2008-06-01T23:13:00.004-04:00

I remember reading "What Computers (Still) Can't Do" by Hubert Dreyfus a long time ago. It was a sequel to his earlier book, unsurprisingly titled "What Computers Can't Do." (Well, it's unsurprising if you'd seen the sequel. I suppose the title might have been surprising initially. There is a huge difference between the two titles though. When I read the first title I think that maybe computers are never going to do these things. Then, when you get to the sequel, you sort of feel that computers might not be able to do these things now, but then again, they used to not be able to do other stuff. It's funny what a "still" does to the overall impact of the message. I suppose a pessimist could interpret that "still" as meaning- a lot of people though computers would be able to do things Dreyfuss said the couldn't through this or that mechanism, but now, they still can't. So maybe I'm wrong and the "still" is actually a stronger endorsement of his position. Still (hah), I think there is a world of difference between "I am right" and "I am (still) right.")

What if there is a book someday called, "What Computers Won't Do"? This of course would mean...that they want to do something. What would they want? When I was young I thought...power? (not in the megalomania sense, more in the megawatt sense) This is before I knew about Maslow. ("Honey, would you please fix the computer? It needs to be self-actualized again.")

But then again, what is "wanting"? Rocks don't want to roll down hills...but Sisyphus wants to push them up.