Ruby, Javascript, Erlang blog: Snakes, Gems & Coffee

DRAFT: Why proponents of marriage equality should love graph databases, Part 2: A Reply to 'The Database Engineering Perspective'

2010-04-03T11:37:00.001-05:00

So in part 1 I provided an overview of problems relational databases (RDBMS) have with modelling relationships as well as entities that have datasets that are only partially structured in structure and definition based on my experiences.

This post will:

introduce new concepts utilized in graph databases and specific terminology for Neo databases
review the most flexible schemas that accommodate both same-sex and opposite-sex marriages alike that were presented in the blog post I am responding to, 'Gay Marriage: The Database Engineering Perspective'.
describe how graph databases could overcome the relational database shortcomings
finally I will present a snippet of code (not meant for production use, but merely to demonstrate how we can use Neo4J) that makes it more natural to represent richer relationships between entities (or nodes as Neo4J calls them) that have substance to them (i.e. attributes in this case)

New Concepts in Graph Databases

Before I can explain how graph databases on a conceptual level can overcome most if not all (for most of the time anyway) of the relational databases (RDBMS) shortcomings mentioned in the previous post, we need to be introduced to new concepts and terminology.
There are two basic classes of objects in a graph database, they are:

Nodes: a node is basically an entity as RDBMS people (like probably you and I) are familiar with. A node could represent a person, customer, blog post, photograph, video or tweet. It isn't always true, nor is it a good idea to think of graph database concepts only in terms of RDBMS concepts, but we could consider most tables that do not represent an actual relationship or association between other tables as an entity. It is a simplification, so use only in this initial learning phase cautiously. A node can have zero or more properties.
Relatioships: a relationship is the concept of an association between two nodes that needs to be represented in your data store somehow. A relationship can also have zero or more properties, just like a node.

The third concept in graph databases is that of a property. It unifies nodes and relationships as both first class citizens of the data store, which is quite unlike relational databases.

In a relational database we try to either reduce an association between two entities/tables to a small handful of cases, which might use references (if supported by that particular vendor and version of RDBMS), create "join tables" or we might force a natural relationship to become a table itself so that we are able to capture pertinent attributes for it. There is no other way of handling relationships. In many cases it might not be an issue to reduce the problem in these ways, however, the more and more interconnected our entities become with new types of relationships it seems (to me) that this method of modelling data quickly becomes problematic.

The following are other useful terms that the Neo graph database family uses, but conceptually most of these will also have a synonym in other graph databases:

Traversers: in graph databases "querying" for specific data is not done via declarative query statements and clauses like SQL. Rather we define traversers that are composed of the following elements:
- the starting node
- the relationship types needing to be traversed
- the stop criteria
- the selection criteria
- traversing order (e.g. breadth first, etc.)
We need to know the starting node so the graph database knows where to start the traversal. The traverser will then only move along relationships of the types given by the second element of a traverser, evaluate the selection criteria to know which nodes are relevant tot he "query", then evaluate whether the stop criteria applies to know when traversing should stop.
Indexers: In Neo4J there are several indexing utilities backed by a Lucene backend that make it easy to index actual nodes, full text and based on timelines. This allows us to look up relevant starting nodes for traversing.

How can graph databases overcome relational database shortcomings?

Now we will have a quick peak at how the new concepts and ideas from graph databases can resolve the complaints I had about relational databases:

In response to A lot of real world data isn't highly structured (only partially): with graph databases we only need to set the properties a specific node actually has. We do not need to fill in lots of nulls in attributes that aren't relevant like we do in RDBMSes. Our graph database "model" interface might be able to add any constraints that are necessary. While, in situations where highly structured data sets really do occur, I prefer having two sets of constraints - those on the relational database level and those in the "model" (application tier) - however, it can quickly become hard to manage when two sets of constraints that should be identical are defined in vastly different ways (languages).
In response to Object to relational mapping (ORM) constraints and disjoints: using nodes and relationships conceptually on the graph database layer you have little if any translation between 'objects' that represent nodes or relationships.
In response to Weak and inefficient "traversal" support: this isn't a problem with graph databases. Traversing data is much more relevant when dealing with data that is naturally in a network or graph formation already. Relational databases cannot handle networks or graph-like data sets without a lot of workarounds.
In response to Maintaining and evolving relational schemas: when the data sets you are dealing with are not highly structured and densely populated evolving strict data schemas do not need to be maintained and with Neo4J's flexible attribute setting/getting, data structures can be flexibly evolved when the data itself changes, however, this raises another issue of application code being able to read the old attributes in the graph database, but I will talk about that in Part 3.

Gay marriage: the graph database solution

Why proponents of marriage equality should love graph databases, Part 1: A Reply to 'The Database Engineering Perspective'

2010-03-13T10:28:00.003-06:00

This is a response to the excellent Gay marriage: the database engineering perspective blog post from November 2008 by Sam Hughs. The only discussion that didn't happen in his blog post was offering a NOSQL alternative to the gay marriage database problem, but I will not hold it against Sam Hughs because it was a comprehensive look at relational database schema designs for modelling marriage!

This response is a little dabble (i.e. not as thorough as the original post) into how graph databases could model monogamous same sex marriages without much problem and also allow for polyamorous marriages quite naturally. My goal here is to demonstrate how graph databases provide a great amount of flexibility without very much work at all, especially when the data application is just as concerned with connections between entities (aka records in RDBMS or nodes in graph databases) than just the entities themselves.

First off I want to look at the inherent problems with modelling real world data in relational databases and how the approach of graph databases can overcome many, if not all, of these problems. Then I will launch into a specific snippet of code to demonstrate modelling marriages that can be between any two consenting adults rather than just one man and one woman. Then I hope to outline how this could be extended to model poly amorous (including polygamous) relationships without much more work. Near the end I will also include a section at the end that discusses shortcomings of the graph database approach as well for completeness. My discussion will be focused on using Neo4J as it is the only comprehensive graph database I am aware of that is open source.

Update (2010-03-13): Boris (in the comments) mentioned that there is another open source graph database called HyperGraphDB

Update (2010-04-03): Johannes (via email) told me about InfoGrid, which looks like a very interesting alternative to Neo4J with a slightly different way of doing things.

What is wrong with relational databases?

For the last 14 years I have been using, designing, maintaining and administering relational databases in some capacity as a software programmer, developer, engineer and now as an applications architect. During that time I have found the following problems with using relational databases:

A lot of real world data isn't highly structured (only partially): this is probably the biggest problem with using relational databases (for everything) the way they are supposed to be. Sure you can add a blob, clob or text field to a table and add an arbitrary structure of data that depends on the record, which can be parsed by the application, but that would be a violation of all things relational and you would be paying a heavy price for doing this in a relational database depending on how big these "blobs" generally are. Not to mention you miss out on query-ability, which is something relational databases do well on highly structured datasets with the relevant indexes defined and a decent attempt at normalization. Note: I think relational databases are a fine thing when utilized to model data that truly is highly structured in the wild where the entities, not the relationships matter the most.
Object to relational mapping (ORM) constraints and disjoints: For almost a decade I have been using ORM libraries such as TOPLink (Java), Persistence (C++), Hibernate (Java), ActiveRecord (Ruby), DataMapper (Ruby), SQLObject (Python) and others. They each had their own specific problems at times, but they also possessed a set of common problems that resulted in creating a far from seamless integration between the object oriented layer (that a vast number of business systems and web applications are currently written with today) and the distinct properties of a relational database. These common object to relational constraints and disjointedness is not an ORM tool issue, rather it is a problem of trying to shove a round pin into a square hole (paraphrasing a comment made in page 3 of 'The Neo Database: A Technology Introduction').
Weak and inefficient "traversal" support: When I first started writing rich domain models in the nineties (that is last century for all you whipper-snappers) the focus of design was on the actual entities not relationships between the entities. Sure occasionally you had to model a relationship as an "entity", but for the most part you could do your best to reduce your domain relationships as much as possible to a combination of contain a(n) (aka belongs_to in ActiveRecord), contained by (aka has_one in ActiveRecord), one-to-many (aka has_many in ActiveRecord) and/or many-to-many (aka has_and_belongs_to_many in ActiveRecord) as much as possible because you can't attach attributes to relationships in a relational database unless you attempt to make it an entity. If you are interested in deep and/or rich object graphs, however, you have to pay the penalty in any RDBMS with multiple joins, which are an expensive operation. Even with all the right indices defined and queries optimized you will find more than three levels of indirection (aka JOINs) will take a while on even a medium sized dataset. So as application developers, we are forced to add lazy loading and/or customized eager loading logic into our rich object domain layer for a number cases where performance matters, which in my view pollutes business logic and makes the code less maintainable going forward. This is far from ideal. It is also not ideal that data set traversal patterns change with new the introduction of new features, so often you will need to fork lazy loading logic in the business logic layer to satisfy you new requirements. Adding a lot more complexity to manage in the application tier.
Maintaining and evolving relational schemas: modifying relational database schemas has always been a little tricky at best. Today I take advantage of using ActiveRecord's "migrations" that loosely orders (by creation timestamp) a set of relational schema modifications to run (and we wrote a simple extension to wrap them in a transaction to save our sanity - why wasn't that the default to begin with? anyway...). Even though there is a method to the madness now, it is still a little crazy, especially when it comes time to run these migrations on the production server. I always miss a heart beat or two when I run the rake db:migrate RAILS_ENV=production command (yes, even after taking a snapshot).

It has been fun thinking out loud, but I have stuff to do today. Hey, I sometimes have a life, honest!:) So I have made an outline of what is to come in the next parts of this topic. I have already written some (bad) Java code to model same-sex and opposite-sex marriages consistently using a graph database (in this case Neo4J), but I plan on offering a Python snippet too, since I really can't stand the look of Java any more and the API doesn't make the case of Graph Databases for those coming from more terser feeling languages like Python, Ruby, Haskell, Erlang, Javascript (well if you use sane APIs like jQuery at least).

What is coming in Part 2?

How can graph databases overcome relational database shortcomings?
Gay marriage: the graph database solution

What is coming in Part 3?

Poly amorous marriage using a graph database
Graph databases: problems to watch out for

How agile practices improve code review

2010-01-30T21:27:00.015-06:00

On a mailing list recently someone posed a question about how to incorporate code reviews into an "agile methodology". While I had questions about the way the question was posed it got me thinking about how agile practices make code review happen more often, more organically and more transparently. Having different members of the team review code at different times during the development process on agile projects is both explicit, but also implicit.

Values

Let us back up a little here. First off agile is at its core a set of four ideals. Prefer:

individuals and interactions over processes and tools
working software over complete documentation
customer involvement over contract negotiation
adapting to changing needs over completing a stale plan

Principles/Tenets

From these ideals there are about 10 or so principles or tenets that are most commonly discussed among agile practitioners from most of the main camps. This is where it starts getting controversial as some agile camps acknowledge some of these principles and others do not necessarily, so here are just a few that I want to talk about in relation to increasing code review on agile projects:

Collective team ownership of product: meaning everyone on the team owns the product and nobody on the team is exclusive to just one or a couple of components of the product.
Embracing change: instead of continuing down a set path determined long ago that no longer applies, the team needs to adapt to customer's changing requirements at set (and regular) intervals (these might be called iteration planning meetings/negotiations, for example). This has various implications throughout the iteration/sprint.
Verify expectations continuously: meaning whenever possible, without impeding development, verify that customer expectations will be satisfied and no new additions have negative impacts.
Customers set priorities: instead of technical staff or outsourcing companies specifying technical priorities, customers specify business/feature priorities.

Practices

This is the point where we start seeing the translation into agile practices, many of while you will start to recognize:

Pair programming which arose from both collective team ownership and embracing change tenets.
Test-first practice came about mostly to satisfy embracing change, but also verify expectations continuously
Assigning new stories independently of previous ones. This means that no developer or pair has a niche component or product area and is used to satisfy the first tenet listed above.
Continuous integration that helps satisfy the verify expectations continously principle.
No fear refactoring, which is only advisable if your test/spec suite has good enough coverage,. This means you will likely revise poorly constructed code on agile projects vs non-agile projects more often and have the chance to do it more often too.

Obviously with the pair programming practice that some agile methods promote, you have another set of eyes reviewing code as it is written. This catches errors closer to the time it was written, which is a big time (and money) saver.

Applying a test-first approach means that you start out defining custom expectations of the code first before writing the actual code. This is reviewing, perhaps not code in the traditional way, but the requirements, which in my view is even more productive usually.

When a project manager assigns new stories to a developer or a pair without confining him/her/them to specific areas of the codebase, this increasing the transparency of the codebase and exposes hidden dragons to be exposed before too long and hopefully before being launched into production (but you never know). Again this increasing the number of eye balls on parts of the codebase.

With continuous integration you can incorporate metric tools to capture some metrics of the code so that parts of the code that are obviously in need to technical refactoring get highlighted earlier rather than later. Increasing transparency of the codebase, which is always a goal of code reviews.

Hopefully you can see where I am going with this and not think about code reviews in the traditional top-down way where every month or quarter a piece of code is reviewed from each developer in a formal way.

There are a number of different agile (and common) software practices that could be used, I can think of 4 more off the top of my head that improves codebase visibility across more of the team.

So Long, Farewell, Auf Wiedersehen, Goodbye...to Ruby & Rails (sort of)

2010-01-17T18:37:00.006-06:00

It's been obvious to many of my close geek comrades that for the last 18-22 months I have been getting more and more frustrated with the Ruby community and the "quality" of products in this space. The quality of people in the Ruby field has been tending towards the average PHP developer (PHP is a gateway drug for programming - i.e. not something to aspire to) and the recent Ruby mindset (as a whole though some dark corners still look interesting) has not been about doing things differently, but rather blending in with mainstream ideas. For some that is exciting as it means not fighting management just to get to use Ruby in the enterprise. I understand, I really do. Fighting management on technical platform is never fun.

I think it is safe to say the direction of the major libraries and frameworks that are prominent in the Ruby landscape will suffer from very similar conceptual limitations as the Java, J2EE and JEE frameworks that DHH once ridiculed (for good reason). Of course, this is all to be expected in the usage lifecycle of a programming language. In fact, I believe I predicted that I would get bored of Ruby in a blog post in 2006 within the next few years though it doesn't make this any easier.

Thanks mates!

I wish all friends in the Ruby and Rails worlds luck (I think you'll need it) and I thank (ex-)coworkers and project collaborators that have taught me something new for all the intellectual stimulation you have provided me. I am sad to be saying goodbye to those of you.

But...

I look forward to grazing new pastures. I will be working on my next generation financial products using the right tool for each primary job. Currently most of the Finsignia Platform is quite back-end and different pieces could take advantage of Haskell's concurrency and/or Erlang distribution. Yet there will inevitably need to be (at some point) rich web applications for UI, so (J)Ruby and Rails I might come knocking on your door again, but for now, adiós.

Cloud Computing, Part 2: Law, Privacy & Maturity

2009-09-29T21:59:00.007-05:00

At the CEO roundtable event I mentioned in my previous blog post about in Cloud Computing, Part 1 the other major business consideration that came up was how to protect your business' (and/or customers') assets properly. Ultimately anyone that wants to build a solid foundation for their business needs to tackle this well to be successful long-term. One person asked the speaker to elaborate about a recent case where a UK bank used servers in datacenters inside India where a datacenter employee stole customer information and used it to effectively steal money from these people's accounts for his own benefit. This could happen in any country, however, in this case this datacenter employee actually didn't break any laws inside of India (or broke one very minor law that had a very limited penalty). Ultimately this lack of relevant law and regulation in a foreign country is the major risk for a business using offshore datacenter resources. Unfortunately I didn't hear the name of the bank involved to be able to find a news article about this incident, however, the speaker had heard about this incident (it wasn't just a figment of the questioners imagination) and his response (I am paraphrasing here) was:

Countries with the most mature infrastructure, laws, regulations and standards regarding data and information law (as well as privacy) will probably be the places that larger firms will invest in for sensitive and mission critical systems of their organization despite the likely higher cost of building.

As you determine whether you are going to be a cloud consumer or provider in the future, make sure to keep an eye on these risk factors as some executives seem to only focus on the raw bottom line without conducting a thorough risk analysis to see if the cost savings are really worth the higher risks. Choose the best cloud computing partners for all your cost trimming AND risk mitigation needs.

Cloud Computing, Part 1: Delivery Models

2009-09-11T21:10:00.011-05:00

Today I went to a regional CEO Roundtable lunch talk given by Professor Roy Campbell at UIUC on Cloud Computing and why executives of any kind of business should care.

While I am not new to the concepts of cloud computing and multi-tenancy SaaS, since I have consumed various cloud services and designed multi-tenant architectures for two previous clients. However, there was one particular "overview" slide that Professor Campbell showed that got me thinking a little. It was titled Delivery Models.

It was a text slide with bullets, but I think a diagram (or two) could have communicated to the higher-level audience more than most of the first half of his presentation could have (see below and let me know if it is more effective).

I especially liked Professor Campbell's categorization of the Delivery Models. Until this lunch time I had really only considered there to be two cloud delivery models: SaaS and PaaS. I wasn't making a distinction between Amazon services like S3 and EC2 and that of Google's AppEngine. On reflection, I am glad I took two hours out of my busy Friday if only to consider the distinction made in this talk.

In a nutshell the slide has three major bullets:

Software as a Service
Platform as a Service
Infrastructure as a Service

Nice! Google AppEngine, essentially being an application container (that supports various frameworks now), is a platform, whereas S3 and EC2 provide infrastructure on demand.

Now unfortunately this is where Campbell left it. For those of us not unfamiliar with cloud computing, we can see the importance of this distinction so that we can develop a more solid business plan (for those of us that might be interested in using the cloud to generate revenue from). It may also help technically savvy people that aren't familiar with the cloud.

Hope this helps people make a choice between container services like AppEngine or Heroku (platform) and those such as S3 and EC2 (infrastructure) from a consumer side and perhaps those devising business models to focus on their core competencies.

Next time I want to talk about privacy, legal and compliance issues surrounding Cloud Computing as well as how business should weigh the risk of outsourcing their data handling (however that happens) to countries will less mature laws, regulation and oversight. Perhaps cloud governance would be the most appropriate term?

Alan Turing Can Now Rest In Peace

2009-09-10T21:42:00.007-05:00

While I am by no means the biggest fan of the New Labour government in my homeland (mostly due to Blair's sociopathic reign), I have to give Gordon Brown props (or at least his aids) for giving credit to Alan Turing and saying Britain, as a country, fucked up big time (sorry, I really couldn't think of another phrase that matched the intensity - I'll replace it if you suggest a good alternative).

You can read the Downing Street blog post here:
Treatment of Alan Turing was “appalling” - PM

Thanks from this particular graduate from University of the Manchester's Mathematics department who now works as a software engineer who also happens to be gay!

"Ta luv", as the Manc locals would say.

Faker, Ruby on Rails Snippet & Generator

2009-09-09T19:07:00.012-05:00

Hey blog readers (if there are any left), I know it has been a while, and I know I have left you high and dry, so today I decided I will share some code and software development thoughts with you. I am moving to a more snippet oriented blog posting style from now on. Previously I preferred long ramblings about the political ideology of programming languages and even how Ann Coulter was _more_ comprehensible when I used my Metafusion Crypto library on her words (actually that was a marketing lie - she sounded just as incomprehensible using Metafusion Crypto as she did when not using it). Of course, this was all in 2006-2007 when blogging was all about spewing small tidbits that taught the reader nothing, so I had to be different and attempt to find a reason for things. Now things are swinging so I am trying to do the opposite from the norm again. Ok, on with it. Over the last week I have been using the Rails 2.3.4 "seed" mechanism, which in essence attempts to provide 80% of seed-fu's features inside Rails core. In addition on my personal projects I have been using machinist as my factory framework of choice (although I am forced to use factory_girl for my clients), through which I was introduced to faker. A beautiful gem indeed. So to cut a long story short, I have found myself doing the following a hell of a lot in my brand new Rails 2.3.4 projects this week:


# in db/seeds.rb
def load_seed_files!
  seed_files = Dir["#{RAILS_ROOT}/db/seeds/all/*.rb"] + Dir["#{RAILS_ROOT}/db/seeds/#{RAILS_ENV}/*.rb"]
  seed_files.each do |file|
    puts "Loading seed file: #{file}"
    load(file)
  end
end

load_seed_files!
# When I add files to the db/seeds/all directory it seed ALL environments
# When I add files to the db/seeds/{development,staging,test,production} directories it seeds 
# for only that environment.

Note: I wouldn't recommend using seeds in just staging, test or production generally. If you need seed data in these environments you _probably_ need them in all environments OR (in the case of the test environment) you need to use a factory framework like machinist or factory_girl. Now where is faker? Well it's what I use inside these .rb files where faker comes into play:


# Faking a company name
>> Faker::Company.name
=> "King-Ruecker"
>> Faker::Company.name
=> "Cartwright, Prosacco and Smith"
>> Faker::Company.name
=> "Pagac, Adams and Barrows"
>> Faker::Company.name
=> "Satterfield and Sons"
>> Faker::Company.name
=> "Blick-Deckow"

# Faking a person's name
>> Faker::Name.first_name
=> "Ruthe"
>> Faker::Name.first_name
=> "Jameson"
>> Faker::Name.first_name
=> "Avery"
>> Faker::Name.first_name
=> "Jordan"
>> Faker::Name.first_name
=> "Ruthe"
>> Faker::Name.first_name
=> "Winona"
>> Faker::Name.last_name
=> "Brown"
>> Faker::Name.last_name
=> "Hayes"
>> Faker::Name.last_name
=> "Jerde"
>> Faker::Name.last_name
=> "Lindgren"
>> Faker::Name.last_name
=> "Considine"
>> Faker::Name.last_name
=> "Rodriguez"
>> Faker::Name.last_name
=> "Turner"
>> Faker::Name.name
=> "Mekhi Lebsack"
>> Faker::Name.name
=> "Esteban Brown"
>> Faker::Name.name
=> "Velma Crona"
>> Faker::Name.name
=> "Rollin Zemlak"
>> Faker::Name.name
=> "Casper Sanford"

# My favorite is coming up with BS (yes, BS is what you think it is):
>> Faker::Company.bs
=> "redefine ubiquitous e-commerce"
>> Faker::Company.bs
=> "synergize viral e-business"
>> Faker::Company.bs
=> "innovate real-time experiences"
>> Faker::Company.bs
=> "aggregate scalable systems"
>> Faker::Company.bs
=> "e-enable back-end functionalities"
>> Faker::Company.bs
=> "implement 24/365 models"

So the point of faker, if you haven't already figured out is that it generates data the way you would probably have in production so you are generating meaningful data in your development environment(s) where you want to show your client what the interface will look like with data inside. Now a word of caution: I would not advise you to use this in an end-user facing test environment. I sometimes call that staging or qa, but ultimately it is an environment where real end-users or clients test out the features in a stable environment (not development or continuous integration environment) so they can sign off on your stories. In these environments you should let the user create the data themselves using the admin forms or whatever UI you provide them because you want to see how they will break the system (and we all know they will somehow). It gives you a much better indication of how things will end up working in production, which is very important. Thanks for reading and let me know if this is yet again too rambly? How does it happen? I have the best intentions to just blurt out tidbits with no nuance, yet I always end up rambling on about things and give it a thought process (well sort of). Anyone want to be my editor to strip out all thought and insight so I can be just blog brainless "tip" blog posts like everyone else in the Rails community that wanted to be a micro celebrity (trust me I couldn't handle that kind of paparazzi attention and I don't even have a Macbook Pro to sling over my shoulder in a fashionable messenger bag)? I can't pay you anything, but you could share in my measely $0.63 that Google Adsense pays me each month (actually they don't actually pay until a threshold is met). You could buy one third of an MP3 from iTunes each month. Any takers? What an awesome deal! No really! PS Coming soon a simple generator and Rails application template (or applate as I like to call them) that will set you up with this so you don't have to copy and paste all the time. It'll be on GitHub.com/mbbx6spp. Ta ta. Update: As Damien in the comments pointed out indirectly, I really didn't do Faker justice. Please read the Faker RDocs for more information.

Design Patterns Book Club

2009-07-03T10:05:00.006-05:00

I just discovered that Hashrocket has a video book club where people can join in the conversation with a live videocast on USTREAM or watch older episodes on their own schedule. They are currently going through Russ Olson's Design Patterns in Ruby. This is a great idea, kudos to Hashrocket. I watched the latest episode on the Strategy pattern and had a couple of thoughts I would love Desi and the rest of Hashrocket to consider for subsequent episodes:

I liked that this specific episode wasn't very long (I know you were a little worried)
I think discussing anti-patterns that could be solved with this pattern you are discussing the episode is very valid (as the person sitting next to Desi said). Perhaps that could be explored more. Note: I have only seen that one episode so perhaps you do this in other episodes much more, but if you don't please consider it, because I feel that anti-patterns are very important to identify. Think of the design patterns as the treatment or medicine and the anti-patterns as the symptoms. Physicians need to be acutely aware of the symptoms and symptom combinations to be effective at prescribing the best course of treatment or medication. Therefore we should be teaching software engineers in the same way.
Also please consider putting the name of the design pattern under discussion in the title of the video for archival purposes.

Thanks goes to Hashrocket for the great idea and work of putting it together! It's about time the Ruby community starts to mature w.r.t. language-neutral concepts and these efforts will hopefully help those that came into software development from a less formal software engineering education path (I used to be one of them and had to learn from scratch myself) with a little more assistance. It was pretty lonely bookwork and experimental projects for me to learn this stuff when I first started out.

Google Wave is not the answer, XMPP is!

2009-05-29T13:00:00.013-05:00

Since yesterday afternoon (my time) all over the Twittersphere (and blogosphere) I kept hearing a ton of great reviews of the Google Wave API/protocol and how it will revolutionize communication and collaboration on the web. This morning I watched the Google Wave demonstration from yesterday and I have to say I am completely underwhelmed considering everyone else's reactions were ridiculously positive and had nothing measured or thoughtful to say about it. You can find my initial reaction of it, "Google Wave is just a simple XMPP extnesion", on my tumblelog. As someone that has had to write XMPP extensions for clients and my own internal apps, servers, API interfaces (mostly distributed monitoring of services, but a few more for media processing) I am saddened that anyone would think the current Google Wave protocol (which is just an XMPP extension) is extraordinarily ground breaking. There have been a number of existing extensions that tried to do VERY VERY similar things already and specific features of Google Wave were already exactly defined. What made the demonstration of Google Wave yesterday appear so impressive were the applications: the wave client and the web embedding. The responsiveness of the web application really made it stand out, NOT the underlying Google Wave protocol/API. For those who have never really gotten their hands dirty with XMPP in any real technical way, I expected they would not really know what it already provided in XMPP. Many "geeks" hyping up Google Wave seemed to think XMPP only supported instant messaging. That just happened to be the most popular and apparent XMPP extension, which many people call Jabber. XMPP itself is a lot more far reaching than just an instant messaging protocol and the fact that Google Wave was based on top of XMPP should tell you how flexible it is. Yes that is right, Google may not have really made a big deal about it (why would they if they can get extra credit when it isn't due), but Google Wave is just an XMPP extension and compared to other extensions I have had to interface with, not a particularly complicated one. Sure the applications that Google demonstrated as part of the Google Wave presentation were nifty for personal communication, but are they really ground breaking, multipurpose, or ubiquitous? No, it is only as ground breaking, multipurpose and ubiquitous as it's applications make it. Let us all now return to earth for a moment and praise Google for the things they do really well like web applications, but let us not pretend that Google is the mover and shaker in terms of the visionary of future underlying protocols. They still advocate developing HTTP applications for the future. HyperText Transfer Protocol (HTTP) is virtually on every digital device made by man today, but does it really lend itself to providing the distributed service oriented architecture we should be striving for and the protocol of the future? REST is not about the HTTP protocol. RESTful APIs can be creating in a number of ways and through a lot of different protocols. My take is (from a technical level) XMPP ought to be the protocol of the future, NOT XMPP on just the back-end and then adding tons of bloatware in between the client and the service just so that we can create an API or UI on top of an antiquated protocol (HTTP) that has seen better days. Anyone want to create a truly modular XMPP "browser" for the new "web" with me? Email me at susanpotter dot net if you want to join the future. UPDATE: Please subscribe to the "Join the Future" Google Group if you want to get involved in the XMPP browser project.

Erlang doesn't support currying? News to me!

2009-05-20T16:13:00.004-05:00

I keep seeing developers who only glance at Erlang and think that currying isn't supported. Nothing could be further from the truth...unless you are defining currying support only in terms of specific syntactic sugar in a language for currying. So let us step back and define what currying support really means:

Currying is the process of transforming a function that takes multiple arguments into a function that takes just a single argument and returns another function if any arguments are still needed. http://www.haskell.org/haskellwiki/Currying

Let us look at a simple (but hopefully not too contrived, albeit it _slightly_ contrived) example:


%% currying_example.erl
-module(currying_example).

-compile([export_all]).

multiply(X, Y) -> X*Y.
doubler() -> fun(X) -> multiply(2, X) end.

In your erl shell try the following:


1> c(currying_example).
{ok,currying_example}
2> D = currying_example:doubler().
#Fun
3> D(6).
12
4> q().
ok

As you can see in the module code above, Erlang supports what is called anonymous functions (e.g. fun (X) -> somePredefineFunction(OtherVarOrConst, X) end), which then allows us to manufacture single argument functions in a way that is clear and expressive in the code IMO. For those that haven't been following the functional programming debates over the last week or two around the blogosphere, please see the following blog posts (some are pretty funny and will make you smile at least twice with their intended absurdity):

Previous posts on this blog that are related are:

I will try to put together a few snippets on currying support in Python, Javascript and Ruby tonight after my Greek class (which I am late for now) if I have time as a quick exercise:)

Twitter4R 0.3.1 and the future

2009-05-18T12:16:00.005-05:00

Hi Twitter4R users, I know I have been rather absent in terms of my Twitter4R contributions and ongoing development, but despite my absence I have _finally_ officially released 0.3.1 with a lot more new features and a few bug fixes more than I initially expected to release 0.3.1 with. Some of the goodies included in the official 0.3.1 release are:

Added raw Twitter Search API support. Meaning you can use any of the HTTP parameters the Twitter HTTP-based Search API will expect. E.g. twitter.search(:to => "twitter4r", :rpp => 10) or twitter.search(:q => "twitter4r", :rpp => 15, :geocode => "40.112186,-88.207592,250km")
Profile, theme API coverage on Twitter REST API.
Social Graphing API. E.g. twitter.graph(:friends, user)
Added "reply" support at all levels (low-level to model-level)
Bugfix: the infamous URI.encode to CGI.escape change.
Rate limit information (still needs to be able to support the HTTP header method, which I actually suggested to Twitter developers some time back, but they only recently added it when my bandwidth for OSS contribution was minimal) :(
much more

Now the future of the Twitter4R project. I have made a LOT of mistakes with this Twitter4R OSS project venture, so I am going to try to fix a bunch of these things:

Create a new set of Git repos on GitHub that separates the education portions (screencast, etc.) from the core source code of the project so that people do not need to download ridiculous amounts of data to their local drive just to fork Twitter4R on GitHub as it required at present. Announcements about this will be coming soon!
Have a consistent issue tracking system for Twitter4R. Previously it has appeared inconsistent to users of the Twitter4R library where to submit defect reports or new feature requests to. So I have had to field these requests from the Google Groups mailing list to personal emails and also @twitter4r replies on Twitter. I plan on using GitHub's new issue tracking system for this as it seems a logical extension.
I plan on documenting things a little more obviously on the official website from now one. It appears many users did not want to go to the RDoc unless specifically pointed to it, even though a large majority of the library is fairly well documented in the RDocs.

Please be patient while I fix these process/marketing/workflow issues of the project and in the mean time I will also be tackling OAuth support as well. Again thanks for your patience thus far and tweet me on twitter at either @SusanPotter or @twitter4r if you have suggestions or questions about the direction. If you have specific queries on using the actual API, please join the twitter4r-users Google Groups mailing list and search the archives for a solution first and then if you can't find a solution, please post a message giving the group members your environment details and the version of Twitter4R you are using. Thanks! Susan Twitter4R Developer

The Scala vs Erlang Debate, Part 2: The Geek Off

2009-04-20T15:47:00.028-05:00

In the first part of The Scala vs Erlang Debate article series, I discussed various ways to go about adopting either Scala or Erlang and in what scenarios it made sense to go with one over the other from the manager's perspective. In summary the major advantage of Scala is that it runs on the JVM so it can access pre-existing Java libraries your enterprise has already developed. This is also Scala's major disadvantage because it makes a rather large assumption (which in my opinion would be incorrect the majority of the time) that the Java libraries being used are thread-safe. Erlang on the other hand doesn't allow you to integrate in the same OS process with Java libraries, because Erlang was built from the ground up with scalability and fault-tolerence in mind and ideally you shouldn't put Java into the run-time stack because it compromises Erlang's fault-tolerance features significantly. Instead the preferred approach when writing the biggest bottlenecks of your system in Erlang and integrating with pre-existing Java libraries/system is to use SOA to create separate, but integrated systems that are easy to swap out and upgrade at a later point. In my view, you should be refactoring your enterprise architecture by defining services anyway, since this will decouple systems and remove a great deal of headaches in IT maintenance and deployment. Today I wanted to geek out and differentiate the two environments on a more technical level for the middle/technical managers.

Scala: Technical Overview

Scala is an objected oriented, statically typed language with many functional programming capabilities built-in. However, it must be noted (because few Scala documents do this) that Scala is hybrid actor programming language and therefore a number of assumptions cannot be made. So by being a hybrid functional language it losses some of the benefits. As well as it's Java run-time support on the JVM it is also supported in Microsoft's Common Language Runtime (CLR) so .NET integration, in theory, is a breeze. Again this also assumes that the .NET libraries you use from Scala are thread-safe. On a related topic, .NET also has F# available, which is Microsoft's attempt at a hybrid functional programming language also. I have no comment on F# myself, since I have zero experience with it. Please chime in on the comments section if you have experience with F# and are able to contrast it with Scala.

Erlang: Technical Overview

Erlang on the other hand is a dynamic, purer functional programming language that supports light-weight processes in its own run-time environment (separate from OS processes), which send messages to each other asynchronously. One major side effect of being purer in its actor treatment is that there is virtually not state. There is no way to pass "state" around (a couple of minor exceptions), but there is a lambda calculus optimization that can be taken advantage of that will ease away our troubles called tail-recursion. Tail-recursion only partially works in Scala. For many developers that only have experience in procedural and Object-Oriented (OO) programming languages like Java, Python, Ruby, C++, C, Perl, C# this sounds completely insane. Two years ago I may have agreed with you, but today life is much simpler without state in my Erlang travels than when I had to write thread-safe Java code and I wish I had discovered this concept earlier. I will not pretend this notion didn't take some getting used to but today I see things a lot clearer and I am thankful for that clarity.

Scala: Concurrency

Concurrency in Scala is modeled using a concept of actors, except there are two types of actors: thread-based and event-based actors. Since Scala was implemented on top of the JVM, which is stack-based, Scala had to compromise on Erlang's one actor approach. This is because native threads, which the JVM uses for concurrency support, are much heavier-weight than Erlang's lightweight processes thus reducing the number of thread-based actors that could be launched inside one JVM. So Scala created this concept of a very peared down event-based actor, which does not have all the functionality of thread-based actors but does not incur the penalty of needing to be backed by a native thread. To be as complete as I can be in this section I should also note that Scala treats everything as an object, even though it has these two types of actors. Actors in Scala are just objects. In turn this may create duplication between objects and actors.

Erlang: Concurrency

The Erlang VM was built from the bottom up with lightweight processes (a.k.a. actors), therefore it is stackless and much better at lightweight concurrency. Erlang also has a built-in mechanism for load balancing these lightweight processes across all available CPU cores. This means Erlang has a massive advantage over JVM based functional programming languages at getting the most out of multi-core hardware, which is at the crux of the problem domain that software needs to address going forward.

Scala vs. Erlang: Performance

In terms of linear performance (sequential) Scala has been shown to beat Erlang in certain cases. However, concurrent performance consistently shows Erlang beating Scala, sometimes by wide margins on multi-core hardware. Not surprising considering what we learned in the concurrency sections above. When considering the problem we are trying to overcome - optimizing software on multi-core hardware - it seems clear which of these benchmarks are more important! :)

Conclusion

I think it is clear, which side of this debate I firmly stand, but I have tried to demonstrate both good and bad traits of each programming language from a technical perspective that might translate into a business case or reason for choosing one over the other in your organization. If I have time I might do a part 3, which will talk a little about syntax and DSL support. This might round out the argument since it could be argued that Scala wins in the third part by a hair, but all is not lost for Erlang if the underlying problem you need to solve is related to optimizing software for modern hardware and building robust systems at the same time. My vote still goes to Erlang despite the lack of built-in features that make designing DSLs easier, but it is not difficult to build your own DSL engine in Erlang either.

Limitations of this series:

There are quite a few topics I left out of this post that might be of interest for some applications or services. Namely:

Hot swapping: Erlang has great hot swapping capabilities for production environments that minimizes or possibly eliminates downtime of systems. This feature in Scala is not close in capability or reliability at this point.
Garbage collection: without getting into too many details Erlang probably wins this section by a hair (not significant to write home about). Long sweeps will not be a problem in Erlang and only a problem in JVM languages on rare occasions.
Distributed nodes: The clear winner here would be Erlang because built-in to the environment are all the multi-node features that make distributed systems a breeze. Scala on the other hand has different frameworks and libraries to facilitate distributing nodes of logic on different hosts.
[Added Nov-2009] Frameworks: As far as I know, Scala doesn't have an OTP killer, not even a contender to compete with Erlang's Open Telephony Platform (OTP). Sure it will take some time, even Haskell, which has been around much longer than Scala is slightly jealous of OTP (I saw the way Don Stewart looked at me when I asked him if Haskell had framework similar to OTP;) ). Supervision trees along make it stand out as the best thing since sliced bread, at least for this particular software engineer.

Resources:

The Scala vs Erlang Debate, Part 1: The Manager's Overview

Corrections (added February 2010):

Erlang is not a pure functional language. However, it does not have a polluted (impure) treatment of actors like Scala:) Some think this is a good-enough approach to be able to leverage the JVM, when running on the JVM is really important, perhaps 4 times out of 5 it might be "good enough". I just wonder if it really matters for most projects that you run on the JVM!?
"Scala's lack of tail recursion is entirely due to a trade-off. The JVM doesn't do tail recursion itself. It is possible to add tail recursion on top of the JVM using various tricks, but they are expensive. SISC does its own stack based heap and Kawa uses trampolining." -- James Iry

The Scala vs Erlang Debate, Part 1: The Managers Overview

2009-04-15T09:37:00.017-05:00

Note: The discussion is primarily between Scala and Erlang, but other languages could potentially be substituted in the arguments for and against. Scala stands for any language that attempts to solve the multi-core issue software developers face with threads and Erlang is a language that has light weight processes which solves the biggest issue of going with a multi-process solution in software. The debate started with the realization that Moore's Law died a while ago. Tomorrow (as we see it now) is destined for multi-cores. A problem of massive proportions now faces software engineers, since hardware engineers changed the rules on us. Now I am not giving hardware engineers a hard time, I am simply pointing out that there has been a massive shift in the way hardware is built, which directly affects the performance and reliability of the software which runs on it. It is not a transparent change that software engineers can ignore. With that background laid out, as software engineers, we must appreciate the problem we now have to tackle, both server-side and client-side developers alike will need to tackle these issues promptly. However, this article only considers server-side software as there is where my specialized expertise lies. This part of the article is the high-level. I'll get into the specifics in the second part. So the question now is how do we leverage the advantage given to us by new hardware progress of multi-cores? There are two fundamentally different problems to solve: software products/projects that already have large complex systems already built in languages that support threads half-decently such as in Java and JVM languages including Scala; but there are also those startups that don't have baggage and can start with a better long-term solution by using a true concurrent language like Erlang.

The Legacy Enterprise

The problem with utilizing JVM threading (whether directly in Java or via other languages built on the JVM such as Scala) is that third party code you use or existing Java/JVM code written isn't necessarily thread-safe. You might have all your business entities and models defined in Java, but can it really be threaded? The pain of migrating or porting your existing system from a JVM language to another truly concurrent language like Erlang might be much greater than dealing with threading issues in just a few key bottleneck areas that will increase throughput and CPU utilization enough. In this case I doubt many pragmatic engineers or architects will advocate starting from scratch in a language you know little (if anything) about. However, identifying those key pain points might take some time, but will be a necessary cost if you are serious about solving this problem in the near-term using a solution like Scala. However, what these engineers and architects should stress to management is that this is likely to become a monkey patch. There will be a tipping point where moving to a pure concurrent language like Erlang would be more beneficial that sticking to the monkey patch and living on the JVM forever, no matter what Sun tries to tell you. There are hybrid solutions for the medium-term, that depending on the project/enterprise specifics may be a good solution until you are able to fully port the code to a langauge like Erlang (or whatever the trend may be at the time you get to it). SOA principles can be used to design and define a meaningful architecture that will declutter the system and allow you to tackle just the bottlenecks of the system in an Erlang port/rewrite and still give you (the manager) a low risk PoC project so you can evaluate Erlang in your enterprise. Having worked on a number of large enterprise projects where upgrading the "legacy" system was the goal, I know full well that management are likely to prefer the monkey patches (i.e. Scala where the pain of rewriting components in a PoC way doesn't exceed the pain of threading your existing code at the bottlenecks), especially when the middle/technical managers do not highlight the drawbacks of the "solution" adequately. Three to nine months later the management (especially middle management) are hurting big time, but it is not usually the most appropriate time to do your "I told you so dance". Trust me!:) The the monkey patch isn't an ideal solution when you are simply trading one complexity of equal of greater value for the other and in 2-5 years the likelihood that there will be many more skilled engineers and architects that truly understand and have solid production experience in a truly concurrent language like Erlang is high in my estimation, so making a move to such a language/environment would not be as risky at that time. However, for 2-5 years you have been getting little out of your hardware and software together unless you overbuy hardware. In fact, in the case where legacy code already exists, there is never really an ideal solution. However, starting out with meaningful PoCs and structuring low risk exit points after conducting objective evaluations of these projects is key to the success of every "legacy solution". If in doubt and your legacy system isn't showing signs of hurting yet, you might want to wait and beef up on hardware. This is a transition period and if you are in a conservative IT enterprise this might be a better near-term approach.

Startups

Now for the startups or those starting afresh, there is mixed news: very good news and not-so-bad news. Green field projects, in theory, are always the best. You can get everything right from day one....well at least you can hope to do so (let us pretend for this article). My advise for your team is to take the plunge to Erlang. It is actually relatively low risk to do so for the following reasons:

It is a mature language and environment already tested by large and complex real world systems at telcos where reliability and scalability were and are top priorities.
There are a few good books out there on that will help you convert to Erlang and Functional Programming (see resources section below), although documentation and education on Erlang, generally, is somewhat lacking.
A framework for services has been designed, developed and put into production called OTP, which stands for the Open Telephony Platform and actually nothing specific to Telephony applications in it, it was simply developed by a Telco.

Erlang's approach to concurrency is interesting and takes some getting used to as it is a pure functional programming language unlike Scala which is a hybrid. Now the bad news is that there are very few people that know Erlang (or other pure concurrent languages), you need to hire at least one specialist with Erlang experience to guide your team to avoid the potholes and perhaps even stop you before you get to the edge of a cliff. They might not be cheap, but starting out it will mitigate project risk significantly. Another negative on the Erlang side of the argument is that you will not be able to use your Java classes straight from within Erlang (in a meaningful way - I'll cover this in the next part). The positive about this negative though is that you can encapsulate your Java code in an SOA way so that you can decouple and declutter subsystems within your enterprise, which is an important task to do regardless of which legacy solution you choose. Finally, good luck and let me know how it goes!

Resources

Programming Erlang: Software for a Concurrent World by Joe Armstrong

This book, for the moment, is the definitive guide for Erlang. If you have already decided to go the Erlang route then you will not mind (or perhaps not even notice) the "selling" of Erlang throughout the book. Overall this is an essential book to anyone learning Erlang at this point.

Real World Haskell

While Haskell isn't either Scala or Erlang, if you do not come from a functional programming frame of mind you will need a good book to introduce you gently into the realm of functional programming. This book does a pretty good job. While the syntax for Haskell is different to both Erlang and Scala, the hardest part of moving to functional programming isn't the syntax. Trust me on this one:)

Also consider me, Susan Potter, a resource if you need an Erlang software engineer with high-level enterprise architecture background in Java/J2EE/JEE and Ruby/Rails production environments. I am developing a risk management product in Erlang right now and I am available as a consultant through my consulting firm Finsignia, which mostly works with financial services firms such as investment banks and hedge funds, which is where my business domain expertise lies, but also works on designing, building and deploying reliable and robust server-side solutions for any industry.

CUBarCamp: Champaign-Urbana's Tech Meetup Update

2009-04-04T20:13:00.005-05:00

This is just an update blog post to let geeks in and around the Champaign-Urbana area know that the CU BarCamp is now scheduled for April 25th, 2009. The meetup will start at 2pm CDT and will be located at the Collective Turf Coworking space:

110 W. Main Street Urbana, IL 61801

It is right between the Heartland Gallery and Priceless Books and opposite the Urbana Business Association office and Crane Alley.

Want to participate?

If you want to volunteer, present or demo any topics at this BarCamp please update the CUBarCamp wiki page and join the Google Group for the CU BarCamp event. Hope to see you there!

YUI MenuBar: A Facebook-like Application Bar

2009-03-27T18:59:00.005-05:00

Below is a snippet of HTML code with embedded Javascript code that will create a top anchored menu bar very similar to what you see at the bottom of the Facebook page:

<!-- Add the following to your HTML/HEAD -->
<link rel="stylesheet" type="text/css" href="http://yui.yahooapis.com/2.7.0/build/fonts/fonts-min.css">
<link rel="stylesheet" type="text/css" href="http://yui.yahooapis.com/2.7.0/build/menu/assets/skins/sam/menu.css">
<script type="text/javascript" src="http://yui.yahooapis.com/2.7.0/build/yahoo-dom-event/yahoo-dom-event.js"></script>
<script type="text/javascript" src="http://yui.yahooapis.com/2.7.0/build/container/container_core-min.js"></script>
<script type="text/javascript" src="http://yui.yahooapis.com/2.7.0/build/menu/menu-min.js"></script>
<script type="text/javascript">
YAHOO.util.Event.onDOMReady(function () {
  var oMenuBar = new YAHOO.widget.MenuBar("menubar");
  oMenuBar.render();
});
</script>

<!-- in your HTML/BODY modify the following for your own "Facebook Application Bar" menu -->
<div id="menubar" class="yuimenubar">
  <div class="bd">
    <ul class="first-of-type">
      <li class="yuimenubaritem first-of-type">
        <a class="yuimenubaritemlabel" href="#file">Applications</a>
        <div id="file" class="yuimenu">
          <div class="bd">
            <ul>
              <li class="yuimenuitem">
                <a class="yuimenuitemlabel" href="#">Notes<em class="helptext">Ctrl + N</em></a>
              </li>
              <li class="yuimenuitem">
                <a class="yuimenuitemlabel" href="#">Links<em class="helptext">Ctrl + L</em></a>
              </li>
              <li class="yuimenuitem">
                <a class="yuimenuitemlabel" href="#open">Photos<em class="helptext">Ctrl + P</em></a>
                <div id="open" class="yuimenu">
                  <div class="bd">
                      <ul class="first-of-type">
                        <li class="yuimenuitem">
                            <a class="yuimenuitemlabel" href="#">Upload Album</a>
                        </li>
                        <li class="yuimenuitem">
                            <a class="yuimenuitemlabel" href="#">Browse Albums</a>
                        </li>
                      </ul>            
                  </div>
                </div>            
              </li>
            </ul>
          </div>
        </div>
      </li>
      <li class="yuimenubaritem first-of-type">
        <a class="yuimenubaritemlabel" href="#"edit>Something Else</a>
        <div id="pim" class="yuimenu">
          <div class="bd">
            <ul class="first-of-type">
              <li class="yuimenuitem">
                <a class="yuimenuitemlabel" href="#"> Bla 1</a>
              </li>
              <li class="yuimenuitem">
                <a class="yuimenuitemlabel" href="#"> Bla 2</a>
              </li>
            </ul>            
          </div>
        </div>
      </li>
    </ul>
  </div>
</div>

For further details about customizing the YUI MenuBar look and feel you should visit the API documentation

CUBarCamp: Champaign-Urbana's Tech Meetup 4/22 (tentatively)

2009-02-24T00:04:00.003-06:00

Just a quick post to mention that with the assistance of Brian Marick and hopefully a few more people in the weeks to come, BarCamp is coming to the Champaign-Urbana area and is to be held at the Collective Turf Coworking space in downtown Urbana (110 W. Main Street, Urbana, IL - opposite Crane Alley and above Priceless Books). To keep updated on the latest developments and participate in the event organizing join the CUBarCamp Google Group. Hope to see you there!

Coworking Space in Champaign Urbana (Illinois)

2009-02-09T15:26:00.006-06:00

Since I moved to Champaign-Urbana, IL (aka CU) in 2005 I have been looking for work space that would provide me with a more productive environment and social yet professional outlet to work in since I work primarily on telecommute contracts and lack face time with clients. Today I decided that I will setup a local coworking space for solo entrepreneurs and freelancers in and around the Champaign-Urbana area to fulfill my needs as a consultant and fulfill my new venture urge of late. Essentially coworking is defined as the following:

Coworking is a movement to create cafe-like community/collaboration spaces for developers, writers and independents.

This is exactly how I imagined my dream work space since I work a lot in coffee shops all around town throughout the work week and even during the weekend sometimes and find it surprisingly (initially) productive. I have seen these types of office spaces in larger metro areas and the terminology used tends to be different from place to place. Some terms used to mean a community of independent professionals sharing office space, facilities and amenities that I have come across on my travels include:

Coworking or Co-working (obviously)
Collaborative work environments (not used in the context of a company job description:))
Work Space Cooperative (this might be organized at the top levels slightly differently though)
Shared Work Space
Serviced Office Space

There are probably some slight variants on the definitions of these as compared to that of coworking, so some places will have more structure and a more formal setting, whereas coworking itself strives to create a more creative and collaborative environment for independent professionals. If you are in the CU (East Central Illinois) area and would be interested in working in such a coworking space, please drop me a line at any of the following locations:

twitter: @SusanPotter
email: me at susanpotter dot net
facebook: My Profile | Champaign-Urbana Coworking Group

Are you a freelancer or entrepreneur in another area or region that is part of a coworking environment experience there? What do you like about your coworking space and why? What would you change about it if you could?

GEdit Extensions Project

2009-01-14T23:54:00.006-06:00

Just a quick post from a long blogging hiatus to let people know that I launched gedit-extensions where I will maintain a set of commonly used GEdit snippets and eventually plugins and tools that others might find useful. You can view the snippet source at the GitHub gedit-extensions repository. I plan on adding documentation to the gedit-extensions wiki in the next couple of days, but for now see the sneak peak in the Ruby example snippets at the end of the post. The name in the comments directly above the example snippet is the shortcut you can use. Type the first few characters of the shortcut and then Ctl+Space key combination to pull up all shortcuts the characters match so far. Choose the relevant shortcut from the drop down list. If the snippet contains variables the first variable will be highlighted for your to fill in, then press TAB to navigate to the next variable to fill in. As you will see most of the shortcuts so far are Ruby, Rails, RSpec or Cucumber specific since there are a few Python snippets I need to untangle from our in-house code references.

# shebang
#!/usr/bin/env ruby
# absrequire
require(File.join(File.dirname(__FILE__), "config", "enviornment")

# habtm
has_and_belongs_to_many :idiots
# unique
:unique => true
# habtmthru
has_and_belongs_to_many :idiots, :through => :village

# belongsto
belongs_to :village

# createtable
create_table :sessions do |t|
  t.column(:name, :string, :limit => 34)
  t.column(:description, :string, :limit => 255)
end

# addindex
add_index(:sessions, :name)
# changecolumn
change_column(:sessions, :name, :limit => 70)

# removecolumn
remove_column(:sessions, :name)
# irreversible
raise ActiveRecord::IrreversibleMigration, "the reason the migration is irreversible"

# resource
class PersonResource < ActiveResource::Base
  self.site = "http://domain.com:8080/"
  self.element_name = "contacts"
  self.prefix = "v1_0"
  self.user = "supo"
  self.password = "geditrocks"
  self.timeout = 60
end

# describe
describe PersonResource do
  before(:each) do
    @resource = mock(PersonResource)
  end

  it "should bla" do
    
  end

  after(:each) do
    @resource = nil
  end
end

# specmodelerror
it "should have an error on name" do
  @user.should have(1).errors_on(:name)
end

# specmodelrecords
it "should have 50 records" do
  @states.should have(50).records
end

# spectemplaterendered
it "should render 'common/user' on get '/user'" do
  get('/user')
  response.should(render_template("common/user"))
end

# specresponsetext
it "should return text 'You were signed out.' in response to post on '/signout'" do
  post('/signout')
  response.should have_text("You were signed out.")
end

# specresponseredirect
it "should redirect to action 'index' on post '/users'" do
  post('/users')
  response.should redirect_to(:action => 'index')
end

# specroutefor
it "should map #index to '/users'" do
  route_for(:controller => "users", :action => "index").should == "/users"
end

# specparamsfrom
it "should generate params for #update" do
  params_from(:put, "/users/1").should == {:controller => "users", :action => "update", :id => "1"}
end

# spectagexists
response.should have_tag('ul.menu#sidebar')

# specnestedtagexists
response.should have_tag('ul.menu') do
  with_tag('li', 'my menu item')
  # specchildtagexists
  with_tag('li', 'my second menu item')
end

# specblocktagexists
response[:sidebar].should have_tag('ul#sidebar')

# mockviewhelper
template.should_receive(:curent_user).and_return(mock(User))

# specsuccess
response.should be_success

# specredirect
response.should be_redirect

# feature
Feature: Search Jobs
  In order to find relevant jobs
  As a Job Seeker
  I need to find jobs relevant to me

  Scenario: Location Search
    Given a search form
    When I enter my zip code
    Then I should see all jobs in and around my zip code
  # scenario
  Scenario: Skills Search
    Given a search form
    When I enter skill keywords
    Then I should see all job descriptions that mention those keywords

Free Coffee for Voting

2008-11-03T10:50:00.008-06:00

As if anyone needs any more enticement to vote this election season, Starbucks has decided to reward people who cast their vote Tuesday November 4th, 2008 with a free Tall coffee. I do not believe you have to vote for anyone in particular:) Just go and vote. While you are enjoying your free coffee at Starbucks, perhaps explore the latest Merb 1.0 release candidate. To install simply do the following:

$ sudo gem update --system
$ gem --version # make sure it is at least 1.2.0
$ sudo gem install merb

Now to create a new Merb application do the following:

$ merb-gen app your-app-name-here
$ cd your-app-name-here
# now start editing the files you need to edit for you app.
# I promise more tutorials will be coming once I have 
# free time

Book Shelf Clearing

2008-09-27T21:52:00.020-05:00

In an effort to reorganize my office, I am clearing off space on my bookshelves and selling books (on Amazon marketplace) that I have already read. Since they are all software development related I thought I would list the most relevant below in case any blog readers are looking for books on the subject for cheap:

[SOLD] ~~Deploying Rails Applications: A Step-by-Step Guide (Facets of Ruby)~~ (buy it new or used)
Very good for the beginner or intermediate Capistrano recipe writer. It also contains some good pointers at the end of the book on benchmarking. Released relatively recently (within 3 months I think?)
SOA in Practice: The Art of Distributed System Design (Theory in Practice) (Published Aug 2007)
This book is more for the unRESTful SOA types than the RESTful folk like me. The author obviously has some old-time experience, so if you are into old school SOA standards (WSDL, SOAP, UDDI) then this would provide a lot of good architect-level areas to consider, which are often ignored including: service lifecycle, versioning, security, service management and more. One gripe I had with the book is in the versioning section. Let's just say his preference for naming is definitely NOT my style:) Otherwise this contains good reading material for the architect just getting into SOA in an enterprise setting.
Rails Cookbook (Cookbooks (O'Reilly))
Great for starting Rails projects with limited Ruby on Rails experience. It covers Rails 1.2, but most recipes will only require minor changes.
The Art of Agile Development
Covers all aspects of agile development from planning to delivery. If you aren't already an agile development ninja, this is in my top three agile books.
Webbots, Spiders, and Screen Scrapers: A Guide to Developing Internet Agents with PHP/CURL
Last year I had to work (for a short while) with PHP in the crawling/bot arena. This book contained got me developing bots and crawlers in no time in PHP.
Ajax on Rails
This would be a decent book for Rails developers that aren't believers in Unobtrusive Javascript (UJS). I have to purge this book from my bookcase now that I am a big UJS convert:) I hope you understand. I got this when it first came out, before I had seen the light. Seriously though, if you don't care about UJS principles, this book covers good ground on using AJAX Rails helpers. You should realize that the book was written pre-2.0 (Rails release that is).

Also those wanting to branch out on their own as a consultant, the following books might be for you as I found both very helpful when first starting my software consulting practice in 2001:

Getting Started in Consulting
If I could only recommend one of these two consulting books it would be this one hands down, since this is more about general consulting principles and practices and the second book might be slightly dated in the initial chapters, because both were written before the software outsourcing/offshoring mania.
Getting Started in Computer Consulting

PS I have been using Amazon Prime for two months now and love it. You can try Amazon Prime FREE for one month if you like too.

Tips to Enterpreneurs Hiring Ruby on Rails Consulting Firms

2008-09-22T10:20:00.041-05:00

I know I am taking a break from my DataMapper/Merb series of posts today, but I have seen enough fakers out there in the Ruby on Rails world that I just have to get a few things off my chest: Below is a list of things to avoid when hiring a Ruby on Rails consulting firm:

Overexcited marketing speak. Drop any firm that has something like the following on their website: "We believe Ruby on Rails is the most elegant, powerful programming language on the planet". These people might know a thing or two about configuring WordPress or Drupal or perhaps only Photoshop, but definitely no serious Object Oriented development of large scale Ruby server applications. Basically anyone that calls "Ruby on Rails" as a whole a programming language has an inadequate skill set to build a non-trivial web application. In addition anyone that knows more than three non-similar programming languages will know Ruby is not "the most powerful programming language on the planet". Sorry I had to find a quote on the web to prove I wasn't exaggerating. There are many more fakers on the web too, so look out! Remember you aren't looking for people using the most powerful programming language, you want results.
Agile this, agile that, agile everywhere. Beware of firms using the word agile like there is no tomorrow. Some use is fine, but ask them what they mean by agile. And what benefits you, the client, will receive from an agile team. Don't get me wrong if a web application is built in an iterative fashion with lots of client involvement and sign off, with lots of useful automated tests/specs that are run often then this is almost certainly a great thing for the project. However, beware of the firms or freelancers that are using agile simply as a buzzword. (Note: some firms like mine use agile a lot in web page titles for SEO purposes, so not all firms that use agile a lot in titles are faking. So make sure to ask firms what they mean by the word agile when they use it).
Why Rails? If any firm tells you that a web application written using Django, TurboGears, Merb or other web frameworks will never be as good as a Rails web application, then drop them off your consideration list immediately. The same is true if the converse is said. If they answer with something more measured and sensible like "our engineers have found developing applications in Rails to be more productive than X, Y and Z, which we previously used, and it provides us with the right amount of control/flexibility without us needing to worry about all configuration than previous frameworks" then at least they aren't misleading you just to get a sale. Of course, you might be looking for other points in the answer (other attributes of a web framework that would make business sense like active community, skillset, etc.). Also in general you should look for measured answers to all your questions, not answers that lack insight and promise they can do EVERYTHING under the sun for you in 2 weeks time. You will be disappointed if you hire that firm or freelancer.
Expert in everything, expert in nothing. Consulting firms and/or freelancers need to choose a specific set of technical competencies so they can be experts in those areas. Beware of consulting firms or freelancers that claim to be experts in every framework, programming language and operating system.
Ubiquitous buzzwords. Beware of too many buzzwords. Even if you know what a buzzword means, ask the person using too much jargon to clarify so you can evaluate whether they really know what they are saying.
Everything AJAX. I previously worked through a consulting firm on the west coast (I'll not name names). The consulting manager who led project management functions on all the firms' projects was convinced that if we just put an AJAX form in here and there it would solve all our problems. My experience with this firm wasn't overly positive. I like the people on a personal level, but their work quality and focus tended to create hard to use web applications. Beware of people who think AJAX is the answer to everything. It has its place, but don't go overboard. PS Make sure the people you hire write unobtrusive Javascript rather than embedded Rails Prototype helpers. UJS has many benefits including graceful degradation (benefiting those that do not have Javascript enabled), avoiding browser incompatibilities and improved code organization (separation of concerns). Users of your web application will appreciate the forethought and your developers can be more productive!:) Win-win.
Novice.has_many :problems Look for previous Object Oriented modeling experience for the engineers on the project team. Even though ActiveRecord (the default Ruby on Rails Object Relational Mapping framework) provides a higher level of abstraction for ORM associations than other frameworks, don't be fooled into thinking anyone can model your business domain correctly without experience and/or know-how. Many developers who are used to using Hibernate or TOPLink (as only two examples) before in Java or Smalltalk respectively will have likely done a lot of OO modeling. Ask them about the most complex business domain they have modeled previously to be able to guage their skill level and experience in this area. This point is most important in applications that have more complex business domains.
Why should I care about your data? Look at my cool Photoshop spinning logo. Rails consulting firms and/or freelancers should have a good understanding of the importance of data modeling. Even if they do not have specific data modeling expertise they should at least know that indexing appropriate columns and defining column size limits where appropriate can save not just storage space, but also shave time off queries (depending on the query and usage scenario). That way when your application gets to the point where the database needs to be fine tuned, the data architect you hire will have an easier job than if your schema was designed by a PHP weekender that thinks the Drupal or WordPress database schema is finely tuned (sorry, it's horrendous and yes you should know that anyway). This is a very important consideration if your application is supposed to hold a considerable amount of data and/or if you have strong reporting requirements. As a side note, make sure to ask them if different database schemas are sometimes needed for querying large data sets (classical reporting requirements) versus CRUD usages. The answer should be yes and you should ask them to explain this a little to see how much they really understand.
I can design, develop, deploy and manage perfectly. Usually firms and freelancers are much better in one or two areas than all equally. Some are better at graphic design and got into development to earn $20/hr extra. That's fine, but you should know what you are getting yourself into. Ask what their strengths and weaknesses are. For example, I am excellent at developing and very good at deploying and systems engineering. I can use Photoshop and Illustrator (just like graphic designers can copy Rails code from forums and paste it into their editor), but ask me to create a visually stunning logo that encapsulates your company's image and I am lost (just like the graphic designers that can't write original code themselves). Know what is most important for your application needs and hire the appropriate people. Usually outsourcing graphic design to a design firm and hiring a flexible developer that can work with external teams well, gets you the best value for money. Throw away the firms and people that don't have any weaknesses. They are lying.

One point I would like to make to round this piece off is that if you find a young, eager engineer with higher than average aptitude, he/she can really do a lot for your project. However, you should expect them to make a few mistakes. Nobody is perfect and without prior experience with large scale server applications it is virtually impossible for someone not to make mistakes. If you can afford a few mistakes here and there, then this might be the way to go. Give them an equity stake if capital is an issue for you, they are more likely to take a risk with you than a seasoned consultant with a mortgage to pay or spouse and children to feed. However, never hire someone who claims to be a Ruby/Rails expert at a high rate if they simply copy and paste sample code from Ruby/Rails forums online (you would be surprised how many there are out there). It's a terrible waste of your capital and frankly it takes work away from better firms and/or freelancers out there and you will end up with badly written software that runs poorly. As a final note, hire a firm, freelancer or in-house engineer that has similar energy level and compatible personality to yours. Or at a minimum hire a firm or freelancer that you can trust and has been recommended, but not just through popularity contests like those at WorkingWithRails.com (I know three freelancers who are excellent Ruby engineers with only have 1 recommendation on the site and I have worked with one person that has more than ten recommendations and consider them subpar to most Ruby/Rails developers I know).

DataMapper: Flexibility of mapping model attributes to table columns

2008-09-04T10:13:00.015-05:00

I know an esteemed ex-colleague of mine (even if he is an Apple zombie now - yes, becoming a fan of Apple on Facebook deserves some heckling;)) has reservations about the apparent unDRY-ness of DataMapper with migrations. I used to share his reservations before I wrote two applications using DataMapper. Neither Merb application was particularly spectacular, but both had to work with legacy database schemas. Oh the joys of crazy DBA naming conventions! Let's look at a brief example of DBA naming madness (some names were changed, but the conventions used are the same):

ordersTable:
  - uid_int: int (PK)
  - acc_id_int: int
  - amt_int: int
  - ref_code_string: varchar(64) natural key
  - desc_string: varchar(128)
  - entered_dt: datetime
  - changed_dt: datetime

In another table managed by a different group we had:

Account:
  - ID: int (PK)
  - Balance: int
  - UserID: int (FK)
  - EnteredDateTime: datetime
  - ChangedDateTime: datetime

I kid you not. Two different conventions, both relatively unreadable as Ruby attributes as they both violate popular coding conventions in some way. In ActiveRecord you would be left doing non-trivial coding to hide the ugly column names in the model and map Ruby-friendly names to the appropriate column. In DataMapper this is not so. Lending from the design pattern's primary purpose - to explicitly map model attributes to database columns to decouple the two - we can write something like the following using DM:


class Order
  include DataMapper::Resource
  set_table_name("ordersTable") #ridiculous name really, but what can you do if other older applications rely on this already?

  has(1, :account, :child_key => 'acc_id_int', :repository => repository(:accounts))
  property(:id, Integer, :serial => true, :field => 'uid')
  property(:amount, Integer, :field => 'amt_int')
  property(:reference, String, :field => 'ref_code_string', :key => true)
  property(:description, String, :field => 'desc_string')
  property(:created_at, DateTime, :field => 'entered_dt')
  property(:changed_dt, DateTime, :field => 'changed_dt')
end

class Account
  include DataMapper::Resource
  set_table_name('Account')

  property(:id, Integer, :serial => true, :field => 'ID')
  property(:balance, Integer, :field => 'Balance')
  property(:created_at, DateTime, :field => 'EnteredDateTime')
  property(:updated_at, DateTime, :field => 'ChangedDateTime')
end

This way we can write readable Ruby code with the model. Next time we can look at lazy loading: how to switch it on and off and when to use it for greater effect.

DataMapper does have migrations

2008-09-04T09:27:00.007-05:00

I wanted to bust a myth out there that DataMapper does not have regular migrations, just the auto migrations. At least in post 0.9 versions DataMapper has both. In other blogs about the topic I found quotes like: "DataMapper migrations pull your database structure directly from the Ruby code for your models, so there’s no need to write separate migration files...". This, in my opinion, gives the wrong impression to those that are worried about non-trivial schema migrations. The author of that quote appears to have never moved one column into multiple or vice versa (a non-trivial schema change), among other schema changes where a DataMapper auto migration would not work in a production environment. In merb you can just create a new DM migration with the following: $ merb-gen migration name_of_your_migration (Assuming you have set the use_orm setting to :datamapper in config/init.rb.) A simple DataMapper migration might look something like:


migration 1, :create_orders  do
  up do
    create_table(:orders) do
      column(:id, Integer, :serial => true)
      column(:amount, Integer)
      column(:completed, Boolean)
      column(:description, String, :size => 255)
      column(:created_at, DateTime)
      column(:updated_at, DateTime)
    end
  end

  down do
    drop_table(:orders)
  end
end

If you have problems with Boolean not being recognized, throw in the line: include DataMapper::Types to the top of the migration (yes, this needs some finessing obviously). Hope that helps a few people that may have been mislead by some blog posts inadvertently. Migrations are needed for many cases I come across where auto migrations just will not cut it without losing data in production (not a good thing if data matters in your application:)). BTW A minor annoyance of mine in Merb is that for DM migrations you need to use this ugly Rake task: dm:db:migrate to migrate to latest version. To rollback migrations you need to use this ugly thing: dm:db:migrate:down[version]. Where those using AR in Merb just keep using db:migrate and db:rollback. This is an area that needs cleaning up to promote ORM agnosticism the way Merb is supposed to do.

Next Generation Syndication and Publishing

2008-08-30T08:34:00.012-05:00

For the last year or so, I have been focusing on implementing RSS, Atom publishing and related standards for a particular client in the podcasting industry. Both RSS and (to a lesser extent due to its standard being finalized much later) Atom are being heavily used not just for blogs, podcasts, videos and multimedia content, but also used heavily for web 2.0 (or social media) activity "watching". Most people should be familiar with FriendFeed by now (if not, where have you been? really!). This month (August 2008) FriendFeed came out with their Simple Update Protocol (SUP) which sits on top of RSS/Atom publishing to make it easier for content consumers (very self-serving for FriendFeed, but at least they are coming up with some kind of "solution" instead of just bitching about the status quo) to identify feeds that have changed before they make a timed request for the feed every X minutes regardless of whether the feed has changes or not. In a nutshell, SUP requires a meta JSON feed that notifies consumers of newly updated feeds to publish data. The first time an RSS/Atom feed is subscribed to, the consuming agent (e.g. FriendFeed) will strip out the SUP-ID and the SUP feed URL from the xpath:channel/link element and associate this information with the relevant user/account or whatever. Then using the identifier for a feed returned by the meta JSON feed, the consumer knows the URL associated with that ID for feeds they need to be updated by. While this might solve some problems, it really does feel very much like a monkey patch. On the flip side I do understand to some extent why they took this path of monkey patching as opposed to coming up with something more revolutionary. Revolutionaries tend to get killed and only the second wave settlers reap the benefits of their blood, sweat and tears. However, I did want to mention SixApart's efforts to broadcast frequently updating consumable data in what should be a much more obvious way for former enterprise MOM developers like myself. The Six Apart Update Stream broadcasts over HTTP new public updates/posts as they happen. It's an interesting idea and much closer to the XMPP ideal that companies like Twitter have attempted. Any other former enterprise architects merely see this new trend in web [defacto] standards in publishing and consuming data as just another message oriented middleware (MOM)? And I am hopeful that finally the web has found a MOM in XMPP that can stand the test of public scrutiny.