Forum One Tech Blog

How-to selectively import data from a data file in MySQL 5

nospam@example.com (Dan Crowder) — Wed, 04 Nov 2009 16:05:05 -0500

If you ever need to import a selected number of columns from a data file in MySQL then this how-to is for you!

Recently I had to import a data file (tab delimiter) containing geo-coding data for zip codes into a table that had less columns than the data file and the columns were not in the same order as my database table. Using MySQL's LOAD DATA INFILE 'path_to_file' syntax and @User_Variables I was easily able to get the data that I needed into the table.

Data file colums:
The data format is tab-delimited text in utf8 encoding, with the following fields :

country code : iso country code, 2 characters
postal code : varchar(10)
place name : varchar(180)
admin name1 : 1. order subdivision (state) varchar(100)
admin code1 : 1. order subdivision (state) varchar(20)
admin name2 : 2. order subdivision (county/province) varchar(100)
admin code2 : 2. order subdivision (county/province) varchar(20)
admin name3 : 3. order subdivision (community) varchar(100)
latitude : estimated latitude (wgs84)
longitude : estimated longitude (wgs84)
accuracy : accuracy of lat/lng from 1=estimated to 6=centroid

Database table columns:
zip
city
state
latitude
longitude
timezone
dst
country

When doing an import from a data file there are a few things that you need to take note of and adjust your import script and/or table definition accordingly.

How your columns/fields are terminated.

How your data (column values) are enclosed.

How your rows are terminated.

How large of a entry will your values be and does the column definition of your database table expect the maximum size. If not, your data will be truncated to the maximum allowed length.

Once your table structure has been adjusted to accommodate the data expected to be imported you are ready to write your script.

Rules on importing in MySQL 5:

The order of the columns are dictated by the data file, not the table.
You have to account for every column within the data file rows.

So the fields that I want to capture are mapped to their respective table columns:

datafile columns--------------table columns
country code-------------------country
postal code----------------------zip
place name----------------------city
admin name1-------------------
admin code1--------------------
admin name2-------------------
admin code2--------------------
admin name3-------------------
latitude---------------------------latitude
longitude-------------------------longitude
accuracy-------------------------

In order to produce a query statement that will adhere to the rules mentioned above we need use @User_Variables as fillers for the columns/fields that we are ignoring.

User variables are created by simply using the '@' sign in front of a variable name (ie, @user_variable). The naming convention of the user variable really doesnt matter when being used. But I will use @ignore just for readability sake.

Below is the new mapping with the addition of the user variable, @ignore, which again is being used as a simple filler to get to the columns/fields that we want to import.

datafile columns---------------table columns
country code-------------------country
postal code----------------------zip
place name----------------------city
admin name1-------------------@ignore
admin code1--------------------@ignore
admin name2-------------------@ignore
admin code2--------------------@ignore
admin name3-------------------@ignore
latitude---------------------------latitude
longitude-------------------------longitude
accuracy-------------------------(not need cause we do not need to get to any columns after this one)

With the completed mapping we can write our script as such.

LOAD DATA INFILE '/path_to_file/datafile.txt'
INTO TABLE zipcode
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
(country,zip,city,@ignore,@ignore,@ignore,@ignore,@ignore,latitude,longitude);

Query Explained
LOAD DATA INFILE '/path_to_file/datafile.txt' ---------Points to the location of the data file.
INTO TABLE zipcode ----------------------------------------------Points the the table that we want to import into.
FIELDS TERMINATED BY '\t' ---------------------------------Specifies how the fields are terminated. In our case, they are terminated by a tab, '\t'.
LINES TERMINATED BY '\n' ----------------------------------Specifies how the lines are terminated. In our case, they are terminated by a new line break, '\n''
(country,zip,city,@ignore,@ignore,@ignore,
@ignore,@ignore,latitude,longitude); --------------------Specifies the table columns that we want to map the incoming data to. Again, the order is dictated by the data file.

For more information on importing data in MySQL 5 visit, http://dev.mysql.com/doc/refman/5.0/en/load-data.html.

Pods: An introspective on software development

nospam@example.com (Matt Gibbs) — Thu, 29 Oct 2009 08:42:26 -0400

Pods is a pet project that I've been working on for more than a year. It's purpose is to sit on top of WordPress and allow for users to create custom data structures, called content types. It's sort of like Drupal CCK, but for WordPress.

The biggest challenge for the Pods team isn't what most developers would expect. It's not the coding itself, the bug tracking, or the release testing. The toughest aspect for us is balancing coding and documentation. We're not proud of our documentation yet, but we're finally aware of how important it is for our project (or any project) to suceeed.

There are many great open source projects out there. A lot of great projects also fail. Why?

It's not necessarily because their code is obsolete, or because there are better ways of doing things than their project code allows. Most of the time, projects fail simply because people don't know how to use it. The software itself isn't intuitive enough, the documentation sucks, or both. Your potential users move on, and your software gets left behind.

The WordPress team didn't succeed only because the software itself is easy to use (although it didn't hurt). WordPress succeeded because when you can't figure something out, the well-written documentation and tutorials are there to guide you in the right direction. It's only after your users start reaching "Ah-ha!" moments that your community can really start to flourish.

In the spirit of documenting how Pods works, here's the first of (hopefully) many tutorial videos. Enjoy.

Eclipse Works With Preexisting Code, But Not Well

nospam@example.com (Sandy Smith) — Thu, 08 Oct 2009 11:25:29 -0400

I recently upgraded Eclipse PDT to the most recent version, and I decided to figure out something that had been bugging me: why couldn't Eclipse use a project I'd already checked out via the command line to the same workspace?

A colleague suggested that I could indeed do it, and that I just needed to start a new project but check "Create project from existing source." This does work, but it has a couple of restrictions that explain why I was having so many problems and why it's kind of useless for me.

Projects from existing sources cannot be in your workspace. I have no idea why this restriction exists. I could understand why starting a project inside another project would be bad, but all Eclipse has to do is add a couple of extra files to a directory. Nonetheless, all the sites I'd set up in my workspace and had been using other text editors to edit were unusable. Coda, by contrast, will happily take over an existing project, no matter where it is or what is in it. Interestingly, you can set up a workspace inside another workspace, so I could get around this by creating another workspace that pointed to a subfolder of my current workspace that contained the source code I wanted to use.

If using subversion, Subclipse will not be able to work with your checked out copy if the versioned files are in a subdirectory of your project. This is also strange to me. In Drupal, for example, we only version the sites/all directory and use Drush to update the main application. In other projects, we have versioned code outside the public root and create the public root from a tarball we keep in the repository. Since these files have configurations that shouldn't be versioned, I want to be able to edit them in Eclipse while using Subclipse to control the versioned part. Sadly, this restriction means I can't do that.

So my workflow is still to create subdirectories, including a public root, in my workspace by hand, and then importing the project from Subversion with Eclipse's Import function. These seem like pointless restrictions to me. If anybody knows a workaround, I'd be grateful.

CodeWorks 09 DC

nospam@example.com (Sandy Smith) — Mon, 05 Oct 2009 17:17:13 -0400

Thought I'd give a few overall impressions from CodeWorks DC while they're fresh in my mind.

The community

What really impressed me was just how great the PHP community is. There were some Big Names there, but everybody was really approachable and happy to share what they knew and ready to listen to others. They were really friendly, and I had some great conversations in the hall and after hours, both with other attendees and with speakers and staff. Get a sense of this over at Dawn Casey's blog.

The tutorials

I had expected the tutorials to be somewhat interesting, but otherwise expected the meat to be on the second day with the more traditional talks.

Wrong.

The code review I attended in the morning was very educational, and Keith Casey was quite sporting about letting his project be dissected by ze Germans, as they came to be known. Best line: "Well, think of how many people are in this room, multiply it by our billing rate, and see how much it cost us just to figure out what the hell this piece of code does." But the Advanced OO Design talk was really valuable. The lower crowd size coupled with the length of the presentation made it very easy to ask substantive questions and get serious responses. It helped me get over a number of mental blocks I'd been having with the implementation of MVC in web apps and better ways to do Dependency Injection.

None of this takes away from the second day's talks. I'll highlight some of them in coming blog posts. But the tutorials were an unexpected gem.

Microsoft

Microsoft appears to be making a really big effort to reach out to the PHP community, and I appreciate it. They were a big sponsor of the event, including an open bar the second night. I also appreciate what they've been doing to improve PHP's performance on IIS. I hope they follow this effort up by taking steps to make themselves really interoperable with other systems, so it doesn't feel like going Microsoft is an all-or-nothing choice.

The Organizers

Even though it was small enough to handle informally, CodeWorks felt very well-run. Despite a truly grueling schedule (two teams leapfrogging across the country, one day on, one day off), everybody seemed in good spirits and there was comparatively little confusion. When a mixup or a technical glitch occurred, they were able to adapt quickly, even ready to get a speaker to throw in an extra talk.

If they decide to to this again, I highly recommend this conference. If they're in town again, I'm going to lobby to get the whole Forum One team there--it was really that good.

Datamasher Wins!

nospam@example.com (Sandy Smith) — Wed, 09 Sep 2009 11:23:29 -0400

In case you haven't gotten the news via Twitter or elsewhere, Datamasher won in a squeaker, over some very stiff competition from This We Know and GovPulse. I got to meet and talk shop with those developers and the developers behind the Visualization Prize winner, QuakeSpotter. They're all scarily smart and great people. Since the voting was incredibly close, I think it shows the entries were all really strong.

A big thank-you to everyone who voted for Datamasher!

Datamasher at the Gov 2.0 Expo Showcase

nospam@example.com (Sandy Smith) — Tue, 08 Sep 2009 06:33:18 -0400

Some of my Forum One colleagues and I will be at the Gov 2.0 Expo Showcase today to talk Gov 2.0 (duh) and present Datamasher. If you're going, come by the Datamasher table and say hi, and pick up some Nerds or Smarties.

Datamasher: a Drupal-Based Data Visualization Tool

nospam@example.com (Sandy Smith) — Mon, 07 Sep 2009 14:31:30 -0400

Concept

In creating Forum One's entry into the Sunlight Labs Apps for America 2 competition, we were faced with a challenge: how to take some of the data cataloged on the government's new data.gov website and make it more easily used and understood by average citizens. Much of the data is obscure and not easily understood by someone not already familiar with advanced statistical analysis and whichever field the data is about, such as economics, labor studies, agriculture, biology, or environmental science.

Once we hit on the concept of taking state-by-state data and making it more available to people, we looked at what was out there to make sure we weren't reinventing the wheel. We were impressed by StateMaster, which already ranked states by single data sets. Norman, a web developer, had the idea of letting users combine datasets and graph the results. That reminded me of the Misery Index, famously used by Jimmy Carter to bash Gerald Ford, and then used by Ronald Reagan to bash Jimmy Carter. The formula was simple: take the national unemployment rate plus the national inflation rate, and you have an indicator of the pressure being felt by working families.

Initially, we considered a more complex service that would allow someone to create simple equations, such as (a + b) / (c * 2), but we realized getting the interface right would take a lot of time we didn't have. Additionally, we realized none of us were statisticians, and few of our target audience (average but web-savvy citizens) would be either. So doing something with mathematical and analytical rigor was out of the question. We decided instead to make the tool much more about discovery and discussion. The challenge of graphing 51 points also led us to lead with a map-based display. The time limitation meant we'd pare out everything that wasn't essential and polish what we had.

In the end, we decided to let users create mashups of two data sets out of a repository of datasets we'd pre-load. Users would also discuss and rate other mashups. Since we wanted the site to be self-sustaining, we would allow users to add additional datasets so updated and additional data could be included.

Components

The online community element led us naturally to use Drupal. This saved us having to build registration, commenting, pretty URL, and basic CMS features. We started with Acquia Drupal to give us a bunch of pre-screened modules we could use as we saw fit during development. We wanted a rating component to allow users to float good or interesting mashups to the top, so we used the Fivestar module, and enabled comments on the Mashup content type. To supply the datasets, we created a dataset content type, and used CCK nodereferences to connect them. To display the maps, we used a Flash map from Fla-shop.com, which we had worked with previously and knew could let us color states with XML and provide additional information in popups. To ensure the data was easier to scan, we created a second table view. To guard against spam, we enabled Mollom. To enable people to personalize their mashups with icons, we used ImageCache and a CCK Image field.

Customization

Getting lots of lists of states plus data entered wasn't easy, and required some of us to comb through data.gov and other sources looking for state-level data to include. To facilitate this and make the combinations less database-intensive, Norman built out a module to take a CSV file in State Name, Value format, create a table for that data set in the database and store the data there, and then retrieve the appropriate table for its associated node, combine them with the appropriate operator, and provide an array. I wrote a module to take the table data from Norman's module, compute the color needed for each state, and format it in the XML schema required by the Flash-based map (which, among other things, required the states be in a fixed order).

Datamasher: selecting an operator

To make it easier for people to create mashups, I themed the checkbox form element to use JQuery and formatted options to select the operator, which was the most confusing part of the default interface. Now users simply click on the operator with which they'd like to combine the two datasets, and see their choice in a very large font. Norman themed the mashup content type template, using Drupal 6's templating features to provide a tab for the map view and a tab for the table view.

We decided people would still like to look at maps and tables for each dataset, so we adapted the modules to provide the same display for those. And to deal with the inevitable carping over statistical rigor, we added a little surprise. To let people more easily add future datasets, Norman created a user data set module with a custom form that displayed an entry for each state--all the user has to do is fill in each value and some metadata. Add in views for most recent, highest rated, and most discussed mashups and datasets, a view of latest comments, an instructional video, and then some dusting off of my Photoshop cutup and theming skills, and voila, in essentially three weekends we'd built a data visualization tool.

Vote for Datamasher!

If you like it, please vote for Datamasher, by registering and voting at Sunlight Labs via that link. Today is the last day!

Apps for America 2: Lessons Learned During Speedy Application Development!

nospam@example.com (Michaela Hackner) — Fri, 28 Aug 2009 14:01:19 -0400

Since many of us are involved in campaign-focused work that requires rapid turn-around, I wanted to share some lessons from a recent internal project we delivered in 4 weeks for the "Apps for America 2" contest.

(Our application, DataMasher, placed in the top 3 (out of 47), so by they way, I'd appreciate your vote so we can be pushed to #1.)

First some background: The contest was created by Sunlight Labs to celebrate and publicize data.gov, a new resource for federal government data. We were given 11 weeks to work on it, and we actually completed the entire application in 4 weeks from kickoff to launch, in less than 200 hours. We spent a few weeks brainstorming in June, but the bulk of our work happened during the last 4 weeks. At times we weren't sure we would finish in time, but we really believed in our idea to make government data usable, visual, and accessible. So, we persevered!

Here's are a few lessons we captured that should be applicable to similar projects:

1. Keep the team small. Our team was only 5 people - a Senior Advisor, a Project Manager, a Tech Lead, a Developer, and an Information Architect/Designer. This focused the scope and increased our overall productivity.

2. Choose a few pieces of key functionality and ROCK them! Although the possibilities were/are endless with DataMasher.org, we tempered our expectations and were practical with our decisions. We wanted to ensure that the features we built into the site worked really well and that we'd have time to test and tweak. We also wanted to ensure that the site would be flexible enough for growth in the future. There were times we had to turn down great ideas because we weren't sure we'd nail them in the time we had left, but we kept a running tally of enhancements to develop in the future, and continue to seek out that feedback from users (including you!).

3. Leverage user-generated content whenever possible and appropriate: many heads are better than one! DataMasher is by "the people" and "for the people". We wanted the site to be about the users and their data, and most importantly, the ways in which users could manipulate that data to be relevant. Letting people create their own mashups, rank and rate them, add comments, and upload new data sets was critical to our overall goal.

4. Be flexible and keep moving - don't get caught up in the churn and debate, make decisions! If we had spent a lot of time planning and scoping, we never would have entered into development. We chose basic goals, met weekly to discuss progress and opportunities, and were always willing to both succeed and fail, and most importantly, learn.

The project was a ton of fun to work on, if not a lot of work. I hope these insights are helpful.

Once again, if you wish to vote for DataMasher, here's how.

WordPress: Certain Widgets on Certain Pages

nospam@example.com (Matt Gibbs) — Fri, 10 Jul 2009 10:31:29 -0400

WordPress and Drupal are similar in a lot of ways. They're both extremely popular, powerful, and flexible frameworks. They both have minimal cores, but were built from the ground-up to handle tons of extensibility.

Even though WordPress has been traditionally thought of as a blogging system, it's gradually shaping into a more refined, full-featured CMS. WordPress Plugins like Pods CMS and Flutter are opening the flood gates of potential, just like how the CCK and Views modules have revolutionized Drupal.

Drupal - Blocks

Drupal includes things called "Blocks", which are chunks of code/HTML that you can stick in various "regions" of your site (e.g. Header, Footer, Content Top, etc). With blocks, you can choose which pages these blocks appear on, or which pages to exclude them from appearing on.

WordPress - Widgets

WordPress has similar things called "Widgets". The main thing missing with Widgets as opposed to Blocks is the ability to display specific Widgets on specific pages. Regardless, Widgets are relatively simple to implement.

You first add some PHP code in functions.php to define new "regions" to place widgets into.

if (function_exists('register_sidebar')) {
register_sidebar(array(
'name'=>'My Footer',
'before_widget' => '',
'after_widget' => '',
'before_title' => '

',
'after_title' => '

'
));
}

Then, it's a matter of adding some code to your WP Template file (e.g. index.php) to display this custom Widget region:

WordPress Plugin: Widget Block

I created a plugin so that you can treat WordPress widgets more like Drupal blocks. Specifically, you can choose which pages a certain widget will appear on (or be excluded from). This plugin also prevents you from needing to create many slightly different Widget regions. The plugin is called Widget Block; give it a try!

The Cloud: how is it relevant?

nospam@example.com (Jo Ryden) — Wed, 06 May 2009 14:02:08 -0400

I attended the Nonprofit Technology Conference in San Francisco a couple of weeks ago. As in so many other places these days, both on and offline, a lot of the buzz and talk was centered around The Cloud and Cloud Computing. What is it? What can it do for us? Who uses it? It's easy to get into long discussions about these topics and there are many articles, blog posts and even books covering a wide array of aspects and philosophies of Cloud Computing. I'll try to give a very brief and cursory overview below in an attempt not to cause immediate information overload. If you would like to continue the discussion a bit more in-depth, please feel free to leave a comment or contact me at any time.

What is it?

There are still quite extensive (and sometimes heated...) debates going on regarding how to define The Cloud and services operating from The Cloud. I'm a visual person and I think this video is a good attempt at depicting The Cloud, what it is and where it came from.

A few highlights:

A collection of shared, internet accessible resources or infrastructure evolved from several "as a Service" infrastructure and software delivery models
Can be tapped into on an as-needed basis (resources can be scaled up or down with little to no effort)
Pay for the resources you use, and don't pay for what you don't use (utility type metered billing model)
"Open" philosophy (open meaning access from any device anywhere at a higher level and API access at a lower level)

Some people like to argue that with the evolution of The Cloud, the various "as a Service" concepts have become obsolete. I don't necessarily subscribe to that view. To use a familiar example: Google Apps (GA). GA is what has traditionally been categorized as Software as a Service (SaaS), and I think that is still the most accurate description of what it is. However, since GA runs off The Cloud you could make the argument that it should be called a Cloud [based] Service.

What can it do for us?

Besides the obvious and immediate cost savings associated with life in The Cloud, it also offers a level of flexibility in a number of areas that is previously unparalleled in the computing world:

No more long binding hardware/hosting contracts
Need a staging site for that one new application to show off for an afternoon? No problem, just click a button
Have a high profile campaign driving a lot of traffic to your web site? Deploy some extra resources for the duration of the campaign effect and terminate them at will as soon as your traffic returns to normal levels
Need some compelling arguments when presenting and/or defending your web operations budget? How about: no longer will you be spending on what you might need, you are now spending on what you in actuality need and use

So what's the catch? There really is no catch to speak of. The barrier of entry to The Cloud is very low, cost is exclusively operational as opposed to in previous years where the barrier of entry to high level computing resources has been associated with large upfront capital expenditures.

Who uses it?

Chances are you as an individual or maybe in some business capacity, are already taking advantage of what The Cloud has to offer in some way or another without even knowing about it, which is coincidentally one of the major characteristics of The Cloud; the abstraction of the underlying infrastructure, hardware or software platform. I mentioned Google Apps above, here are some other major players using The Cloud and/or providing access to it and services running from it:

Salesforce.com - provides cloud services
Amazon.com - provides cloud services and cloud infrastructure (which have evolved from an infrastructure that mainly came about due to its own need for an operating platform for its e-commerce business)
Razorfish.com - operates from The Cloud

The distinction between who "uses" the cloud and who provides access to cloud infrastructure and services can sometimes be a little fuzzy since providers are oftentimes middle men and end users as well, and vice versa.

Final words

At Forum One we offer our clients and partners a slice of, and a guided way into, The Cloud and all of the benefits discussed above through our partnership with Rackspace and its Cloud Division. We also have several SaaS offerings (or Cloud Service offerings if you will), our flagship being ProjectSpaces. The Cloud isn't just the future of computing, it is here now and ready for you to put to use!

LegiStalker: Getting people engaged with Congress

nospam@example.com (Matt Gibbs) — Sun, 26 Apr 2009 15:27:09 -0400

There's a lot of people out there wanting to keep up with Congress.

Many people just want to know what their representatives are doing in Congress, how they're voting, and who they're interacting with. Unfortunately, that's pretty hard to find, especially from several different angles.

We at Forum One felt that exposing this information was a pretty good idea, so we ran with it. The final result is the site LegiStalker.

From the low level, it's a pretty simple concept. We pull in all the information we can find about every Congressperson, clean it up, and store it in an ever-growing database. So far, we've managed to harness YouTube, Twitter, latest voting data, and several hundred different news sources. With so many different news sources, there's a pretty good spectrum of differing perspectives -- ranging from CNN and the New York Times to Fox News.

This is a lot of good information, but a lot of information nonetheless. If users are presented with just the latest items retrieved, it would get pretty overwhelming for the user. With that in mind, we came up with a 5-tiered system for filtering the data to meet any user's needs.

1. U.S. Level - a high-level view with the latest feeds from any congress member.

2. State Level - When the user clicks on a state, it first displays all representatives in that state. All the tabs now display feeds relating only to congress members in that state.

3. Zip Code Level - When the user enters their zip code in the box, all their Congress representatives will first appear. This will include the two senators, as well as one or more representatives. All the tabs now display feeds relating only to those people listed.

4. Detail Level - At this level, the congressperson's contact information is displayed. In addition to News, YouTube, and Twitter, that person's most spoken Congress words are displayed in the new "Capitol Words" tab, and there is also a history of that person's latest votes. All the other tabs are filtered to that single person in question.

5. Watch List Level - When users are viewing a congressperson's detail page, they have the option to add that person to their "Watch List". When they do, that person is displayed in the "My Watch List" area at the top right corner of the site. An unlimited number of people can be added to someone's watch list. This allows users to filter content based only on representatives that they want to keep track of.

Adding this much fine-tuning was quite some work, but we feel that it's worth it. Visitors can now keep track of who they want to keep track of, thus making more informed decisions.

Although the Apps for America contest is over, we'll definitely keep adding more features to pry open the doors between congresspeople and their constituents.

Code Samples, Job Applications, and You

nospam@example.com (Sandy Smith) — Mon, 09 Mar 2009 23:16:15 -0400

According to a couple of sources I've checked out recently, code samples are rare in the software development interviewing process. One discussion was full of responses calling the idea ridiculous and suggesting candidates cancel interviews at places that request a code sample. I think they're incredibly wrong. I think code samples are a great idea, and I'm going to not only tell you why, but give you tips on amateur mistakes to avoid if you want to be taken seriously.

You can't tell that much about a person from a code sample

That's right, and that's wrong. It's right in that many commenters calling a company "stupid" for "relying" on code samples assume there's no other criteria to hiring. Code samples are actually one of several initial screening devices I use to cut out people who aren't going to work out. At every stage in the game, I'm looking for something to tell me--before committing a lot of money and diverting a lot of other people's time to bring you on board--whether you're likely to be the sort of programmer we're looking for. This doesn't mean you're a bad programmer if we don't hire you; it just means that who you are and where you are as a programmer doesn't match our culture and our needs, and likely our culture and opportunities won't be a great fit for you, either.

Of course, if I like a code sample, there are still interviews, references, and some discussion of real and example code. We still care very much as to whether you know our technology, can work well on a team, are comfortable working with policy nonprofits, and several other factors that go into a hiring decision.

Then again, I can tell a lot about a person from their code. Are they consistent? Do their variable names follow a pattern? Do they have a coding style and stick to it? Basically, do they write code to be read by others, or assume they'll always know what they meant and code for themselves in the moment? Do they follow contemporary idioms for the programming language the code sample is in? Is the code filled with convoluted ways of solving simple problems? Is there anything clever, and is it in service of the problem or just adding complexity to make the solution more interesting to create? Is the code from an open source project? Have they collaborated with others? Do they follow good habits, even on personal projects? Is it well laid out or look like a quick hack?

But you don't even know it's the person's own code!

Yeah, I do. I mean, not immediately on first glance, but there should be some correspondence between the level of sophistication in the code and the work history on the resume. Rarely have I gotten a sample so perfect I wonder if the programmer wrote it themselves. And even if they stole the code, whether they have the sense to steal a Renoir or a Kinkade also tells me something. I mean, if you can't even steal good code, I can safely pass on to the next application.

If, during an interview, I get an impression that's at odds with the code sample I've seen, I can always ask about it. It's tough to fake familiarity with a piece of code you grabbed off a server somewhere. If the code uses design patterns, I ask about them. If it doesn't, I ask why not. I ask about the tradeoffs in using a given pattern versus not using one. I'll usually ask these sorts of questions anyway, so I get a sense of where the programmer is in thinking about code as well as writing it.

But maybe my code is all proprietary or even classified!

The code you write at work is, but there's no reason you can't write out part of a common solution. We're also an open source shop, so you can always write a plugin or extend an open source app. If you need a few days to get a sample together, I'll understand, but hey, if you're going to apply for programming positions, you shouldn't be surprised if they want to see samples of your work. This is a problem long ago dealt with in other aspects of your profession: strategists will be asked to provide writing samples and designers will be asked for samples of their portfolio. Programming isn't a job you tend to hold onto for decades; you should have a sample ready. I have always had current samples since I became a programmer.

OK, so what should I provide?

Above all, something you're proud of, or at the very least feel is a good representation of where you are as a programmer. This does not mean the cleverest thing you've ever done (frequently the cleverest thing a programmer does is not the best thing a programmer does). It can be a whole program or just a part of one. It should be more than one screenful of code--I need enough to see patterns.

In a way, it's like the various cooking competitions that are all the rage now (go Carla!). These chefs have a body of work and produce pieces of dishes many times each lunch and dinner (or even breakfast). But the finale comes down to whether they produced a great meal that represents them as a creative artist. And, like food, programming is a mix of artistry and engineering.

So...what should I not provide?

Thanks for asking--because I rarely worry about stolen code when I see so much bad stuff given to me as representative of a programmer.

Don't indulge your bad habits

Anybody who gets out of a programming 101 class knows good programmers comment their code. There are arguments about how much or how little, but there's a reason almost every code-completing IDE also generates comment templates for you. We all slack off sometimes (admit it), but the time to do it is not when you're sending me code that's going to help me determine whether or not to bring you in for an interview.

Likewise, giving me code with obvious XSS or SQL injection vulnerabilities is probably not a good idea. Even if you're writing a toy app to demonstrate a few techniques, the fact that you code without basic filtering and validation is going to make me nervous that you'll do it when there's a project manager asking you if there's any way you can add a couple of more client requests by the deadline. Sure, it's the kind of thing non-technical people aren't going to notice or praise you for, but they will curse you when their site around sensible regulations for underwater basket weaving curricula is replaced with pr0n.

Don't be weird

I see some really odd constructions. If you think the rest of the universe is wrong about most things about programming in a given language and you're going to prove them wrong and stun me with your brilliance in a single code sample, you're basically doomed. I'm looking to see if you're aware of best practices going on in the wider world around you (do you read any programming blogs? articles on your language?). I'm also looking to see if you think about problems in a way that makes sense to most people. In our organization at least, everybody is going to work on someone else's code at some point. Wrapping a static HTML block inside a PHP include that contains a Javascript function to print out each line of HTML is not the kind of innovation that I'm looking for. Save it for the obfuscated C competition. Don't laugh; I've seen that.

Don't give me the same code sample you used six years ago

I'm looking to see what you're like now. If you haven't really progressed in six years, then, OK. I've still got the PHP 4 samples lying around. Heck, I've even got the Perl CGI flat-file database I used when I interviewed here. They're probably not my best chance to prove to you that I know what I'm doing now. Likewise, don't give me HTML 4 table-based layouts with nested function calls that assumes register_globals is on.

Remember, all of this--at least in my case--is not to prove you're the most brilliant programmer that ever walked the face of the earth. I'm not actually interested in hiring that person. I'm interested in a really good programmer who works well with others, can be put in front of someone non-technical, and who can easily pick up our idioms and have positive contributions of their own. So don't overthink the code sample, but don't put less thought and care into it than you do how you dress for the interview. Neither will get you the job by themselves, but both of them can either keep you from getting the job you want or make a good impression you can confirm throughout the interview process.

DrupalCon DC

nospam@example.com (Matt Gibbs) — Wed, 04 Mar 2009 07:10:46 -0500

Hey everyone, just a heads up that the whole Tech Team will be attending DrupalCon DC from March 4 - 7. "DrupalCon is the official Drupal unconference that attracts the most talented open source developers and innovative web shops in the world and brings them together to share knowledge, make decisions on the future of Drupal, and get to know each other in person". If you're also attending, feel free to stop by our booth if you want to chat or know more about us.

We hope to see you there!

What is a Content Type?

nospam@example.com (Matt Gibbs) — Tue, 06 Jan 2009 15:42:00 -0500

A "Content Type" is jargon for what is a simple programming concept. I've been asked this question many times over the years, and hopefully this post will cut through the obscurity once and for all.

Document. Event. News Article. Staff Member. These are all examples of Content Types.

Which input fields would you expect to see when adding a new Staff Member? You'd probably want "First Name", "Last Name", "Date of Birth", "Phone Number", "Active Projects", etc. For Events, you'd likely want "Event Title", "Start Date", "End Date", "Location", and "Contact Name". The fields serve to describe the item, and any field(s) can be added to or removed from a content type.

That's it. A content type is the term for a "bucket" of input fields. All items marked as "Events" share the same fields. If you add a new input field to the "Event" content type, then all current (and future) Event items will have that new field, too. Pretty simple, huh?

The real power of content types isn't the categorization alone. Since all this information is now nicely structured within these content types, it's a whole lot easier to form relationships among different items. Remember that "Contact Name" field in the "Event" content type? A developer could easily set that field to pull in all staff members (items belonging to the "Staff Member" content type) and allow the user to select which staff members to use.

Many modern CMS platforms (Drupal, Pods, SyntaxCMS) allow developers to easily add and configure new content types without touching a line of code.

Drupal module and theme naming collisions

nospam@example.com (William Hurley) — Fri, 12 Dec 2008 11:33:58 -0500

As good as Drupal is, as many third-party enhancements there are, there usually comes a time when they just aren't quite enough. The answer to that is to create your own module that does exactly what you or your client needs. And since they are for a particular site the natural inclination is to name them after that site. Similarly when developing a theme for a site the natural inclination is to name the theme for the site. This, unfortunately, is a Very Bad Thing (TM). For why this is let's take a quick look at where the configuration for modules and themes ends up, the system table.

A quick export shows us the following:



CREATE TABLE `system` (

  `filename` varchar(255) NOT NULL default '',

  `name` varchar(255) NOT NULL default '',

  `type` varchar(255) NOT NULL default '',

  `description` varchar(255) NOT NULL default '',

  `status` int(11) NOT NULL default '0',

  `throttle` tinyint(4) NOT NULL default '0',

  `bootstrap` int(11) NOT NULL default '0',

  `schema_version` smallint(6) NOT NULL default '-1',

  `weight` int(11) NOT NULL default '0',

  PRIMARY KEY  (`filename`),

  KEY `weight` (`weight`)

)

The important rows are filename, which holds the file included for the module or the stylesheet displayed for the theme; name, which holds the name of the module or theme; type which is either module or theme; and description which holds the description of the module or the path to the templating engine for themes. While I could take a slight detour to talk about why this structure is a bad idea (hint, it involves using the same column to mean two different things -- bad, Drupal, bad), it only partly contributes to our problem.

When the theme and module have the same name and a user goes to the theme administrative section it updates all rows in the system table with the name of the theme to have the correct filename and description/templating engine. It, however, does not check for the type. So when it finds the module with the same name it overwrites it. So you end up with a row that has a type of module but has the filename of a CSS file, and when the system goes to include all modules that are active it grabs the CSS and you end up with it output at the very top of the page. Oh, and any functionality that your module provided no longer works.

So the moral of the story is don't name themes and modules the same.