SysAdmin's Journey

Case Study in Migrating XML to Drupal using Migrate

Justin Ellison — Thu, 15 Jul 2010 02:43:32 +0000

Sorry for the lack of posts as of late -- a massive upgrade operation at $DAYJOB has had me out of commission for a few weeks. Also, I've had the great fortune to be able to be part of a migration to Drupal which exposed me to migrate and friends. Yes, I said "great fortune" in the same sentence with "migration" without using a negative - that's just how awesome this module is.

My first impression when looking at the documentation for migrate was that it didn't seem complete. While it's true that the documentation could be better (what module couldn't use better documentation?), the problem is that no two migrations are alike. Because of this, the best documentation is not going to be written by the module authors, it will be written by the module users - they are the ones that come up with the recipies to fill the cookbook. There are several good reasons why there aren't many recipes available:

Developers don't like doing migrations. It can be painful, and often takes quite a bit of time.
Users don't like migrations. They see a migration of data as something easily done, and they often get sticker shock when presented with estimates for a large migration.
Migration code is written in a flurry before the site is active. Right before launch, development crescendos, and then is often never used again (because no two migrations are the same).

This being my first migration, I vowed that I would document my experience, because I learned so much from it. In this particular migration, we had to migrate a huge XML file into about 2,200 nodes in 3 content types. Read on for my contribution to the cookbook!

My Thoughts and Ramblings on DrupalConSF 2010

Justin Ellison — Sat, 24 Apr 2010 02:15:34 +0000

I had the great pleasure of attending my first DrupalCon this week. Held in downtown San Francisco at the Moscone Center, it was my opinion that this was Drupal's "homecoming". While Drupal wasn't "born" in San Francisco, it seems to be the city that has the strongest following. The attendance numbers didn't lie - I'm pretty sure they broke 3,000 ~~geeks~~ attendees. I made this trip solo -- I only knew three people that were going, and those three were only acquaintances I'd met via email/IM a few months before. When I left, I didn't come home with "leads" or "contacts", I came home with friends and role models, many of whom I plan on staying in touch with. I met most of the authors of the Drupal books I've read, associated faces to the podcasts and RSS feeds I subscribe to, and I even had the opportunity to quickly say thanks to Dries and shake his hand.

For those who didn't know, archive.org has made the sessions available for download, so be sure to check those out. Read on for my "takeaways" from DCSF2010.

2010: What a Year!

Justin Ellison — Tue, 20 Apr 2010 14:28:49 +0000

If you would have told me 5 years ago that:

Sun would be gone
I would take personal time off from work to attend DrupalCon (and not regret it)
I would attend a Microsoft party

I would have told you where to shove it. Well:

For all those wondering, I went to the party as a saboteur on a mission: to drink as much as I could to try and directly impact their bottom line. Mission accomplished :)

Good times @ DCSF2010!

Ask SAJ: Any SA's at DrupalConSF?

Justin Ellison — Sun, 18 Apr 2010 15:50:05 +0000

It snuck up on me so fast, I forgot to ask - are any of my SysAdmin's Journey readers going to San Franciso for DrupalCon? If so, contact me, and we'll see if we can meet up for a beer! I'm posting this from DIA, so if you happen to be here and want to grab a bite to eat, ping me. I'm flying out to SF at about 12:30pm MDT.

New Linode StackScript: Pantheon Mercury (High Performance Drupal in 10 Minutes or Less)

Justin Ellison — Tue, 13 Apr 2010 01:03:03 +0000

For those who might not know, Pantheon Mercury is:

... a drop-in replacement for your Drupal website hosting service that delivers break-through performance. Mercury can serve two-hundred times more pages per second and generate pages three times faster than standard hosting services.

Mercury achieves this by using open-source technologies like so many ingredients of a complex dish - a little Varnish here, a dash of Memcached there, a hint of the Alternative PHP Cache, a healthy dose of Tomcat and Solr, all based upon the Pressflow distribution of Drupal. None of it is anything you couldn't do yourself -- many before Chapter Three had done it actually. However, they were the first to tie it all together using BCFG2, and release an Amazon EC2 AMI image of it.

As word spread, many liked the idea of Mercury, but wanted to brew their own non-EC2 instance. While they posted a wiki article on how to do it yourself, they went to work on native support for RackSpace. When I read Josh Koenig's post on the Linode blog stating he wanted to bring Mercury to Linode, I made a mental note. Some time passed, I became much more involved in Drupal, and I decided to volunteer to write the StackScript. Josh said okay, and put me in touch with Greg Coit, their resident sysadmin, and we went to work.

Fast forward a couple weeks, and we've announced a beta! The StackScript is quite complete - it supports Ubuntu Jaunty and Karmic, and can use the current stable branch or the soon-to-be-released 1.1 development branch. Once Lucid is released, we'll test to make sure it works there as well.

I want to thank Greg for all his help. We found some bugs in Ubuntu, some quirks in the memcached init script, and fixed many bugs and added some features to their BCFG2 bazaar repo. Thanks also go out to Josh for his oversight and guidance. It was a great time, a great learning experience, and I came out of it with some new colleagues (and some free beers at DrupalConSF).

Feel free to read up on my experiences with Linode, and if you like what you see, click on one of the many links to Linode from my blog. If you sign up and stay a customer for 90 days (trust me, you will), I'll get $20 credited to my account. Feel free to comment below about the StackScript and let me know about any issues you might find.

Tracking Drupal's Outbound HTTP Requests using tcpdump

Justin Ellison — Thu, 11 Mar 2010 17:42:32 +0000

While working on tweaking performance for a client, I was able to shave 7 seconds of PHP execution time time off the homepage load. The cause was eventually tracked down to calls out to TinyURL for every node being rendered. The core problem came from the service links module. We were able to fix it by disabling short URL's in the module, but the problem has been addressed in the current pre-release 2.x branch by using caching. We might have ended up discovering this by disabling module after module one at a time, but that would have taken forever.

Drupal StackScript for RH Derivatives on Linode (Instant Drupal!)

Justin Ellison — Thu, 25 Feb 2010 02:40:01 +0000

StackScripts are a relatively new offering from Linode that allow users to build their own installation script by "stacking" previously existing scripts together to build the machine you want. You can keep your StackScript to yourself, or publish it for the world to use. Deploying a distribution with a StackScript takes only about 5 minutes, afterwards you have a fully configured system with applications up and running. Here's a sneak-peek at a my Drupal StackScript for RH Derivatives deployment just before launch:

Performing a CentOS Anaconda-based Install on a Linode for Kickstart, Root LVM and SELinux Features

Justin Ellison — Tue, 23 Feb 2010 02:48:14 +0000

Linode rocks. Seriously, read my review. I was talking to a co-worker (whom I converted to Linode as well) about how I would pay double the amount to keep my Linode now that I know how much I use it. Don't tell them that, they're cheap :) If you find this article helpful (or my article about moving VM's to and from Linode), please consider clicking one of the links in this article to sign up for a Linode - if you sign up for 90 days, I'll get $20 credited to my account.

I was setting up a second Linode that was to be a testing ground for some StackScripts I'm working on. The new Linode will eventually replace my existing one. For whatever reason, the most recent version of CentOS they had available was 5.3. Not a big deal, I can 'yum upgrade' up to 5.4 after installation. Well, after doing so, I found that a lot of features that I wanted had been stripped out. In Linode's defense, it's in their best interest to offer very stripped down images for their customers. The one feature I wanted that I couldn't get enabled was SELinux, and simply installing the packages still wouldn't let me use 'setenforce 1' to get it turned on. My best guess as to why is that the Linode kernel didn't support it, but I honestly didn't troubleshoot it too much. I really wanted root LVM capabilities as well, so I decided that a full-on anaconda based installation was the way to go. Plus, I couldn't find anything in the forums about it, so there was the lure of being the first to do it ;-)

Well, thanks to the flexibility offered by Linode, not only can you do a anaconda-based installation (with optional Kickstart), but you can do so using the GUI over VNC if you're so inclined!

Sysadmin Humor

Justin Ellison — Mon, 22 Feb 2010 15:12:17 +0000

I laughed out loud when I saw this XKCD comic this morning:

Assign Different Values to Different Nodes via One Action in Views Bulk Operations

Justin Ellison — Thu, 04 Feb 2010 20:24:31 +0000

The Views Bulk Operations module (a.k.a. VBO), is a godsend for busy Drupal site administrators. Don't just take my word for it - Lullabot wrote a chapter about it in O'Reilly's Using Drupal, it's included in the Open Atrium Drupal distribution, and it's even used on Drupal.org! Out of the box, VBO does a lot to streamline the things you do everyday, so that you spend less time doing them. A perfect example is bulk content moderation - with a few clicks of the mouse, you can mark a huge amount of comments as spam. You can even enable batch processing with a single click of a mouse so that you can literally do thousands of these without timing out.

VBO was attractive enough that we decided to offload the bulk/batch operations of Node Gallery to VBO. Integration for the most part was surprisingly easy - VBO "speaks" in Drupal Actions, so by writing actions, we were writing integration with VBO.

There's one undocumented case where VBO can be used that was critical for us. Most VBO actions you will find perform one action to a set of nodes, one at a time. Often times, that one action is to set a value of some sort on said nodes. In the case of Node Gallery, we wanted to be able to assign different weight values (used for sorting) to a bunch of nodes. The key here is that we aren't assigning a value of '2' to all selected node's weight, we want to assign a weight of 2 to node #1, 3 to node #2, 8 to node #3, and so on. While not straightforward, it's definitely achievable.