Funkatron

The MicroPHP Follow-up FAQ

2012-02-07T00:00:00-05:00

My previous post, The MicroPHP Manifesto, resulted in much excitement. In between fits of rage and crying, I found some time to answer folks questions, and also discuss the topic on the /dev/hell podcast with my cohost Chris Hartjes. To summarize and address some of the common questions, I felt I should write a small FAQ.

Got a question? Ask me. I’ll add additional entries here as things come up.

So you think full-stack frameworks suck?

No. I think sometimes they’re very appropriate. It depends on your needs: will the pros you get with library/component/framework X outweigh the negatives? If so, it’s probably a good choice. If not, it’s probably not.

You need a large framework to enforce best practices!

Sometimes you do. My experience at FictiveKin has been that our small team is able to work faster, smarter, and more efficiently by minimizing the size of our PHP codebase and removing all unnecessary layers of abstraction. In some cases that meant not doing certain tasks in PHP anymore (almost all HTML generation was moved to the browser). In others, it meant ripping out a bunch of code and replacing it with a simpler solution that required far less boilerplate and replication. We still kept some code that had more dependencies than we’d like because the wins we get with it outweigh the downsides.

I’ve certainly seen situations where choosing a popular full-stack framework is a better idea. As teams get larger, enforcement of coding standards and not doing Dumb Shit becomes harder. Hiring and training engineers is usually easier with popular, full-stack frameworks. On the other hand, we’ve found that devs coming from non-PHP backgrounds liked how quickly they can be productive with simpler libraries and frameworks. Your mileage may vary.

So you’re saying we should write our own framework/libraries/components?

Good God no. There is lots of very good, well-written code out there that’s already solved the problem you’re facing. Most of the time I don’t want to try to solve an issue like oAuth request signing, because it makes my brain hurt and I’d rather focus on building stuff. So, I’ll look for an existing solution that fits my needs first. I sometimes choose to write something from scratch because the existing solutions (that I can find – discovery is a whole other issue) don’t fit well with my existing application structure, or I feel it will introduce more maintenance issues than I’m comfortable with.

You should check out my microframework!

Sure. Generally I think people should work on writing libraries/components, personally. We have plenty of framework choices. But this is PHP, so you have to write your own framework sometime.

Is “X” a microframework?

Long answer: I tend to believe that the reference implementation of “microframework” is Sinatra. Routing, request/response objects, sessions, maybe some hooks for template rendering. Generally I think the inclusion of an ORM is a clear sign of non-micro-ness.

Short answer: I don’t care, really – and you shouldn’t either. If it works for you, awesome.

How do you choose what gets listed in the MicroPHP code collection?

Generally I think about these things:

Does it try to solve one task, or a small set of closely related tasks?
Would it be easy to use with almost any existing code base?
Is the code as short as it can be, while still being clear and easy to follow?

None of these are hard and fast rules, though. I encourage people to share things with me they think others would find useful.

Why do you hate Rush?

I don’t. I like some of their songs, but don’t own any of their work. I also think they’re incredibly smart, talented musicians. My point was to suggest there are other valid approaches, not to reject complexity outright.

The MicroPHP Manifesto

2012-01-03T00:00:00-05:00

Update 2012-01-06:

The manifesto itself and the collection is now at MicroPHP.org
Here’s a Czech translation
The MicroPHP Fallacy
O Manifesto Micro Php E A Macro Discussão.
Meus Dois Centavos Sobre O Microphp Manifesto
The MicroPHP Manifesto – What People Are Overlooking

Update 2012-01-04:

I’ve started a collection of lightweight PHP libraries and frameworks on Gimme Bar [json] [rss]. If you have suggestions, let me know!

The standard line about the history of Punk is that it was a reaction to the excesses of modern rock, particularly progressive rock of the time. The reality is undoubtedly more complex, but I suspect there is some truth to that. Rock n roll did seem to be the realm of Golden Gods in the late 60s and 70s, inaccessible to average folk. The contrast between bands like Rush and Black Flag –– both supposedly playing “rock” –– was extreme.

For fun, let’s take a look at Rush drummer Neil Peart’s drum kit:

Now, here’s Black Flag playing in LA in 1979:

You can fit the entirety of Black Flag in the space of Neil Peart’s drum kit. And they would still play awesome shit and rock you the fuck out.

In the past few years, the PHP Zeitgeist seems like it’s been moving in the Neil Peart direction. Lots of work by lots of smart people is going into complex, verbose solutions. Lots of files, lots of nested directories, and lots of rules. I frequently see PHP libraries/components that look like this:

All that, just to start your application.

It doesn’t mean this approach is bad, per se. But when I see it, I have a visceral negative reaction. My brain screams

FUCK.

THAT.

SHIT.

I can’t do it. I don’t want it. And I don’t think we have to do it this way to do cool things and build awesome stuff.

The approach I’ve been taking lately is to start with as lightweight a foundation as possible, in the form of a “microframework.” A few of these exist for PHP (of course), including Slim, Epiphany, Breeze, Limonade, and others. For additional functionality, I pull in lightweight libraries that help me accomplish only the tasks I need. Clarity and brevity are my top considerations.

My other big consideration is the commitment I make when I use code I didn’t write. Typically I don’t have time to do a full code audit on libraries, so there’s a level of trust that goes with it. And each dependency means more trust. Not just that there aren’t bugs (I expect that if they’re anything like me), but security issues – both whether they exist, and how they will be handled. Will an announcement go out to a mailing list? How long will security fixes be provided that don’t break backwards compatibility? Will I have to upgrade all my dependencies if I upgrade to the next PHP point release? And all of that assumes the author will have the time and motivation to provide prompt fixes. If they don’t, you’ve just added a bunch of technical debt to your codebase.

Finding lightweight libraries that don’t pull in lots of additional code dependencies is much harder than it should be. Mostly I think that’s attributable to PHP devs being more interested in framework-specific development. Some work is being done to make mature frameworks less monolithic, and many devs on Twitter have recommended Symfony components as an option. Unfortunately, I think my definition of “lightweight” is not the same as theirs.

Here’s the cloc output for a git clone of the symfony2 HTTP Kernel component:

Mon Dec 26 19:42:23 EST 2011
coj@PsychoMantis ~/Sites > cloc HttpKernel
      94 text files.
      93 unique files.
      12 files ignored.

http://cloc.sourceforge.net v 1.53  T=0.5 s (164.0 files/s, 18736.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
PHP                             72           1175           3440           4290
Bourne Shell                    10             56            155            252
-------------------------------------------------------------------------------
SUM:                            82           1231           3595           4542
-------------------------------------------------------------------------------

Here’s the same for the Slim framework:

Mon Dec 26 19:42:27 EST 2011
coj@PsychoMantis ~/Sites > cloc Slim
      54 text files.
      51 unique files.
      13 files ignored.

http://cloc.sourceforge.net v 1.53  T=0.5 s (82.0 files/s, 17752.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
PHP                             31            660           4473           3280
Bourne Shell                    10             56            155            252
-------------------------------------------------------------------------------
SUM:                            41            716           4628           3532
-------------------------------------------------------------------------------

and the Epiphany framework:

Mon Dec 26 19:42:30 EST 2011
coj@PsychoMantis ~/Sites > cloc Epiphany
      83 text files.
      70 unique files.
      31 files ignored.

http://cloc.sourceforge.net v 1.53  T=0.5 s (102.0 files/s, 5246.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
PHP                             40            218            309           1632
Bourne Shell                    10             56            155            252
HTML                             1              0              0              1
-------------------------------------------------------------------------------
SUM:                            51            274            464           1885
-------------------------------------------------------------------------------

When there are more files and lines of code in your component than in my entire base framework, I can’t call it “lightweight.”

It doesn’t mean that stuff is bad, in the grand scheme of things. It doesn’t mean it has no value or is the wrong approach for many. But it’s the wrong approach for me, for sure. And I don’t think I am alone.

I don’t want to be the prog rock superstar, writing a pretentious rock opera. I want to play shitty power chords in a punk rock band that plays shows in a VFW lodge with no stage, and leaves you so fucking pumped that you go out and form your own band. That’s the coder I want to be.

I don’t want to be Neil Peart. I want to be Gregg Ginn.

So I wrote this. A “micro PHP manifesto,” if you will. I plan to use it to guide my PHP dev. Maybe you will find it useful as well.

I am a PHP developer

I am not a Zend Framework or Symfony or CakePHP developer
I think PHP is complicated enough

I like building small things

I like building small things with simple purposes
I like to make things that solve problems
I like building small things that work together to solve larger problems

I want less code, not more

I want to write less code, not more
I want to manage less code, not more
I want to support less code, not more
I need to justify every piece of code I add to a project

I like simple, readable code

I want to write code that is easily understood
I want code that is easily verifiable

Building a Tumblelog with Gimme Bar and PHP

2011-11-05T00:00:00-04:00

One of the coolest things about working on Gimme Bar has been the opportunity to build a platform. While most folks interact with our service via the web site, the site is just one application built on top of the Gimme Bar content collection and curation system. Our web site interacts with the system via our HTTP API, which is open to everyone, not just our internal team. That means that anyone can build applications on top of our platform to suit their own needs or interests.

To demonstrate this, I built GimmeMe, a simple tumblelog-type web app that’s powered by Gimme Bar. It uses methods in our API that don’t require authentication, so you don’t need to register with us to use it. And it’s a drop-in, set-one-option-and-go install, so you don’t need to fiddle with a bunch of stuff to use it (but you can if you want).

I have a running demo on my web server at http://funkatron.com/GimmeMe. It shows the 50 most recent pieces of content I’ve grabbed with Gimme Bar.

Installation

To install the web app yourself, you’ll need a web server with:

PHP 5.2.x or above
APC 3.0.8 or above

Install as follows:

Download the source code as a .zip or .tar.gz from the project’s download page
Decompress the files somewhere under your web server’s document root
Edit the $config['username'] value in index.php 4 .Optionally set the $config['gb_addthis_pubid'] value to include AddThis widgets, if you’re into that kind of thing
Load the appropriate URL in your web browser to view the install on your web server

The Code

If you’re a developer type, you’ll probably want to poke around the source code in the GitHub repo. The main work is done by libs/GimmeMe.php, which handles grabbing data from the API, caching it, and rendering the HTML.

In GimmeMe::getAssets(), we load the data using the GET /public/assets/:username method. This returns a JSON object with an array of asset data in the records propert. We decode the response JSON into a PHP object (the $assets variable), and add a couple properties that we’ll use in the HTML template. Then we cache the object in APC, and return it. Next time around, if the cached object has not expired, it will just pull it from APC and return it, so we don’t make more requests to the API than necessary.

The GimmeMe::render() method loops through the assets we retrieved and makes each one into a GimmeAsset object. GimmeAssets are view objects used by the Mustache templating library to encapsulate display logic, so we set up special methods or data massage in there. After we set those up for each asset, the entire data structure is passed to the Mustache library for rendering, using the ‘page’ template defined in templates\templates.php.

Right now I also am including a a simple router class at vendors\Router.php. We’ll need this for stuff that would involve different URLs, like paging back through assets or loading single-asset pages, but I haven’t quite gotten that far.

I’d be remiss in not mentioning http://hereslookingathue.com/, a collection of beautiful imagery collected and organized by Gimme Bar user hereslookingathue. It’s another cool example powered by our public API methods.

Hopefully this gives you a taste of what’s possible with the Gimme Bar API. Most read-only things, like embeddable widgets or reposting tools, would be easy to do with just the public API methods. Beyond that, the vast majority of our system is exposed through the public API, so powerful collection, replication, and organization applications can be built.

If you have questions or comments, please hit us up on Freenode at #gimmebar-api, or email us at api@gimmebar.com.

Photo by rovingI

Stepping Down From Spaz

2011-10-07T00:00:00-04:00

The Spaz project has been an very important part of my life for nearly five years now. It’s been extremely rewarding personally and professionally. I’m proud of what we’ve accomplished. In recent months, though, the stress of running the project has become overwhelming. My ability to meet my responsibilities as a father, husband, and member of the FictiveKin team has suffered. So, I’m stepping away from Spaz, and letting others take the reins.

I’m very pleased to say that Josh Roesslein (@applepie) has agreed to take over as project lead for Spaz. He’s an excellent coder, very experienced with cross-platform JavaScript applications, and committed to open source. I’m confident that Spaz is in good hands with him.

I’m still going to have some involvement with Spaz behind the scenes, but it will be very reduced. I’ll probably code a bit here and there, and will continue to maintain existing resources as needed, but I plan on taking a “Project Lead Emeritus” role from here on out.

To all of the users, supporters, and friends who have made Spaz one of the best experiences of my life: thank you for everything.

photo by Paolo Margari

HP: To Save The webOS Development Community, You Must Open-Source Enyo Now

2011-08-20T00:00:00-04:00

I’ve talked previously about why Enyo needs to be FOSS. I won’t repeat those arguments here, but I’d encourage you to read them again.

Done? Good.

Recent events make this even more important. With HP killing all hardware that runs webOS, and Enyo locked to webOS due to the license¹, there’s no reason for devs to stay. It’s a closed platform with no hardware.

webOS dev wasn’t really sustainable commercially anyway. But now, with no hardware and no roadmap, you’re going to lose your dev community entirely.²

No matter what, you will lose a large portion of your webOS dev community. But here’s why opening Enyo will allow you to hold onto a core during this transitional period:

It will allow current webOS devs to create desktop and mobile applications that will run on webOS with little or no work.
It will likely attract more developers to Enyo, given that JavaScript development community largely abhors closed-source libraries and frameworks
When webOS is up and running with a new hardware partner, these Enyo apps will be able to launch with the new devices

Here’s what you should do

Release Enyo under a liberal open source license (Apache, MIT, etc)
Provide documentation and examples of using Enyo to develop desktop apps (via Titanium Desktop or AIR Desktop) and cross-platform mobile apps (via PhoneGap)
Get your top dev rel folks, like Dave Balmer, talking about Enyo and advocating its use at JS community events

To be clear: the longer you wait, the more you lose. 6 weeks from now, this is a lost cause. A quick, decisive move here will give webOS devs a reason to stay interested in the platform.

Delay, and you lose them all.

photo by swimparallel

Enyo runs great on any Webkit platform. We make builds of Spaz HD as a desktop app using Titanium Desktop, and I use it on a daily basis – much more than I use the TouchPad builds. We’ve also done experimental builds and had success packaging Spaz HD for Android using PhoneGap↩
much of this is because of the incredibly poor wording of the PR coming from HP, but that’s another story↩

Why HP Needs to Make Enyo Open Source and Cross-Platform

2011-05-16T00:00:00-04:00

Yes, I did say “needs.”

Updated with more info on PhoneGap support – Monday; May 16, 2011, 4:55 EDT

Enyo is a great framework, built by people who understand JavaScript and the nature of the JS community. It’s also a cross-platform framework that runs great on most WebKit engines – HP themselves have repeatedly shown it running within the Chrome browser, and anecdotal reports have it running decently on Android and iOS.

However, HP has still not made it clear what the licensing terms will be for Enyo. Will it use a liberal open-source license like Apache or MIT? Will devs be allowed to deploy on other webkit platforms (mobile and desktop)? While hundreds of devs are working with Enyo already, HP still hasn’t addressed this.

Enyo an open source, cross-platform dev framework makes a ton of sense. Here’s why:

1. Mobile devs don’t want to be locked into a single platform

There’s a reason why people are interested in technologies like PhoneGap and Titanium Mobile: because it allows deployment on multiple mobile platforms. The current mobile landscape doesn’t offer an obvious first platform choice, and there’s significant revenue to be made from second and third platforms. Bringing an application to market is significantly faster with a cross-platform technology stack.

webOS particularly can’t afford to be an island. Enyo as a webOS-only tech will be regarded as an interesting framework that’s impractical for their own use. Enyo as a cross-platform tech – probably combined with PhoneGap – can instead be a serious choice for all mobile devs, and will result in more webOS apps being developed.

2. It raises awareness of webOS in the JavaScript community

Most libraries for mobile web/JS mobile app dev don’t take webOS into account. If they do, it’s a token gesture. They don’t because webOS really isn’t on their radar as a platform – despite the appeal of a platform built on webkit, v8 and Node.js. Coming out with a top-tier cross-platform app development framework will immediately raise the awareness of webOS, and provide lots of opportunities to position webOS as the platform for JavaScript app dev.

3. JavaScript is driven by openness and diversity

The thing that JavaScript has thrived upon has been its openness and diversity of platforms. JS runs almost everywhere, and the accessibility of information and libraries is unbeatable. This is because it is an open standard, and driven by open source developers. Openness is in the DNA of JS devs. It is expected. JS devs are leery of non-open tech.

To appeal to JS devs, Enyo needs to embrace the nature of the community. That is open and liberally licensed. Devs will love it and run with it, and come up with stuff HP never imagined, to the benefit of both Enyo and its developers.

4. It expands the market for pay-for tooling and training

With a mature, top-tier cross-platform framework, lots of opportunities open up for commercial tools and training. Being a member of the open source community for 10+ years, I’ve found that most FOSS devs will pay for tools and training. This market is much larger if Enyo is not webOS-only.

5. It demonstrates that HP/Palm “gets” the Web

Under “Why webOS” on the webOS dev site, HP lists “Cross Platform” as one of the major attractions. To quote:

The primary webOS app runtime is built on ubiquitous web technologies, so you can write code that runs with minor modifications on other platforms.

Neutering Enyo with legal restrictions seems at best hypocritical. Worse, HP has shown little commitment to improving the state of webOS support with major JS mobile app technologies like PhoneGap and jQuery Mobile¹. It’s understandable that they’ll put the most time into homegrown tech, but it will be a major missed opportunity if they don’t do everything they can to attract devs interested in cross-platform, web-oriented tech. And webOS can’t afford many more missed opportunities, IMHO.

Update: Dave Balmer at HP reports that Phonegap 0.9.5 contains significant contributions to improve webOS support, and mentions ongoing efforts to aid other major mobile/touch projects. This is very encouraging, and I hope HP gives the Dev Relations team the resources needed to do this right.

6. It doesn’t change what makes webOS special

What sets webOS apart is not just the framework you develop in: it’s the way applications can run simultaneously, interact with one another, and interact with the Web. No other platform does that as well, and nothing in Enyo brings those capabilities to other platforms. webOS is still the platform that truly embraces the open, interconnected Web.

We’re in the middle of building a new version of Spaz on top of Enyo. My hope is that I will be allowed to use it as the single codebase for all versions: desktop, mobile, and tablets. But I guarantee you that Spaz in Enyo will run best on webOS, and take advantage of webOS-specific capabilities.

An open, cross-platform Enyo is a win-win-win: for webOS, for developers, and for users. By not encumbering Enyo with significant legal restrictions, HP will go a long way to ensuring the viability and vitality of webOS as a platform for years to come.

For example, HP apparently was unaware of serious rendering issues in jQuery Mobile until the webOS development community raised a stink about it. It also seem that HP has done little to ensure that PhoneGap runs well on, and is well-documented for, webOS – confusing when the company specifically mentions PhoneGap in their literature.↩

What's wrong with my brain?

2011-05-03T00:00:00-04:00

Warning: this is gonna get whiny, and really rambling.

I’ve been thinking about writing something about my mental illness for a while. I am generally pretty open about it with people I know, but it’s not something I’ve made a huge deal about publicly. However, today something happened that really makes me think I need to address it.

I was kind of having a bad morning. I was having a heck of a time getting an application to do what I want. What application and what I wanted to do are not really relevant. The fact that it put me in such a spiral of frustration and anger is, though. One that led me to throw an embarrassing, childish fit on IRC, aimed at someone who was at worst offering a couple suggestions to help. Now I’m left wondering why, at 35 years old, I still can’t control my emotions properly.

The most challenging thing with writing this is just how hard it is to articulate what happens to me. I definitely think I feel emotions more strongly than normal, no matter what it is: fear, excitement, happiness, anxiety. When it’s at its worst, I’m completely crippled and feel unable to control how I react to things. I have to cut myself off from seeing or reading things that will trigger it. There’s no way I can be productive when I’m like this.

I’ve taken meds for depression since I was 13. Mostly I think they’ve helped; without them, things are harder; darker. When I was 27 or so I was diagnosed with adult ADD, so I take something for that too (Strattera, a non-amphetamine treatment). And in the past couple years I’ve taken Klonopin as needed for anxiety. Mostly that seems to go pretty well.

But some days it doesn’t. Lately it seems like it’s happened more; when I take things ridiculously personally; when I worry far too much about what people think of me. Always had a problem with that; always thinking the worst about it, too, which just leads into more anxiety and fear. All that stress ends up with me lashing out at strangers on the Internet, and colleagues, and my family. None of whom deserve it.

Times like this, I wish more than anything that I hadn’t made Spaz, or started speaking at conferences, or any of that. I wish I just had a normal, stable job where I could hide and do what I should do and just be. I’ve come close to deleting all my source code repos and web site and just killing all of it, because it feels like an unbearable source of stress. If I get so upset and worried that I’m hurting myself and others over it, why do I keep doing it?

And I wonder whether I’m posting this just because I want pity or attention. I suppose I probably do. And that’s pretty sad too.

I just wish my brain didn’t work like this.

Spaz Hackathon at the PHP Community Conference

2011-04-12T00:00:00-04:00

I’m really excited to announce the next Spaz Hackathon, taking place at the PHP Community Conference in Nashville, TN on April 21 from 5:30p-10:00p EDT (tenatively). It will be part of a larger social tech hackathon sponsored by Tropo. Food and drink (non-alcoholic) will be provided. We will have awesome swag provided by the HP webOS dev relations team and Appcelerator, and we’ll be giving away 2 unlocked GSM Pre 2 smartphones to the top two Spaz contributions. It’s super exciting.

As this is a hackathon, the focus will be on code contributions. We have five different projects you can contribute to:

SpazCore – JavaScript, HTML, CSS
Our base set of JavaScript libraries. These power Spaz and other apps (like the Joind.in webOS app).
Spaz Desktop – JavaScript, HTML, CSS
A cross-platform, awesomely expandable and customizable desktop microblogging client. Originally built on Adobe AIR, with current development focused on Appcelerator Titanium Desktop.
Spaz webOS – JavaScript/HTML/CSS
The mobile sibling, available for HP webOS.
Spaz Web APIs – PHP 5.3
RESTful web APIs used by the Spaz clients, built on top of FRAPI. All PHP 5.3-based, ready for easy hacking.
Documentation and end-user support
Less nerdy, but just as important. We keep a knowledge base of data available at help.getspaz.com. We also need people who can volunteer a small amount of time per week to answer user questions.

Note that all the info is at http://j.mp/helpspaz, including links to all of our repos on GitHub and ticket tracking on Lighthouse. If you’re not already boned-up on Git, do so now – we’ll only accept code submissions via pull requests.

If you’re planning on attending, I strongly suggest you sign up for the Spaz-dev mailing list and tell us you’re attending: we want to make sure we have enough stuff for everybody.

If you can’t make it to the conference, we will be on IRC during the hackathon, so people are absolutely welcome to participate that way. We hang out on Freenode in #spaz. Swag and prizes will only be available to in-person attendees, though.

We’re very excited about this latest Spaz Hackathon, and we hope you are too. I’m happy to talk to you about ideas you have on IRC, Identi.ca, Twitter, or email. Hope to see you there!

Migrating from ExpressionEngine to Jekyll

2011-04-02T00:00:00-04:00

I’ve run Funkatron.com on ExpressionEngine for a long time. It’s a strong, flexible, powerful CMS that I recommend to many folks who need to build content-driven sites, especially those where the site will be administered by non-programmers. It’s a very solid choice for consultants and freelancers, who can build the relatively small license cost into their fees – and probably save the client in the long run.

However, over the years, I’ve found ExpressionEngine to be overkill for a blog site. If you’re only maintaining one or two types of content (like blog posts and links), the overhead involved in maintaining the system becomes unwieldy. I’ve long put off upgrading to EE2 because I dreaded having to convert my templates, find replacements for incompatible addons, and finding all the little bugs that seem to pop up in a major version upgrade.

Jekyll is quite the opposite of a system like ExpressionEngine. It’s entirely command-line driven, so you have to be comfortable with the shell and some scripting. It’s really aimed at web programmers who want a simple tool to publish a blog, without worrying about all the overhead of databases, web-based administration, and a monster code base. You have very fine-grained control over how everything works.

The simplicity of Jekyll has been appealing to me since I encountered it when messing around with the GitHub Pages feature. I never ended up doing a lot with GitHub Pages, but the idea of a simple, script-driven blog generator has continued to be very appealing to me.

A few weeks ago, I decided I either needed to bite the bullet and either upgrade to EE2, or switch to another system. I decided to do the latter, and give Jekyll a shot.

This post won’t go into every detail about running a Jekyll site. You should definitely read the docs and get familiar with how it works. I’ll just be covering issues of note particular to my move from EE to Jekyll. I also encourage you to check out my GitHub repo for funkatron.com, which contains almost everything I use for my Jekyll site.

Exporting posts

The biggest challenge was sorting out how to get the data out of ExpressionEngine. Each entry in my blog “weblog” (an EE term for a collection of a content type) consisted of these fields:

title
date
author
categories
slug
summary - rarely used
post body
post body format - a formatter that is applied on output, so you can write posts in textile of markdown and they come out as HTML
extended text - rarely used
keywords/tags - comma-separated

From that, I needed to get my posts into flat files, consisting of some Yaml at the top, and the post content underneath.

The typical approach for exporting data from an existing system is to directly query the DB, but this is a bit more complex with EE. EE has a very, very complex schema that’s basically a case study in why RDBMSes are not suitable for flexible content storage. So yeah, I could dive into that monster, but I was not excited about doing it.

Thankfully, a little Googlin’ turned up a much more sensible solution: Just use EE itself to generate a dump file. Because you can generate just about any kind of content format with EE’s templates, you can let the CMS itself do the dirty work of pulling the post content together, and format it as needed.

I basically followed the approach Robin Fay suggests in her blog. She’s dumping data out to import into Movable Type, though, so I had to do things a little differently.

I made a template group in ExpressionEngine called exp
I created a template in the exp group called comments. Yes, this will output the comments as HTML. More on that later.
I created a template in the exp group called posts_jekyll. Here I am generating the Yaml “front matter” to include some required stuff (like layout and title) and also include some custom properties, like author and old_entry_id. I’m also spitting out category in the Yaml-prescribed way to do an array of items. Some of these custom properties I didn’t end up needing, but I think it’s much better to have more data than less, especially when I wasn’t 100% sure of what I’d need.<p></p> I’m also generating body and extended after the Yaml, without any wrapping HTML – these properties will be output post-formatting filters, so they’ll already be HTML. Finally, if the post has comments, I embed that comments template we created to spit them out as HTML.
Finally, I pulled the data down with curl and into a local file
```
curl http://funkatron.com/exp/posts_jekyll > ee_posts.txt
```

I iterated and refactored this export dump a lot, because I had to be sure it contained all the data I needed. I was dealing with nearly 1000 posts, so fixing them afterwards if I had missed something would have been a huge pain. Depending on your own ExpressionEngine setup, you may have more or less to worry about. Be prepared to experiment.

How to handle comments

Note that in my dump, I just have the comments embedded at the end of the posts as HTML. I did this because:

I didn’t want to lose the comments
I didn’t want to allow commenting anymore

So basically, I wanted to keep what was there for posterity, but I wasn’t going to include commenting ability in the new setup. That made things easier for me. If you want to keep comments, you might look into dumping your comment data into Disqus.

Looking back, I probably could have delineated the comments section better. I’m not sure I’m 100% happy with how I did it, but it’s acceptable.

Splitting up the dump file

So I’ve generated one big text file with all my post data. Jekyll wants each post in its own file, so I needed to sort out how to split the dump file. Anticipating this, I included a bit of HTML at the end of each post to mark the end:

<!-- end of post -->

Jekyll is also rather picky about how the files are names, so I needed to parse the Yaml to generate the filename.

I looked into several bash scripting approaches, but I had trouble adapting them to what I needed. In the end I went with what I know, and wrote a small PHP shell script to split the files as I needed:

This script pulls in the contents of the file, splits it based on my post delimiter, and then iterates over each post in the $split array, pulling out the Yaml data and using preg_match() to grab the date and url_title properties to build the filename. It then writes out the post file with that filename.

Now I had a bunch of Jekyll-format post files. I put these in the _posts directory.

Site layouts

Probably the hardest bit of this process was figuring out the templating system. There aren’t any examples or best practices described in the Jekyll docs, so I ended up futzing around a lot. What helped the most was finding a couple folks who put their entire blog up on GitHub, so I could see exactly how they did it and ~~rip them off~~ borrow their approaches.

Things made a lot more sense when I realized that the templates themselves can have Yaml at the top, and they can use other templates as a “parent.” So in my setup, I have a base layout called default.html, and a post.html which uses default.html as its parent. When post.html is parsed, its content is injected into the default.html template in the placeholder. I also have a page.html layout template for non-post, informational pages (like “About”). So in my _posts files, I put layout: post in the Yaml, and it will use the post.html layout. In my other pages, I use layout: page.

Getting my site looking the way I wanted was a bit of a challenge, because Jekyll will regenerate the entire site every time you republish. It didn’t help that I am using Less, which also requires a conversion step. If I make one change to a template, it will have to process nearly 1000 posts. Making changes was a bit of a chore because of this. Two things helped here:

Use the server feature of Jekyll to get your layout and design stuff sorted locally, before you push it up to your server.
keep only a few posts in your _posts/ directory while refactoring page design. This makes the generation time much more tolerable.
Using tools that let me test design changes on a “live” site. I used CSSEdit a lot for this, and it was very helpful.

Redirecting old URLs to new

Depending on your previous setup, you may need to redirect old URLs to the new ones on your Jekyll site. I do this with an .htaccess file on my server (not maintained in Jekyll, although it could be). It handles more than just stuff for Jekyll, but here’s the pertinent bits for redirecting:

Automating build tasks and deployment

In my Jekyll directory I have a _bin/ folder that contains a few scripts. The most important one is rebuild.sh, which is what I run when I have finished a post and I’m ready to make it live. The file contains some private info so it’s not in my GitHub repo, but I included it here with the juicy bits redacted:

In order, this script:

cds to my local Jekyll site directory
runs a short PHP script to generate a page from an RSS feed
generates the site.css file from the .less file
runs jekyll, which regenerates the site in the _site directory
when jekyll finishes, rsyncs the contents of _site up to the server hosting funkatron.com.

So when I’m happy with this post, I’ll just run _bin/rebuild.sh, and everything is generated and deployed for me. It’s very clean, very reliable, and I know everything that happens. That’s nice.

One warning: DO NOT use rsync’s --delete option if you’re deploying into a directory that has other, non-Jekyll-maintained content. I blew away a bunch of files this way, but thankfully had backups.

Another minor issue is that the RSS feed that is used to generate items.html updates way more often than I rebuild the blog, so it’s woefully out of date after a day or so. It would probably be better to generate this page in particular with its own cron job, but I haven’t been bothered to address this yet. Folks can just subscribe to the RSS feed if they’re really compelled.

I consider the move from ExpressionEngine to Jekyll for my blog a pretty big success. It removed a lot of unnecessary overhead and maintenance tasks. It also kinda reminded me that publishing content doesn’t have to be a complex process. You don’t have to make an application just to publish stuff online. You just need some HTML pages, and maybe a few scripts to make the tedious stuff a bit easier.

GitHub Repo for Funkatron.com

Questions for Spaz Project contributors about communication mediums

2011-03-26T00:00:00-04:00

If you’re a Spaz Project contributor, or have thought about becoming one, I’d really appreciate it if you’d fill out this very short survey. Thanks!!

Links for 2010-12-15 [del.icio.us]

2010-12-16T00:00:00-08:00

audio.js

Links for 2010-11-30 [del.icio.us]

2010-12-01T00:00:00-08:00

A successful Git branching model » nvie.com

Links for 2010-11-20 [del.icio.us]

2010-11-21T00:00:00-08:00

A TextMate Bundle for Lithium

Links for 2010-11-19 [del.icio.us]

2010-11-20T00:00:00-08:00

Higher Order Javascript
"In the presentation I asked for a show of hands from people who’d done higher ordered programming before. A scattering of hands went up. Then I asked who’d used jQuery’s $.each(function {...}) and most hands in the room went up. Which means that most of the people in the room had used higher order programming and never realised it."
CoffeeScript

Links for 2010-11-18 [del.icio.us]

2010-11-19T00:00:00-08:00

How to Install Android on an iPhone in Six Easy Steps

Links for 2010-11-12 [del.icio.us]

2010-11-13T00:00:00-08:00

MathJax | Beautiful math in all browsers
Plupload - A tool for uploading files using Flash, Silverlight, Google Gears, HTML5 or Browserplus
"Allows you to upload files using HTML5 Gears, Silverlight, Flash, BrowserPlus or normal forms, providing some unique features such as upload progress, image resizing and chunked uploads."

Links for 2010-10-28 [del.icio.us]

2010-10-29T00:00:00-07:00

Knockout : Home