psychedeli.ca

introducing active copy

26 Jan 2014

If you've ever browsed the source code of this blog, you might have noticed a directory called lib/active_copy. Opening up that directory is somewhat interesting, as you'll be able to see exactly how I take Markdown files with YAML front matter and render them through ActionView. But unfortunately, unless you fork this repo and hack it to bits, you were unable to use ActiveCopy in your own Rails projects.

Until today.

I've released ActiveCopy to the world as a RubyGem and Rails plugin. While not mountable, ActiveCopy requires Rails, and especially ActionView, for all functionality to work properly.

so...what is it?

ActiveCopy is a replacement for ActiveRecord, but instead of being backed by a SQL database, it is backed by files which use a specific kind of markup format. ActiveCopy was born out of the necessity (or rather, desire) to "wrap Jekyll in a Rails app", giving one the benefits of Jekyll's static file template compilation and simple formatting, as well as Rails' deployment and testing toolchain. Additionally, the use of Rails allows me to experiment a bit with the features of this site, a prospect that would not be possible (in Ruby, anyway) with Jekyll.

get it working

First, add this to Gemfile:

gem 'active_copy'

Install the bundle:

$ bundle install

Now you can generate an ActiveCopy model:

$ rails generate copy article name title

You'll get a free generator that will generate files for you:

$ rails generate article the-id-of-my-article

ActiveCopy supports date prefixing, to do so, run:

$ rails generate copy article name title --datestamp

when you declare the generator for your model.

more information

You can view the complete documentation if you wish to get more involved with how the code works. But the README on GitHub is probably what you want to look at, as it contains a complete setup guide and usage information for the gem.

irssi? in MY notifications center?

8 Sep 2013

As I've said before, I'm constantly working in the shell and have found many ways to improve my flow since I began working this way. I also like to keep in contact on IRC to give out help on Ruby, JS or Rails (and constantly learn in the process), chat with friends, or discuss open-source projects. I use irssi to connect to Freenode, a shell client, and I wondered if it would be possible to get Notifications Center to listen in on my chats and let me know when I get new messages. Thanks to alloy's powerful terminal-notifier, I can!

Like I said, I run irssi through my own VPS which is running an irssi-proxy to keep me connected all the time. This way, connections to chat are speedy and efficient, and people can contact me whenever they want...I'll see it when I get online. I've also begun to wrestle with a project called [bitlbee][bitl] that connects to XMPP chats, so I can talk with my co-workers with my IRC client...but I haven't been able to get that up and running...maybe soon...

Anyway, this post is about getting Mac OS Notifications for your irssi mentions and private messages.

requirements

terminal-notifier
irssi
trigger.pl, an irssi script
Mac OS X 10.7 or greater
The Homebrew package manager, or you can opt to compile irssi/TN from source.

installation

First, install the necessary packages with Homebrew:

$ brew install irssi terminal-notifier

Make sure your scripts directory is in place:

$ mkdir -p ~/.irssi/scripts/autorun

Then, install the trigger.pl script to your irssi config directory.

$ curl 'http://scripts.irssi.org/scripts/trigger.pl' -o ~/.irssi/scripts/autorun/trigger.pl

Now you can open up irssi and type /LOAD trigger to enable triggers!

The triggers I used for building this feature are below. Just change the -regexp pattern to your nick instead of 'tubbo' to get it working right:

-privmsgs -command 'exec /usr/local/bin/terminal-notifier -message "$\M" -title "$\N"'
-publics -regexp 'tubbo' -command 'exec /usr/local/bin/terminal-notifier -message "$\M" -title "$\N ($\C)"'

You should be able to just paste those lines into ~/.irssi/triggers (a file you create), then run /TRIGGER LOAD from irssi to get the triggers working.

And that should do it! Get one of your friends to /msg you or say your name in a channel!

one small caveat

Currently, all spaces are escaped in the notification. Makes it a bit harder to read, but it still works. If anyone out there knows of a great solution, hit me up on Twitter (@tubbo) and let me know about it!

the shape of web uis to come

30 Aug 2013

Recently, I've been very interested in the development of the Ember.js framework, which has grown rapidly in the 3 years or so that I've known it. We've seen great apps being made with (most of) it, and it's gone from something that left me truly puzzled into an understandable and almost indispensable tool in my web development arsenal. While developing a Backbone app, I took a second look at Ember after they released their new routing API, and fell in love with how little code I wrote while developing a simple application. But the way that Ember worked flies in the face with one of my other favorite tools...

NOTE: This post is written from the perspective of a Ruby on Rails developer, using ember-rails to generate my content and ActiveModel::Serializers for constructing my HTTP API.

ember and coffeescript

While I have been writing JavaScript for about 14 years as of the date this article was written, I still prefer CoffeeScript as a DSL for defining my objects. It's cleaner, easier to read, and generates compliant JavaScript that (without too many hang-ups) is very close to or exactly what I would have written by hand anyway. Unfortunately, some of the unusual conventions in Ember, especially related to adding computed properties and observers to methods, make working with CoffeeScript quite difficult. Not to mention that the class/extends keywords do not work reliably in conjunction with Ember.Object.extend().

emberscript to the rescue

Thankfully, Gordon Hempton has created EmberScript, a CoffeeScript-style languaage that compiles to JavaScript. It is a subset of CoffeeScript (you can avoid the special syntax altogether and it compiles to the exact same code as CoffeeScript), but it compiles directly to JavaScript so you're not dealing with trans-compiling issues.

EmberScript adds a number of syntactical features to CoffeeScript that make it well-suited for writing Ember apps. For example, in Ember.js you must retrieve data using Ember.Object.get() (all Ember objects are derived from Ember.Object, by the way):

var App.Person = DS.Model.extend({
  firstName: DS.attr('string'),
  lastName: DS.attr('string'),
});

$(function() { // wouldn't really do this, just pseudocode for "runtime"
  var person = App.Person.find(1);
  alert(person.get('firstName')+ ' ' + person.get('lastName'));
});

But in EmberScript, you can write that code like this:

class App.Person extends DS.Model
  firstName: DS.attr 'string'
  lastName: DS.attr 'string'

$ -> # wouldn't really do this, just pseudocode for "runtime"
  person = App.Person.find(1)
  alert "#{person.firstName} #{person.lastName}"

Notice how I used class/extends to describe the model. This is a visible effect of the fact that EmberScript does not compile to CoffeeScript, because it defines class/extends in a different way.

You can do even more with EmberScript. Let's say I wanted to make the text in the alert dialog a method on Person, because I wanted to use it elsewhere. In JavaScript, I'd have to define it this way:

var App.Person = DS.Model.extend({
  firstName: DS.attr('string'),
  lastName: DS.attr('string'),
  name: function() {
    return this.get('firstName') + ' ' + this.get('lastName');
  }.property('firstName', 'lastName')
});

$(function() { // alright i guess i don't have to tell you again..
  var person = App.Person.find(1);
  alert(person.get('name'));
});

That's a little better, but I feel like we can do a lot more with less code. It's a pretty verbose way to say "combine firstName and lastName in the return value of this method", we have to specify all of the "dependencies" of the method at the bottom and then we still have to use this.get in order to retrieve the attributes. Once again, EmberScript comes to the rescue:

class App.Person extends DS.Model
  firstName: DS.attr 'string'
  lastName: DS.attr 'string'
  name: ~> "#{@firstName} #{@lastName}"

$ -> # blah blah apologizing comment
  person = App.Person.find(1)
  alert person.name

Wow, that's more like it! We define the method just like CoffeeScript, except with a special ~> operator that defines functions wrapped in the .property() method. Also something that's missing is the dependencies declared in the arguments to property(). That's because EmberScript automatically defers dependencies for computed properties. Now that's what I call progress.

There's a lot more you can do. Check out the site for everything. Some other highlights:

Directives for observers (+observer)
The mixin keyword for defining mixins
The with suffix of extends that allow you to compose your Ember objects using mixins

In my opinion, the combination of Ember.js and EmberScript is already a major productivity boost. But we can go even further.

moving past handlebars

Ember uses Handlebars.js as a templating language for "sprucing up" your markup. They're mostly static templates, but they do have some dynamic qualities like basic conditionals and loops for visualizing collections of data. Handlebars is great, but much like Ruby has Haml, someone's gone along and coded a replacement for Ember called Emblem.js.

Emblem is unlike a lot of other templating languages, because it actually compiles down to Handlebars, which is used as the static templating language on the client side. Like EmberScript, Emblem is somewhat of a DSL for writing Ember apps, except Emblem takes care of the templating layer. Since it compiles to Handlebars, it's possible to use both custom Handlebars helpers and the ones you use from a 3rd-party (or Ember itself).

in the wild

The best example I can give is the one I'm currently working on, the official site of [wax poetic records][waxpoetic]. It's built entirely with Ember/EmberScript/Emblem.js, with a light backend written in Ruby on Rails. It also uses the excellent [Zurb Foundation][foundation] CSS framework, also in use on this blog. Check it out, and if you see anything wrong, [file an issue][waxpoetic-issues]. I've made the [source code][waxpoetic-source] available on GitHub if you're interested in seeing how an app like this can be written.

one more thing..

I haven't been able to get EmberScript working with Rails 4, but that's a project I'm going to tackle this weekend. Also, it doesn't seem as though therubyracer is supported, so uhh...install Node.js. :)

pv: a shell interface for pivotal tracker

29 Mar 2013

At eLocal, we use the lovely Pivotal Tracker to track our work progress. It has a really great interface for viewing stories, and some additional features like tasks, rich text comments, and the ability to "predict" future iterations by using the velocity and "points" system (which is totally arbitrary). Another great feature is the story states, which are also arbitrary, but we use them for very specific purposes...for example, Delivered means the story has been placed on the stage server and is ready for testing, while Finished just means that I've pushed the code to master branch and it can be deployed at any time.

But enough about our workflow, I want to talk about a tool I wrote that makes it easier for you to see upcoming stories in a git log-style format, totally within the shell...

introducing pv

pv is a command-line tool that views and edits the Pivotal Tracker stories that have been assigned to you in the My Work pane. It's scoped to just your work, and pv was definitely designed from the perspective of developers working on a project, not project managers who are managing those developers. My opinion is that Pivotal Tracker's UI was designed primarily for people like that, so this shell tool is simply a different way of seeing that, geared more towards developers who don't need to see the scope of the whole project every time they want to check up on their stories.

The goal behind pv is to mirror the functionality of pivotaltracker.com in a manner that is suitable for "worker bee" developers who need only be concerned with their work. We are slowly working towards that, with a roadmap that includes commenting, viewing tasks in a nice bulleted list (in pv show), and more...

viewing stories

The two most useful features are pv log (aliased to pv for convenience), which shows your entire "My Work" pane:

...and pv show $STORY_ID, which shows more details about a single Pivotal story:

From these two commands, as well as pv open (which just runs open on the story ID you pass in, opening the URL for it in a browser), you can see practically every aspect of the story from within the shell environment.

editing stories

You can also modify stories using pv. When I combine it with git-tracker, I don't really have to ever visit pivotaltracker.com at any point during my work day, unless I just want a better interface, or I need to click links (better to do that in a web browser than in the shell, IMO). If you want to modify a story's state outside of a commit message, pass the state and story ID to pv like so:

pv finish 123456

You can pass any state here, like start, finish, deliver, accept and reject, as the first command to change the state of your story. However, at this point it's best to just pv open the story to change details like its title, tasks, description, or to leave comments.

per-project configuration

Check out pv's README for more information on how to configure, but basically you can opt to always use the same config in ~/.pv, or you can place a .pv file in each of your repos to change the account or project ID that pv uses to make API calls.

install that shit if you want to know more

writing jquery plugins with coffeescript

25 Mar 2013

At the job, we're constantly trying to improve the UX of eLocal.com by making the site behave harder, better, faster, stronger. For some reason, recently we've had a surge in these kind of feature requests from the "peanut gallery" of sales and operations staff who are the primary source of feedback for the application. Most of what I do every day is solving bugs or adding new features to this big monolithic app, which powers the vast majority of our business. It is used by the staff on a daily basis to accomplish all sorts of tasks, from an in-house CRM to an advanced billing system (complete with recurring charges and invoicing), as well as basic control of our paying customers' ZIP code ads and lead dissemination.

Recently, I've been trying to codify and refactor the way I write JavaScript code. Using CoffeeScript has been a major advantage, as it makes the code much more readable for the way I develop my jQuery plugins. As a proponent of the object-oriented plugin design pattern, CoffeeScript has always been in my radar for doing this kind of thing, and now that I've learned enough about the language I think it's now possible to codify how I've been writing my JavaScript code, so the next guy doesn't have to write as much.

how we write our javascript components

All of our components are instantiated with the [jQuery][jq] framework. If you're dealing with the DOM, there's just no substitute. jQuery handles a lot of boilerplate we'd have to build ourselves, and is cross-browser to boot! The major reason we're talking about jQuery here, however, is its minimal plugin architecture.

A simple jQuery plugin written in CoffeeScript might look like:

jQuery.fn.viewAllButton = ->
  @each (i, element) ->
    $(element).on 'click', (event) ->
      container = $(this)
      button = container.find('button')
      form = container.closest('form')
      per_page = button.attr('data-per-page')

      form.find('input[name="per_page"]').val per_page
      form.find('input[type="submit"]').click()

Pretty easy to understand. For every element selected by this plugin, bind a click action that submits the form without its per_page param sent over, effectively returning all pages of all content. Most of the details are just for the specific implementation on eLocal's admin system.

But once plugins become complected to a further degree than this, the functional model employed by the previous example begins to break down. Having a lot of functions in your scope (that's any scope, not just the global one) in JavaScript makes your code harder to read, and eventually harder to debug. It is at this point when jQuery is no longer enough to write effective, efficient and most of all easy to debug JavaScript components. You need some bigger guns.

Thankfully, [CoffeeScript][cs] is here to save the day. CoffeeScript "fixes" the absolutely disgusting syntax that JavaScript provides to build objects. If you can imagine trying to stuff OOP into the concept of a function, you basically got what JavaScript is doing. Using a relatively common keyword, class, CoffeeScript gets the job done by sticking to JavaScript best practices and, at the most simple level, builds JavaScript object definitions in much the same way that I used to by hand.

Here's an example of a CoffeeScript plugin that was built to solve a simple UX problem when dealing with Rails fields_for associated forms:

jQuery.fn.undoFieldsFor = (pluginOptions) ->
  @each (i, element) -> new UndoFieldsFor($(element), pluginOptions)

class UndoFieldsFor
  defaults:
    form: 'form'
    item: '.line-item'

  # Assign the `<form>` and `<input>` buttons that we're working with
  # to properly "undo" changes made to the line items in a form.
  constructor: (@element, options) ->
    @options = _.extend @defaults, options
    @form = $(@options.form)
    @item = @element.closest(@options.item)
    @initialize()

  # Bind a removal event onClick 
  initialize: ->
    @item.find('input[type=checkbox]').hide()
    @element.on 'click', @removeItem

  removeItem: (event) =>
    event.stopPropagation() and event.preventDefault()
    @item.remove()
    @_disableSubmission() if @form.find(@options.item).length == 0

  _disableSubmission: ->
    submit = @form.find 'input[type="submit"]'

    submit.removeAttr "value"
    submit.val "Disabled"
    submit.addClass "inactive"
    submit.attr "disabled", true

Basically, it replaces the _destroy checkbox on every additional item with an "UNDO" button. When this button is clicked, the surrounding line item element is destroyed and Rails will not submit that data as part of the form, because it's no longer on the page. The plugin takes advantage of the way Rails associations work, as long as you have an index number higher than that of its previous index, you're good to go. So that _destroy checkbox is really just there for compatibility, as clients without JavaScript will not be able to take advantage of this nice feature.

You can see just in this short example how easy it is to understand a slightly more complex plugin like undoFieldsFor. For each selected item that jQuery.undoFieldsFor() is applied to, the _destroy checkbox is hidden and replaced with an "Undo" button. The click event on this button (actually an <a> link) is bound to a JS action that destroys the current .line-item but also disables form submission in the event that no line items are in the DOM at the time the event is fired.

sumatra tastes exactly how it smells...delicious

You can already see a lot of boilerplate going on in the above CoffeeScript plugin example. For starters, I always have to do that little code on line 1 which sets up the jQuery plugin and tells it to instantiate a helper class every time. Also, because UndoFieldsFor is an object, I need to construct it with an $(element) jQuery object and the options hash passed into the plugin at time of instantiation. Wouldn't it be sweet if I didn't have to do this every time? Wouldn't it be nice if I could use an interface that allowed practically every part of the object building process to be customized, but in most conventional setups, would default to some sane settings that could be used for quickly writing jQuery plugins in a beautiful way.

Here's the same plugin implemented using the JavaScript framework/library I made to codify this technique, [sumatra][sum]:

sumatra 'undoFieldsFor', ->
  class UndoFieldsFor extends SumatraPlugin
    action: 'click'
    defaults:
      form: 'form'
      item: '.line-item'

    # Set up the <form>, .line-item and submit button we'll
    # be working with, and hide the checkbox from view.
    initialize: ->
      @form = $(@options.form)
      @submit = @form.find 'input[type="submit"]'
      @item = @element.closest(@options.item)
      @item.find('input[type=checkbox]').hide()

    # Remove the surrounding .line-item
    perform: (event) =>
      event.stopPropagation() and event.preventDefault()
      @item.remove()
      @_disableSubmission() unless @_lineItemsExist()

    # If no more items are left, disable the submit button on this form.
    _disableSubmission: ->
      @submit.removeAttr "value"
      @submit.val "Disabled"
      @submit.addClass "inactive"
      @submit.attr "disabled", true

    # Check if any line items still exist in the form.
    _lineItemsExist: ->
      @form.find(@options.item).length

By comparing these two bits of code, you can clearly see how we've abstracted some common functionality and increased the readability of our code. I have yet to find a JavaScript problem that I can't apply this solution to. As long as I have to create a new plugin, this always seems like the best way to do it. Let's break down what I'm doing here..

sumatra() is a globally-available function that performs that first line of jQuery from the control example. Since it's written in CoffeeScript, this function also returns the object defined on its last line, which is typically the implementation of the plugin itself. You can name the plugin here, defining how you will call it via jQuery, and that code you pass in as a function is called as the body of the jQuery plugin.

sumatra 'undoFieldsFor', ->

Now we're defining a new class which extends from SumatraPlugin. The object it extends from has some [really awesome time-saving features][sp], but it is purely optional and has no bearing on the plugin you actually make.

  class UndoFieldsFor extends SumatraPlugin
    action: 'click'
    defaults:
      form: 'form'
      item: '.line-item'

In the class itself, we define an action which is the event that we're listening for, and when fired will cause perform() to execute. This perform() method is another thing we have to define in the class itself, as it is the hard-set event handler for the plugin.

Additionally, Sumatra gives us a place to define default option values for the options hash that can be passed into any Sumatra plugin. You can set up this hash in defaults, which is merged with the passed in options to build the public @options property.

    initialize: ->
      @form = $(@options.form)
      @submit = @form.find 'input[type="submit"]'
      @item = @element.closest(@options.item)
      @item.find('input[type=checkbox]').hide()

While the constructor sets us up with some sane, minimal defaults, the initialize() method is for code that would normally go in the constructor. It is bad practice to override SumatraPlugin's constructor, since initialize() provides everything you need to get the class support up and running. Here, we're just caching some "instance" variables to some other elements that UndoFieldsFor needs to know about, like the <form> surrounding all of this, and the <input type="submit"> button that is used to submit the form. Additionally, we have the ability to set the item class in case of a conflict, so we also cache the item property here and delete its corresponding checkbox.

    perform: (event) =>
      event.stopPropagation() and event.preventDefault()
      @item.remove()
      @_disableSubmission() unless @_lineItemsExist()

Now it comes time to actually define what this plugin is doing. Notice that here, unlike the other plugin, it's much less "baroque", and basically defines the control flow of th event handler, instead of its logic. Using Sumatra directly influenced the design of this plugin, because now that we have a trusted framework set up to handle all of the boilerplate crap (that I make EVERY TIME), we can go ahead and abstract more of, if not all of, the actual logic into methods.

Since we've already defined this.item, we can just remove() it as it's just a jQuery DOM object. And the line above that, preventing the event from bubbling up and firing the default browser action, is also standard for plugin/event authoring. That leaves us with the two custom private methods, this._disableSubmission() and this._lineItemsExist(). As you can see, we denote private methods with a preceding _ in the method name. There are no private methods in JavaScript, so this convention is how we solve for that. Note that the distinction between public and private is purely semantic, and has no bearing on the logic of the code. They're to show you that the methods were not written to be called externally on this object.

Let's take a look at _disableSubmission():

    _disableSubmission: ->
      @submit.removeAttr "value"
      @submit.val "Disabled"
      @submit.addClass "inactive"
      @submit.attr "disabled", true

We already know about this.submit because it was saved up in the initialize method. So basically, all this method does is disable submission until a new line item is added. But this method is only called when another method returns, true:

    _lineItemsExist: ->
      @form.find(@options.item).length

Here's a method that offers no real code line reduction, but increases readability and visibility in the perform() method. This is the kind of thing that Sumatra makes it easy for you to do, add methods that increase readability and should enable a higher understanding of the code out of its readers.

doing it yourself because you're hardcore

Sumatra is 121 lines of CoffeeScript and its concepts aren't very hard to implement from scratch in the first place. Realistically, the most complex part is the sumatra() function, which uses a bit of meta-programming to get the job done and allow you to choose any class name as your service object. Other than that, I'm just giving you an "interface" prototype object to build your plugin off of, with very limited resources. This keeps the codebase light and easy to read, and lets you take care of the important things.

when you SHOULDN'T use sumatra

Sumatra's great for small jQuery plugins, but doesn't offer much more than traditional jQuery for building and organizing huge, JavaScript-centric applications. For that, you should use a more complete framework like [Ember][ejs], [Backbone][bjs], or [Angular][ajs], which provide more support and an MVC style for designing your code base for both efficiency and readability. Sumatra is best used as an addendum to mostly-HTML apps which desire little bits of Ajax functionality or fun animations.

check it out!

I'm interested to hear what others think of this, and I always at least read over any contribution made by pull request to http://github.com/tubbo/sumatra. That's where the codebase for Sumatra is stored. To install it, you can use [bower][bwr]:

bower install sumatra

Or, you can use the handy-dandy [sumatra-rails][smr] gem that I made to use Sumatra within Rails' "asset pipeline". For efficiency and consistency, each Sumatra package pushed to Bower is correspondingly pushed to sumatra-rails, so simply by updating your Gemfile you can start using the latest edition of Sumatra:

gem 'sumatra-rails'

The breakdown:

http://github.com/tubbo/sumatra is where the basic JS/CS is located
http://github.com/tubbo/sumatra-rails is where you can find the Rails helper engine.

I hope this was at the very least entertaining, and at best helped you learn a little more about the way we do things here at eLocal. A short disclaimer: we do not use sumatra in production (yet!), but I thought it'd be a great way to show you guys our conventions as sumatra is somewhat of a codification of those techniques.

a case for scopes

8 Mar 2013

There has been a lot of hate towards scopes recently. I'm here to tell you that scopes aren't all bad, and they can be used to create some astonishingly elegant model definitions.

Consider the following bit of code:

# Events fired by Sendgrid that tell us more information about Messages
# we send out on a daily basis. We only respond to a number of
# +EVENT_TYPES+, but they tend to tell us everything we need to know
# about the messages sent out at this time.
class MessageEvent < ActiveRecord::Base
  # The types of Sendgrid events we will be responding to. We are mostly
  # saving events that tell us when a problem has occurred with message
  # delivery, but we do have a 'delivered' event in there so we can keep
  # track of sunny-day message delivery. +View source+ below to see the
  # events we're tracking.
  EVENT_TYPES = %w(bounce delivered dropped spamreport unsubscribe)

  # 'Bad' events trigger a red message icon on the front-end. The
  # bad events include when a message bounced, was dropped, the user
  # unsubscribed to the list, or they marked the message as spam.
  #
  # Only one recipient has to mark a message as 'bad' in some way
  # for it to be considered bad by the system.
  BAD_EVENTS = %w(bounce dropped spamreport unsubscribe)

  # 'Good' events trigger a green message icon on the front-end, and
  # a message is considered good when all recipients fire back with
  # 'delivered' events.
  GOOD_EVENTS = %w(delivered)

  validates :event, presence: true, inclusion: { in: EVENT_TYPES }

  scope :recent, -> { order 'created_at DESC'  }
  scope :bad,    -> { where event: BAD_EVENTS  }
  scope :good,   -> { where event: GOOD_EVENTS }

As you can see, this is a relatively clear model definition. It only consists of code pertaining to the data model, or configuring the validations made just before insertion into a database. We have a number of constants defined, as well as some scopes...but no class methods.

I've avoided these for a reason. I don't like the way they look.

Now this is an obviously stupid reason to cut out code that makes your code more understandable in the long run, and if it's a case where you're sliming in lambdas and trying to make your code look "pretty" even though it's surrounded by a million formal brackets, you're probably not using scopes correctly. But as you can see above, writing those definitions in any other way is not only way more verbose than it needs to be, but is unnecessarily adding lines and decreasing readability usually for the sake of some other purpose.

Typically, people who preach against the use of scopes tell me that scopes rob you of the ability to document methods. Let's look at the model if I used class methods, and give them some extra documentation.

# Events fired by Sendgrid that tell us more information about Messages
# we send out on a daily basis. We only respond to a number of
# +EVENT_TYPES+, but they tend to tell us everything we need to know
# about the messages sent out at this time.
class MessageEvent < ActiveRecord::Base
  # The types of Sendgrid events we will be responding to. We are mostly
  # saving events that tell us when a problem has occurred with message
  # delivery, but we do have a 'delivered' event in there so we can keep
  # track of sunny-day message delivery. +View source+ below to see the
  # events we're tracking.
  EVENT_TYPES = %w(bounce delivered dropped spamreport unsubscribe)

  # 'Bad' events trigger a red message icon on the front-end. The
  # bad events include when a message bounced, was dropped, the user
  # unsubscribed to the list, or they marked the message as spam.
  #
  # Only one recipient has to mark a message as 'bad' in some way
  # for it to be considered bad by the system.
  BAD_EVENTS = %w(bounce delivered dropped spamreport unsubscribe)

  # 'Good' events trigger a green message icon on the front-end, and
  # a message is considered good when all recipients fire back with
  # 'delivered' events.
  GOOD_EVENTS = %w(delivered)

  validates :event, presence: true, inclusion: { in: EVENT_TYPES }

  # Order events by the date they were received.
  def self.recent
    order 'created_at DESC'
  end

  # Find all events where the 'event' is included in +BAD_EVENTS+.
  def self.bad
    where event: BAD_EVENTS
  end

  # Find all events where the 'event' is included in +GOOD_EVENTS+.
  def self.good
    where event: GOOD_EVENTS
  end

My belief is in this class, I've added about 9 lines of useless code and useless documentation. As you can see, I've clearly documented the constants EVENT_TYPES, BAD_EVENTS and GOOD_EVENTS, because this is what data really matters. As we're in a Rails 3 ActiveRecord environment for this case, it's pretty common to chain scopes together to form larger queries when you need them, and so far nothing functionally has changed with this example. But my point here is I've added about 9 lines and that really hasn't improved the readability of my model.

In fact, I argue that it has decreased the readability in this case. When I had my scopes set up, it was 3 lines and with the exception of the lambda surrounding each query clause, there was not a single bit of "baroque" code (e.g., syntax that has nothing to do with readability, but is necessary for the compiler to understand what's going on). Given you're a reader of the source code, as most professional Ruby developers tend to be, you'd see the scopes clearly laid out and most likely if you needed them, they'd be easy to understand. The documentation for the scope sort-of "writes itself", in this case, so there's really no need to over-document those class methods which hopefully won't be used by most people outside of the class.

As MessageEvent is a support model for Message, we're assuming that most interaction with events comes through the Messages interface, and since there's no need to design a crazy search system for events at this time (hopefully ever), I'm pretty confident that the "scope" (hah!) of this particular problem won't be changing anytime soon.

So just to wrap it up, I feel confident that those 3 lines of scope macros are just as effective as writing the methods as actual class methods, and furthermore, increases the readability of the entire class definition. This does come at the expense of lacking documentation for methods that are public and do exist on your class (breaking some rule I'm sure), but my arguement is that this is not always needed.

Use your own judgement. If you feel like your scope is getting too complex, or you're writing a scope with the express purpose of having others/yourself use it outside of the domain of this model (or its original design), you might want to give yourself the freedom of documenting it further by turning it into a class method. Regardless, it's up to you, and I'm glad that Rails gives me the option to make my code more elegant, because it has caused me to attempt to simplify my design patterns and streamline my model definitions to only deal with the data model itself.

don't make this simple mistake...

1 Mar 2013

install bundler-1.3 before installing Ruby 2.0!

So as the Ruby community has been unanimously updating to version 2.0 in the past few weeks, I decided to check out some of the newest goodies available to us developers now that stable versions of Ruby, Rails, RubyGems and Bundler are all available.

If you, like me, are installing this through ruby-build, be aware that ruby-build will not install or update Bundler for you!

This is somewhat problematic because Bundler (if it's installed) has to be version 1.3 or above for RubyGems to install. Otherwise, you'll get this when you try to run any RubyGem (after installing through Bundler which actually works fine)..

There was an error in your Gemfile, and Bundler cannot continue.

Go try and install Bundler. Don't worry, I'll wait:

♬  gem install bundler --pre
Fetching: bundler-1.3.0.gem (100%)
ERROR:  While executing gem ... (Gem::Package::PathError)
    installing into parent path /Users/necromancer/Sites/api.elocal.com/vendor/gems/gems/bundler-1.3.0/.gitignore of vendor/gems/gems/bundler-1.3.0 is not allowed

This will now happen for every RubyGem you try to install.

the fix

What you need to do here is clear out the .ruby-version file so your app isn't running off Ruby 2.0 anymore, then install Bundler 1.3 using RubyGems 1.8. Or, you could just downgrade RubyGems to 1.8 by doing

gem update --system=1.8.25

That should downgrade RubyGems to a point that you can just do a gem install bundler to obtain 1.3. Once you begin installing gems with 1.3, everything in Ruby 2.0 will work out again.

why garage is growing

21 Feb 2013

This began as a response on Facebook to a friend of mine and I felt like more people should know my thoughts on why UK Garage is more respected now as a genre than it ever was before. I'm not an expert on UKG, but I've been listening to it for the past 3-4 years (and before that even, I just didn't know it yet..) and been noticing some exciting changes in its projected legitimacy and adoption.

how i discovered garage

Being a DJ at the time, I was first turned on to UKG by Todd "The God" Edwards with his remix of St. Germain - Alabama Blues. Not only is it still one of my favorite UKG tracks, it's also one of my favorite dance music tracks, period. It still gets me moving and shaking and a lot of my best produced music has been influenced, directly, by the sounds and production values I first heard in that tune.

So that was a UKG song, but it was played in a Daft Punk set, so I wasn't aware that it was Garage. Not only that, but pretty much any DJ I played the tune for didn't know it existed and loved the song very much. This begged a fascinating question: Why didn't UK Garage take off in the late 90s?

Now that I've studied UK Garage, Dubstep and House music a little more thoroughly, I have a few suggestions for why this phenomenon occurred.

why we woke up

People finally realized how intricate and awesome the genre was, and grew bored with the repetitive nature of most dance music, instead taking some of the best parts of House, Jungle & Breaks and combining them into a single, funky idiom.
It has the same vibe as deep house, lots of vocals/pianos/melody with little if any aggression. So it works in a deep house set. A lot of the time it's almost impossible to tell deep house from UK Garage, but also...
It has the same vibe as tech house. With all those tuned and filtered drums, Garage works in a Techno or Tech House set, given the correct tracks.
Now that Dubstep is a big-room sound, House needs to compete. UKG was originally designed as a means for House/Techno producers to compete with what was then the new sound in EDM: Hardcore and Jungle. So that's where UKG gets its speed and "oldskool hardcore" sensibilities from: a desire for House to remain relevant with those listening to Jungle. It's sort-of come around again, except "This Time It's Dubstep!" Now, we're actually speeding up House to slow it down. Since 140 is the average Dubstep tempo, UKG works in a Dubstep set.
Artists like MJ Cole, Submerse and DJ Q have been releasing new UKG music that sounds an order of magnitude better, from a technical perspective, than anything that had ever come before it. The UK Garage from the old days sounds...well...old! Even though some of those classics from Nice 'N Ripe and Ice Cream were and are still absolute smash hits, there's just no way you can play "Hurt You So '99" in a club these days without catching some flack. So "big ups" to MJ, Submerse, Q and Disclosure for really showing the "vast unwashed" that UK Garage is alive, well, and most of all, still has a lot to do!

The artists I mentioned above, as well as pretty much anyone who's been rocking the Juno Download UKG/UK Funky section have been sacrificing a lot by continuing in a genre that was famously dissed on one of the most popular ways to learn about electronic music, and definitely not given the respect it deserves.

the merits of uk garage...for another time

I'll go into further detail on why I believe UK Garage is going to eventually, if it hasn't already in some diversion, usurp House and Techno as the world's most popular genres of dance music. I believe that UK Garage has a lot going for it: almost universal adoption potential, great quality tracks, and a tight-knit community from East London complete with its own slang, events and culture. This is all working to make UKG into something huge, something a lot bigger than what we're all observing right now...

But that's for a later post :)

minitest is still wonderful

15 Feb 2013

So after posting my article on how to use assertions in rspec, I was involved in a discussion with @cheapRoc over the discovery that indeed assertions do work within the context of RSpec. To my surprise, it seemed @seattlerb was following at least one of us and overheard our conversation, mistaking my quite hyperbolic statement for a diss on Minitest.

This was not the case. I still love and use Minitest every day. This blog is tested using Minitest, and we use Minitest at work to run a giant test suite that takes over a half-hour to complete (without the smoke tests!). What has happened here is a colossal misunderstanding on my part about what Minitest actually is. Basically, I was under the impression that Test::Unit had been totally and unabashedly replaced by Minitest when Ruby 1.9 was released. While the Test::Unit class names and assertions remained the same, (as they had in Minitest which was concieved of as a drop-in replacement for the slower Test::Unit in Ruby 1.8), it seems that Ruby 1.9's Test::Unit is inspired by, but not exactly Minitest. From what @seattlerb is saying, it seems as though Test::Unit needed to mostly stay compatible with existant Ruby 1.8 programs and so the new Test::Unit was developed to address the problems of the older version, and as a side effect introduced new inconsistencies and problems when migrating between Test::Unit and Minitest.

My blog post was geared towards people who want to make Minitest look just like RSpec, and behave in the same ways, solely because they prefer the syntax of Minitest to the syntax of RSpec. There are definitely times where assertions just make more sense, visually, than matchers. Basically, I don't believe there's a reason to toil with juggling 5 different gems that make your assertion-based testing framework behave just like RSpec. RSpec is a complete package, you install it and it runs with sensible defaults that make your tests look good and easy to read.

At the same time, it's good to know that tools such as Minitest are there for when you don't need all of that extra fluff in your tests. Maybe you're just testing a little shell-script that you wrote in Ruby and you could practically use Test::Unit if you so desired, but maybe you have the ability to load in extra gems so Minitest would definitely improve your life.

lowest common denominator

14 Feb 2013

We hear a lot of talk by musicians and other "insiders" (who are mostly either aficionados or amateur musicians who aren't doing this kind of thing full-time. I would be included in this list) about entropy in our musical lexicon. We hear talk about DJs not spinning vinyl anymore, thereby making DJing even easier than it once was (and more reliable). We hear about DJs not constructing their set in the same way as they used to, with less variation and care taken to the actual sound levels. The fact that there are people who are paid more money in an hour than most of us are paid in a day, and they aren't even really making sure they don't redline the mixer (which actually can cause damage, I've seen it in real life!) or take care to not ruin their fans' ears, is both sad and hilarious at the same time. After all, your fans aren't going to buy your music anymore if they can't hear it...

Anyway, I'm off track. This post is supposed to be positive, to show the average gormandizer that there isn't some big conspiracy going on and the electronic music community needs people like Skrillex, Deadmau5, and even Tiesto to further its stranglehold on our eardrums. Even though these guys might be less skilled than others who get paid/credited for less, we need them as "evangelists" of our community. Basically, I'm telling you not to tell the emperor that he's not wearing clothes.

musical reaganomics

Reagan proposed the "trickle-down theory" of economics, to the instantaneous praise of many white-haired, over-privileged WASP-like creatures in Congress, and the idea came into America's lexicon just as quickly. While the jury's still out on whether it worked for America, the trickle-down theory of economics has definitely made itself known in the music community, as well as most other forms of mass media. Here's a quick example..

Before Deadmau5 and Skrillex toured the US and spread the "gospel" of electronic music to middle-america kids, it was really difficult to get anyone to respond to The Wonder Bars' music. Nobody really knew what to make of it, and it didn't really help that the only places you could hear it were dirty warehouses or small clubs at the time. I understand that even before my college years (where I was introduced to House music and began composing it), it was much more difficult to find quality house music. But that changed after Deadmau5 played the Electric Factory in Philly. It took a couple years, but now we're seeing an absolute explosion in just about everything that has to do with EDM...

There have been 3, count 'em!, THREE new venues in the last 2 years alone that have produced a purely electronic bill every night they're open. This is proof that even in the hardest city to musically make it in America, you can still turn a profit by booking electronic acts.
Attendance at PEX parties has increased exponentially every year. HeartBurn was cancelled this year for the first time in its history because a suitable venue could not be found in time.
If you have a crew of DJs in Philly and a warehouse you have instant success. Period. There's such a small scene here that those of us who just love that warehouse vibe are forced to go out and hear music like Trap and ratchet-y Dubstep that we may not go out to hear on our own. This is actually a really good thing, because it forces people out of their comfort zone(s) and into new musical territory, which effectively increases our overall comprehension levels for different sounds and types of music.

conclusions

So I like Skrillex. I appreciate Deadmau5, even though I think his music is boring. I like them because they benefit me, by Skrillex existing and turning on new audiences to EDM, the "trickle-down" of House-heads that will inevitably discover the music and be drawn into the love of House increases. It's a numbers game, and while the probability is most likely fractions of a percent, multiplying that by 10,000 or even 100,000 people is still a significant fanbase by default. As a concrete example, take a look at my friends Cousin Brian, who are playing shows all the time in other cities like Richmond, Washington DC and Baltimore, but had trouble finding shows larger than a house basement in Philly. I think they have a big enough name now that they don't have this problem as much as when they started, but compare them with Jack Deezl who has literally played the biggest all-ages electronic venue in Philly, and if I'm not mistaken has been producing and performing music for a much shorter time frame than Cousin Brian has. I think this is significant, because even though there was much less "prep" work that Jack had/has to do to get on stage and play his tunes, the fan base that Jack was tapping into is by default larger than the punk community in Philly, and while Cousin Brian is certainly doing wilder, weirder things with the genre (who the hell plays punk with 9th chords?? amazing!), Jack is going to have an easier time promoting himself because a lot of the groundwork has been done for him by the scene that came before.

give me my assert_response back!

11 Feb 2013

Did you just upgrade Ruby on Rails and now all of your assert_response() tests are failing? Simply add the following code to test/test_helper.rb and everything will work again!

ActionDispatch::Assertions::ResponseAssertions.instance_eval do
  # Override so deprecation warnings stop popping up. This uses a proper
  # +assert+ method but retains the functionality of +assert_response+.
  define_method(:assert_response_test) do |expected_status|
    response_status = response.send :status
    assert_equal response_status, expected_status, \
      "Expected response to be <#{expected_status}>, but was <#{response_status}>"
  end
end

Major shout-outs to Rob Di Marco who actually wrote this and figured out that instance_eval was the proper method to use here, not class_eval... :D

update!

We have drastically improved this little snippet and ripped a monkey patch directly from Rails 4. Here's the full code, pasted here in case you want to read it:

use assertions in rspec

11 Feb 2013

tl;dr -- you can do MiniTest::Assertions inside RSpec examples.

A few years ago, RSpec was extracted from the Cucumber project as a separate entity for BDD and testing with natural language in Ruby apps which may not need all of the weight of Cucumber. It gained traction pretty much everywhere, but it's primary evangelists were Rails developers (well, except DHH), and in the Rails community it was commonly used. But two years ago, DHH lit the spark that would become a major debate and one that "divides" Rails developers all over the map. DHH argued that [test_unit is all you will ever need][dhh] for testing a Rails application. And he's basically right, because you can rails generate model {something and then rake test and watch the assertions run.

why don't people use rspec?

RSpec is a pretty strange testing framework to work with if you're used to doing assertions and unit-testing each component of your application with the built-in Test::Unit library. Test::Unit, and it's successor, Minitest, was designed primarily as a unit testing framework at its most basic level. Since it's included in the Ruby standard library, it needs to follow the paradigms of how Ruby classes are designed by doing one thing, and one thing well. You're meant to add other libraries such as Mocha and Turn to give Test::Unit more functionality, or make it easier to read. These libraries all have their quirks, and tests written for basic Test::Unit tend to fail in strange ways when adding these other libraries on top of them, so typically users of Minitest or test_unit won't be keen on installing a lot of "sugary" plugins that could introduce a bug in the test suite.

the problem that rspec solves

If you completely divorce yourself from RSpec's syntax and the framework which surrounds it to provide a killer testing experience, you realize even less reasons to use Minitest. Speed and efficiency aside, RSpec gives you a lot of sensible defaults and configuration options just for modifying how its returning results to you. Don't like colors? Turn 'em off. How about the way it just shows you every test's name? Try --format=dot to make large test suites easier to read. Or better yet, use --format=progress for really big test suites where you need to see the health of all tests but don't need to necessarily see which tests fails immediately in real-time.

Tools like vim-vroom were built for RSpec. Gary Bernhardt's vimrc is designed for RSpec and works best when you're not constantly switching between Minitest and RSpec. It's more reliable to run single tests and single examples with RSpec, because you don't need a special shell alias or not use things like Turn which muck with the actual name of the test_method you developed.

have your cake and eat it too

You can take advantage of these lovely niceties of RSpec, without having to write in that nasty syntax! Since RSpec is just syntactic sugar overtop of Test::Unit, there's nothing stopping you from writing:

it "should pass" do
  assert true, "not true"
end

So why are you screwing around with Test::Unit and Minitest's difficulties and caveats? Just use RSpec. And make your life easier.

how to bang...like a boss

11 Feb 2013

In the Ruby programming language, there exists two characters you are only permitted to use when naming a method: ? and !. These special permissions are designed to allow you to establish a certain level of convention in your method naming, for example, a method ending in "?" in Ruby is mosty likely always going to return a Boolean response of true or false. Not only is this convention not questioned much, but there seems to be very little sensible use out of making a "?" method not return a boolean response.

Unlike the "?" suffix, which has an established and unchallenged purpose within the Ruby community, "!" is a lot less cut and dry. In ActiveRecord, for example, "!" is added to methods like save() and create() when you want an Exception to be thrown in the case where those methods fail to execute properly. It's easy enough to add the "!" to those methods when you need to change the nature of your business logic, or perhaps you wish to capture the Exception so you can raise a new one overtop of that error case.

# app/models/post.rb
class Post < ActiveRecord::Base
  validates_presence_of :title
end

# app/controllers/posts_controller.rb
def create
  model = Model.create! # => throws ActiveRecord::ValidationError
end

def create
  model = Model.create # => just returns 'false'
end

But Ruby, on the other hand, uses "!" to denote when you're mutating the object that you're calling the method on. This is an entirely different use case from Rails, yet this use case also seems to make sense on a syntactic level:

starter = { foo: 'bar' }

newer = starter.merge(baz: 'bat') # => returns a new Hash: { foo: 'bar', baz: 'bat' }

starter.merge!(baz: 'bat') # => merges in { baz: 'bat' } to the starter Hash instance

bang is the yolo convention

At eLocal, we use "!" as a generalized YOLO convention. It means we're about to execute some actions which may fail, or are connecting to an outside resource (such as an API or our mothership site) In either case, the "!" dictates that our app's control flow should stop.

who needs docs when you have ctags

5 Feb 2013

Ever since I watched Aaron Patterson's Peepcode Play-by-Play, I've been absolutely fascinated with the use of CTags to jump around the various codebases you have installed (if you use Ruby, you know how much of a maze it can be to figure out exactly where an error is coming from!). As the name would imply, CTags were created for C programmers dealing with massive codebases that typically had minimal to no documentation. Developed during the first boon of open source distribution, and created by some of the people on the BSD UNIX project (it was originally released with BSD UNIX), it was championed as a quick, easy and light-weight way of moving around a codebase.

required equipment

I've written about my setup in the past, and I created a shell framework for zsh users that implements all of the concepts I'm talking about here. This technique requires that you're editing your code with Vim (and any derivatives thereof) and managing your RubyGems with Bundler

vim for the win

CTags was designed for Vim, and since I'm a Vim user I'm going to mostly talk about Exuberant CTags. If you're an Emacs user, you might have to do some extra customization in your editor, but the concept of CTags is still relevant and in my opinion, still useful for any modern programmer.

bundler-managed gemsets

If you're a Rubyist, this probably won't work as intended unless you manage your application's gems with Bundler and don't use RVM to manage gemsets/rubies. The reason for this is that Bundler stores your gem's code inside your application's root directory (I have it configured to use vendor/gems, but most people use vendor/bundle or something...I think that's the default), and when you generate your tags file, it will only look at code under the given root directory, so if you're only using Bundler to manage your gem dependencies, you'll be able to access not only your application's code, but the code of the entire Rails framework as well as every other gem you're using. From what Michael Papis and Deryl have been saying on IRC, RVM2 will have support for custom gem directories, so you can use RVM and Bundler to manage dependencies in subdirectories of your project, rather than ~/.rvm, which makes it a pain for CTags to function (though I have seen Rake tasks where this becomes a possibility).

let's get it started

To set up CTags, we first need to install Exuberant CTags and get it working with Vim. Thankfully, Exuberant CTags used to be included with Vim, so they compliment each other quite nicely. It's a separate project, now,

OS X USERS!!!

I'm on OS X, so I actually had to rename a built-in binary bundled along with my OS to use Exuberant CTags. OS X comes bundled with Emacs CTags, which are not compatible with Vim (or practically anything else that uses CTags, for that matter). Not sure if people on Linux-based OSes or other flavors of UNIX also have to do this, but before installing Exuberant CTags, I had to do the following command:

$ sudo mv /usr/bin/ctags /usr/bin/etags

This still kept the Emacs binary around, but renames it to something that won't conflict with the ctags binary you're about to install.

On OS X, you can just do the following to install Exuberant CTags:

$ brew install ctags

This will place the binary in /usr/local/bin/ctags. Regardless of your shell setup, I think it's just easier if you rename the Emacs CTags binary and install from Homebrew or some other package manager, if you have to.

I'm not going to tell you how to install [Homebrew][brew] because you should already know how or have it on your system. What are you waiting for???

use the tags, luke

CTags is just as easy to use as it is to install. Go to the root directory of your project and create the tags file:

$ cd ~/Code/very_important_application
$ ctags -R .

Now, open your application code back up. Browse to a method definition, it would be best if this method was one you knew had only one place it was defined. Highlight the method_name in visual mode and hit Ctrl+] to jump to it's tag definition. Did you notice how your buffer snapped to the file where that method was defined? You can also go back to previous definitions of a method by doing Ctrl-[, if the method has been overridden numerous times. This is especially useful in Rails, where lots of abstraction and overriding is taking place, and in such a big codebase it's terribly useful to be able to quickly move between the different method definitions, showing the evolution of the code that you will eventually run as you call that method from a higher level.

It's so much easier to get work done when leaving the editor is only done when you're really done coding or you need to look up something that maybe isn't explained quite well in the documentation, as is the case with many of Rails' lesser-known methods (especially those nice little helpers in the view layer).

keep it ignant

Remember to keep your tags file in your global gitignore file, ~/.gitignore, in order to keep it out of your repos. It's not terribly useful to have in there and generally, each developer is going to have to generate their own tags index every time they download your code anyway, so it's kinda pointless to version control the index.

I hope that this was an informative crash course on installing, building and using CTags in your daily life. It would be even better if CTags makes at least your work life a bit easier to deal with every day.

blogging from scratch

25 Nov 2012

In case you haven't already figured this out, I am a Ruby programmer. It took me a while before I was really comfortable with calling myself a "programmer" when dealing with Ruby, because I did something most people advise against. I learned Ruby by learning Ruby on Rails. So I learned the framework before I learned the language. Generally, this is frowned upon and causes some very bad habits to develop eearly on. However, learning Ruby this way turned out to not be so bad (even though it took me a bit longer to dive in to Ruby itself), and I continue to make the argument that having quick, tangible results and an emphasis on rapid prototyping allowed me to quickly and easily pick up some of the more advanced concepts of not only Ruby, but HTTP and web development in general. As it is my day job, I do enjoy working in Rails, but sometimes its paradigm just doesn't fit in (well) with what I'm trying to do. So I wanted to use something different.

I tried a number of alternatives to simplify the development process, but I wasn't really feeling any of them. Early readers will remember Typo as one alternative I used for quite a while because it was the only one that had XML-RPC connectivity. The main way I blogged back then was through TextMate, so I could simply push the post to the server after I was finished with it, and keep the text copy on my computer as a backup. I gave up attempting to construct an XML-RPC interface to any of the existing blogging platforms, and basically settled on Typo because it already had that in place. Everything else about it sucked though, it still used Rails 2.2 (Rails 3 was out!), Prototype.JS for the JavaScript, and an antiquated themeing framework that just made everything harder to comprehend.

Then, I came across this new thing that Tom Preston-Werner (one of the founders of GitHub) built as the engine that runs GitHub Pages. It's called Jekyll and it's well, awesome. Jekyll merges HTML layouts with Liquid templates and Markdown text content to build a site that's simple and fast. Since it's just static HTML, there's no waiting around for the database to return with the proper data, nor is there any extra time while the data is parsed to be human readable. Everything is "just there". But Jekyll isn't a complete solution. Designed for GitHub pages, a huge userbase, it's of no surprise that the program isn't all that configurable. For example, you can't change the directory where posts and layouts and plugins are read from, nor can you change the location/filename of the YAML config. Until now, that is.

hacking jekyll

I wanted to design my blog's folder structure in a certain way. For starters, I'm already using 3-character folder names for any custom directories that weren't being preprocessed by Jekyll. I wanted the content rendering engine to be just one part of how the entire app operates, as I have also added Sprockets for asset preprocessing and a Rack app or two for some realtime custom content. Using TryStatic, the static content on http://psychedeli.ca can be served on the Rack stack alongside StatusExchange, my status update feed(s) parser. Rack middleware allows me to handle each request that comes in with a set number of classes. So for example, StatusExchange is only meant to respond on requests to /status, and any other requests just gets sent to the next middleware, which in this case is TryStatic. TryStatic will simply read the request and try to find a matching file in the pub/ directory, which is where Jekyll is storing the site.

back to your roots

After attempting this setup, I was poised to find out that "fighting the framework" against all of Jekyll's better judgment was a bad idea. As I continued to experiment more with my blog, Jekyll became simply an obstacle rather than a solution in my quest to make my blog better and more powerful. In addition, my folder structure a few years ago looked very similar to your typical Rails application, with some minor renaming differences. So I decided to swallow my pride and return to the Rails ecosystem.

Doing so provided a number of significant advantages:

It comes with a test suite built-in, so I can not only test the custom Markdown processor I wrote, but also the other libraries that are used to build the content on this site.
Deployment with Capistrano and on Heroku is easier than wrangling static sites to work inside a Rack environment. This was probably the biggest reason, as updating the blog frequently became somewhat of an issue using my cobbled-together Jekyll site on Rack.
Rails has the ability to cache entire pages to HTML, just like how Jekyll compiled my Markdown to static HTML. Using a custom library for parsing out YAML front matter and Markdown content with ActionView, we can implement pretty much the same thing that Jekyll was doing, with the obvious advantage of being inside the Rails stack and having access to the entire ecosystem of tools built for use with this framework.

As I've become more and more comfortable with Rails 3, I've also noticed a huge push towards modularity in the Rails framework. Now that I don't have to include ActiveRecord and other gems I'm not using, Rails becomes a much lighter framework, and still retains its ease of use. And with the new fragment caching and page caching features coming in the new Rails 4 release, I'm confident that my choice will not only replicate any benefits I would've had from Jekyll, it will mitigate future development headaches when creating simple features and experiments for the blog.

getting it done

Using the ActionView template handling system, I simply passed in Markdown as if it was a Rails view at first. However, Jekyll source files are not merely Markdown. They are a combination of YAML front matter and Github-flavored Markdown, which means not only am I going to have to parse the YAML configuration out of this Markdown source, I'm also going to have to implement some sort of syntax highlighting with Pygments in order to replicate the beauty and simplicity of Jekyll.

For starters, I used Redcarpet to parse Markdown. I like Redcarpet the best out of all the Markdown gems for Ruby, particularly because of its extensible rendering architecture, allowing me to (like I just did) override the directive for parsing a block of code and running a syntax highlighter through it. This one is the easy step: tons of guides exist online to use Pygments.rb and a Redcarpet renderer to highlight your code blocks in Markdown.

Next came the problem of the YAML front matter. Rails' templating system is not advanced enough to parse two templating languages in the same file, sending data from one to the model and data from the other as the actual view. Jekyll source files are in fact used for two separate things, and since my blog articles are all written with Jekyll's YAML front matter at the top, I needed a way to parse out that metadata and display it on the page.

So I created a library called ActiveCopy that allowed me to define a subfolder of my view directories called content/. Here, I can place Markdown files with YAML front matter instructions, and expect them to be compiled into metadata-rich articles that looked exactly like (or better than) my Jekyll transformations.

give it to me

ActiveCopy has not yet been extracted from my blog, but it was designed from the ground up with the intention of becoming its own gem someday, since I'd like to use the concepts I built on my own blogging app with future clients and ideas that I may encounter.

making private methods private

24 Nov 2012

So it's a well-known Rubyism that you can actually circumvent private and protected restrictions on instance methods if you simply use the send() method to access them. I wanted to see if it was possible to rewrite send() on a particular class to throw an exception if the method attempting to be accessed was a private method on the class.

Oh yeah, this post is probably not safe for work...

how i did it

I started out with a simple class:

class PrivateParts
  def initialize
    @boobs = "( . )( . )"
    @butts = "(  )(  )"
    @balls = "( )( )"
  end

private
  def peep_show
    @boobs + @butts + @balls
  end
end

In my test, I am asserting that a NoMethodError will be thrown if I attempt to access the method, even with send().

require 'test/unit'

class PrivatePartsTest < Test::Unit::TestCase
  def test_peep_show_cannot_be_called_outside_of_the_class
    parts = PrivateParts.new

    assert_raises(NoMethodError) { parts.peep_show }
    assert_raises(NoMethodError) { parts.send(:peep_show) }
  end
end

This fails, until I overrode PrivateParts.send()...

  def send(method, *args)
    if private_methods.include? method
      raise NoMethodError.new("private method '#{method}' called for #{self}")
    else
      super(method, *args)
    end
  end

What send() is doing here is checking against Object.private_methods, a collection of method names that are all private. If a match is found, an error is thrown, because somebody outside of the class wanted the data returned by PrivateParts.peep_show. Due to Ruby's own clever scoping of the "actual" send method, overriding send() does not affect the classes own internal behavior, as the private method can still be called as a private member within the class. This is illustrated by the following test:

  def test_peep_show_can_be_exposed_by_exposure
    parts = PrivateParts.new

    assert_equal "( . )( . )(  )(  )( )( )", parts.exposure
  end

And accompanying public method PrivateParts.exposure:

  def exposure
    peep_show
  end

These tests both pass, since peep_show is not using PrivateParts.send() to do its bidding. This override, being done in the "public-facing API" of the class, was scoped to just that portion of the codebase. While this overrides Ruby's "code-as-documentation" appraoch and enforces strict private members in a class, it actually takes advantage of such an approach to provide the functionality in a language that does not provide it out-of-the-box.

conclusion

This whole demonstration is available as a Gist. As this is more of a demonstration in meta-programming, if anything, I wouldn't recommend actually using it. This is of course not an example of how I actually write my code. My personal beliefe is that such enforced rules are unnecessary in the real world, and private/protected methods are simply there to tell developers of an intended purpose for the method. It is almost never necessary to call these methods in a test nor in an outside class, so a case such as this should probably never come up, otherwise I think you may have bigger problems on your hands...

simplifying my ruby life

23 Nov 2012

Ruby is all about simplicity. Make simple things simple, right? So how come we have to deal with all this complexity when we run Ruby? This is how I made my life slightly easier and got some really sweet benefits as a result.

how i learned to stop worrying and uninstall rvm

I've been a loyal RVM user for years, practically ever since it came out. When it first started, it was a simple way to manage the different versions of Ruby you had installed. Simple, right? But then, as the project grew its scope became larger. RVM became more about managing your Rubies, it now had integration with all kinds of software and "gemsets", a way to organize each project's gems in their own separate, sandboxed environments that is still an unmatched, all-in-one way to handle dependencies and Ruby version management.

But RVM has its problems. For starters, it can be very annoying to manage paths in the middle of working on a project because either RVM or you fucked something up. Being made of shellscripts, RVM is incredibly brittle and at one point was under such active development that it was easier to just keep up with the Git repo than keep downloading new stable versions every day. RVM was constantly being forked to keep up with its growing list of strange issues that came with the territory of its quite revolutionary/obscure methods of solving the problems of multiple Ruby installs and projects on the same machine.

Secondly, and this is most of the reason why I switched, as RVM increases in size the time my shells take to open up increases dramatically. As they must load the entire RVM codebase each time, my shells take an exorbitantly long time to open, even when my ZSH configs are optimized to an extreme. The real bottleneck is, and always has been, RVM.

As Ruby 1.8 began to sunset, I felt as though my time with RVM was coming to an end. I hesitated for a moment, then typed rm -rf ~/.rvm into iTerm and hit enter.

I didn't see the prompt again for another 20 minutes.

entering the 21st century

With that said, it was time to actually install Ruby. Knowing that I no longer need 1.8, have no use for 2.0, Rubinius or JRuby (right now), and don't want to lock my projects into older versions of Ruby, I installed the language like I install anything else on my Mac, with Homebrew:

brew install ruby

I use the Ruby in /usr/local/bin because I can test against the latest 1.9 for all my apps, and because the Homebrew Ruby is typically quite up-to-date. Using this as your only Ruby does pose a number of problems:

Using 1 directory for all of your gems may prove a bit haphazard, since many of my projects are on older versions of Rails or other gems and require different versions to be installed in different places.
Upgrading to a newer version of Ruby requires reinstalling your whole gem list --local.
If you change your $GEM_HOME, you have to remember to clear it or change it back to access the gems you installed in this "global" path.

bundler to the rescue...

Of all the great Ruby projects out there, Bundler's got my vote for the hands-down best maintained Ruby project. Not only is it mostly error-free, it's fast, beautiful, has sensible defaults, and generally just makes fucking sense to me. Bundler, the reason for Bundler's creation, and the philosophies that went into its design, are what makes Ruby such an amazing language to work with. In my mind, Bundler and RubyGems serve as golden examples for how dependencies and project-specific dependency management should be implemented in a language, and few others even come close to replicating its symbiotic beauty.

While Bundler is somewhat tedious and confusing to use, I've found it perfect for managing my projects gem dependencies on a conventional basis. It solves all of the problems posed from using a single Ruby, since it installs gems locally to ./vendor/gems. This is a sensible default location, in with the rest of my vendored code, and easy to remember and type. I can write PATH=vendor/gems/bin:$PATH in my ZSH config to always have binaries in vendor/gems available to execute. However, this again poses a problem that Bundler cannot solve...

with his trusty sidekick, autoenv!

RubyGems has a variable called $GEM_HOME that is blank by default. If this variable is not set, RubyGems looks in the default location (/usr/local/lib/ruby...) for your installed gems. Otherwise, it looks in the path provided by $GEM_HOME. In order to only use Bundler as a dependency manager, you must manage this shell variable yourself. I use Autoenv to do this for me. Autoenv is a simple shellscript that overrides cd (just like RVM) and runs a file called .env local to the current directory. Right before cding into the dir, the .env file is sourced and the code inside is executed. Typically, these files are used for configuring sensitive data that you may not want in a repo anywhere, but you can also use them to build "pseudo-gemsets" localized to your app's directory. A perk of Autoenv is you can also add other configuration to this file and not worry about Autoenv clashing with RVM, since they both override the same program.

In order to make your bundle recognizable by RubyGems, you have to execute the following line upon entering the directory:

export GEM_HOME='vendor/gems'

Whether you do this with Autoenv or by other means is up to you, but it must be done. $GEM_HOME can not take multiple values, it is used as a prefix and can only be one directory. I found Autoenv to be the perfect companion, especially for Heroku applications. Autoenv has a special convention by which adding a ~/.env file will be executed upon every cd, no matter where the destination. By symlinking ~/vendor/gems to my actual gems directory in /usr/local, I no longer need project-specific files to tell where my gems are located.

that's all folks

Well, that's how I set up my Ruby environment. All of this stuff is pre-configured for you (if you install Bundler and Ruby 1.9, of course) in my ZSH framework called DOTS. I'll do a whole write-up in the future, but to install it all you have to do is run the following line in ZSH:

gem install zsh_dots && dots install

This will create ~/.dots from the Gem dir, link all of the dotfiles in ~/.dots/config, and reload the shell. When your prompt comes back, you should have a ~/.bundle/config and ~/.env all ready to go, along with Autoenv and the rest of my ZSH goodies. Note: Any dotfiles that aren't already synced will NOT be touched, but it will load configuration for any dotfiles you don't have. So keep that in mind.

declaring war on the framework

17 Aug 2012

Most of my time at my current job has been spent refactoring our affiliates API. When I started, it was a series of disorganized Rails apps in a single Git repo, one solely for the backend and meant to push sales leads to our main web application, and the other solely as a frontend "dashboard" to that API, where the affiliates and the folks working in operations can see statistics on and make changes to affiliate accounts. In order to accomplish this, the past programmer had the genius idea of separating out the entire model structure into a gem, which further complicated the matter, leading to strange errors, security flaws, and flat-out unimplemented garbage code.

The Pre-Game Show

Before I began reworking the API, I was told to completely rewrite the email parser engine (from scratch, don't ever fucking do this). It was originally written in C++, but I had to implement in Ruby ostensibly so other developers could help out with adding features down the road. This never happened, of course, because every time there needed to be something added or a bug needed to be fixed, nobody wanted to tackle the problem based on bad experiences in the past with this codebase. It took months of agony before we finally had a stable, working version that didn't crap out all the time or send everyone 100 alert emails a day. For the same reasons as Joel points out in the article I linked, rewriting a codebase from scratch is always going to be painful. Most likely, the bugs and misunderstandings that took me a lot of stress and energy to work out were already taken care of in the C++ version of the parser.

Don't fight the framework, declare war on it.

A lot of the confusion and deployment difficulties we had while working with this API could have probably been averted if the developer who worked on this project previous to me heard this simple phrase: "don't fight the framework".

When working with frameworks such as Rails, which change the way you code in Ruby, your choices become more limited as to how you can (or should) implement something. For example, it's against the ethos of MVC and Rails to perform database saves and complex logic in the view layer of your application. It's very common for developers to ignore this, and simply go on hacking away at something where a much more elegant, test-able and robust solution can be found by simply learning a bit more about how their whole ecosystem works.

I think this doomed strategy of solving small issues has its roots in a larger, more psychological problem...

Being "that guy"

As a musician, I tend to think of my relationship with co-workers on a creative team in similar ways to how I treat the members of my band. As the bandleader, I'm responsible for coordinating rehearsals/gigs and managing the operations of the band as well as conducting performances and sometimes cueing parts to be played. Sometimes, there's a guy in the band who simply doesn't "get it". He can't seem to fit himself in to the sound of the whole band, and blend the sound of his instrument with that of everyone else. And by the way, for the P.C. crowd...I am purposely not representing the female gender, because I rarely meet women who do this. I'm not sure if that's because there are typically more men than women who play jazz instruments, which is the kind of musician I recruit for The Wonder Bars, or if it has something to do with how women approach this problem. But anyw, it's been 100% men who do this in my own personal experience.

"That guy" is not a good guy to be. When push comes to shove, a guy like that can't be in a musical project that relies on the group being tight and having a specific sound. He is typically unaware of his actions, and is typically quite butthurt by his forced departure and hearing of how he's been behaving. In bands which don't cycle through performers, this is an even bigger problem because it is much more expensive to fire someone and replace them than it is to mend a relationship between two band members, from a logistical standpoint. In other words, it's a lot less work on my part if the entire band works with "that guy" to help him blend his sound in. But sometimes, it just doesn't work.

Why am I telling you this? Because the former developer that built this API didn't know how to be a part of a team. He built things that only he knew how to fix. He wrote in languages that no one else on the team understood, or understood to a point that they would feel comfortable writing (or even worse, debugging) an application of some kind.

The big kicker to all this? "That guy" decided, in his infinite wisdom, that email parsing wasn't possible with standard Ruby, and so he utilized C++ and the chilkat library to do all the parsing. While it did run quite fast, no one else on the team before or since knows C++ enough to debug someone else's (probably horrible) codebase. This is the only reason I was introduced to the project in the first place, and it turns out that an email parser is really, really easy to write in Ruby. So whatever bullshit was spewing out of his mouth was just that, and most likely his real motivation for writing it in C++ is just "because he could". We must remember, however, that just because you can, doesn't mean you should. In this case, this motivator for his decision-making cost the company thousands of dollars and backtracked our entire development effort a bit, because I can't work on the main app while I'm banging out bugs, developing new features, or refactoring the API's existing codebase.

Here's an excerpt of his CoffeeScript code that took me two days to debug. For reference, I rewrote this code in about an afternoon:

bt = (f) ->
  w "<div class='elapibutton' id='elapibutton_" + ak + f + "'><a class='elapisubmit' id='" + ak + f + "' href=\"javascript:e_" + f + "_();\">&nbsp;</a></div>"
czti = (f) ->
  w "<img class='elapi' src='' id='" + ak + f + "' style=\"display:none;\">"
imt = (fa, fi) ->
  w "<div class='elapipoweredby'><a class='elapi' href='' id='" + ak + fa + "' target='_blank'><img class='elapi' src='http://<%= request.host %>/images/pixel.gif'></a></div>"
img = (f) ->
  w "<img class='elapi' id='" + ak + f + "' src='' border='0'>"
lmid = (f) ->
  w "<div class='elapi' id='" + ak + f + "'></div>"
anc = (f,v) ->
  w "<a class='elapi' id='" + ak + f + "'>" + v + "</a>"
lti = (f, v) ->
  @zts = -1
  f.attr "src", "http://<%= request.host %>/v1/" + fv(apk) + "/zipvalidate/" + v
rmvc = (f) ->
  fe(f).css "background-color", "#FFFFFF"
pf = (v) ->
  tv = v
  tv.replace(/\(/g, "").replace(/\)/g, "").replace(/-/g, "").replace(/\./g, "").replace " ", ""
vp = (v) ->
  l(v) == 10 and isnm(v)
cpops = ->
  w gops()
gops = ->
  hm = ""
  catid=""
  $ea.each aks.it, ->
    catid = @ci if @k is fv apk
    #catid = aks.it[0].ci if catid is ""
  hm += "<option value='0'>Select Need</option>"
  $ea.each cps.it, ->
    hm += ("<option value='#{@oi}'>#{@n}</option>") if @ci == catid
  return hm

Seriously, who the fuck wants to go through that and figure out, line by line, what it does? All because of an idiotic, non-informed decision to keep everything "terse" for "smaller file size". These kinds of decisions, made entirely inside one programmers head, are signs that the developer in question is not aware of or doesn't understand how to work with a team. Perhaps he didn't feel comfortable with his team, or there were other reasons why he chose to code this way, but in any case they are causing massive problems for me. Because it would take far too much time to go through this code and refactor its potential issues, I can only hope that it continues to work as we continue development on the API.

Before I [touched the code][dtmc], the JavaScript that controlled this was building the HTML form from scratch, and was about 170 lines of code. By not fighting the framework, and rather letting Rails take care of asset packaging and template rendering (you know, the shit it's good at?), I was able to reduce the code footprint to about 35. That means that the code I was working with was ~4x bigger than it could have been. Not only that, but simply making an Ajax call to an HTML endpoint instead of building everything in JS is less buggy, and it allows us to take advantage of the nice server-side view layer stuff like Haml, helper methods, and Rails' ActionView module. Working in Rails is just depressing if you can't use this shit.

Oh, did I mention that all of the static assets previously used in this widget were in app/views/script, and they were all ERb so he could interpolate <%= request.host %> into the code. Yup.

Reversing This Hell

So as I continued to toil on the API, I decided that my ultimate goal on this project is to refactor it so the entire team could simply jump in and begin working. In order to do this, a few things had to happen:

Everything needs to be combined into a single Rails app.
Data from the MongoDB collection(s) that formerly powered the API must be converted to PostgreSQL, with preserved relationships.
Must integrate with Jenkins, our CI server, and our other development tools Airbrake, Pivotal Tracker and Flowdock.
Implement a tag-based release system and deploy based on those tags, so we can better manage API versioning and the user can always see what version of the API we're working off of (with documentation and an affiliate-centric changelog to boot)

The Big Squeeze

We like to follow the "Pattern Of Least Expectation", and since most of our repos contain single Rails or Rack apps, why would we break from that convention for the API? Thankfully, the 3 apps in the repo were themselves Rails apps, or meant to work with a Rails app, so it was simply a tedious process of moving code around and running the test suite. It was quite easy, actually, to move everything into a single app since we used the same model structure (in a Gem) for both apps. Almost as easy as, you know, DOING IT IN THE FUCKING FIRST PLACE.

This was accomplished in a fairly clever way. @iotr and I created a Rake task that looked at the MongoMapper model definitions, found where the attributes were declared, and executed rails generate model for the downcased name of the class. We also use these definitions to generate the temporary table schema, since it also holds crucial database field => type mapping information...

data = %x(for f in `grep -Rl 'MongoMapper::Document' vendor/elocal_affiliates_data/* | egrep '.rb$'`; do echo `basename $f`; echo `egrep '^ *key' $f| awk '{print $2 $3}' | sed 's/[:,]//g'`; done)
count = 270171
model_name = "?????"

# Create temporary tables of Mongo data 
data.split("\n").each do |line|
  # print the last model on each new model name encounter
  if !!line.match(/\.rb/)
    model_name = line.gsub '.rb', ''
  else
    # translate mongo class into attribute type
    line.gsub! /String|ObjectId/, " VARCHAR(255),"
    line.gsub! 'Integer', " INTEGER,"
    line.gsub! 'Float', " DECIMAL,"
    line.gsub! 'DateTime', " TIMESTAMP,"
    line.gsub! 'Boolean', " BOOLEAN,"
    line.gsub! 'user', "user_id"

    %w(description full_body raw_post questions).each { |text_attr|
      line.gsub! "#{text_attr} VARCHAR(255)",  "#{text_attr} TEXT"
    }

    query = "DROP TABLE IF EXISTS tmp_#{model_name.pluralize}; CREATE TABLE tmp_#{model_name.pluralize} ( #{line.chop}, _id VARCHAR(255), created_at TIMESTAMP, updated_at TIMESTAMP );"
    puts query
    postgres.exec query
  end
end

# Generate Rails models
data.split("\n").each do |line|
  # print the last model on each new model name encounter
  if !!line.match(/\.rb/)
    model_name = line.gsub '.rb', ''
  else
    # translate mongo class into attribute type
    line.gsub! /String|ObjectId/, ":string "
    line.gsub! 'Integer', ":integer "
    line.gsub! 'Float', ":float "
    line.gsub! 'DateTime', ":datetime "
    line.gsub! 'Boolean', " :boolean "
    line.gsub! 'user', ":references "

    %w(description full_body raw_post questions).each { |text_attr|
      line.gsub! "#{text_attr}:string",  "#{text_attr}:text"
    }

    sh "rails generate model #{model_name} #{line}"
  end
end

This script will build temporary tables based on every MongoMapper model you have in a certain directory. It will then generate Rails models for each of those MongoMapper models. In short, these two loops are what translates MongoMapper class information into a database schema and model structure. Doing this requires that your MongoMapper models are in a different directory from app/models, so it may be a good idea to mv app/models vendor/models && mkdir -p app/models before running this script. At eLocal, we ran this in a Rake task.

The Data Translation

I'll spare you guys the entire code for the "real import" task, because it is rather long and tedious. We ran a SQL query that INSERTed into the "app table" (that is, the table generated by the model which we will be using in the application) the contents of our temporary tables, which are prefixed with "tmp_". It turns the _id column on all tables into bson_id, and sets up the reference columns for other models by their relational ID. It is important to run these scripts in a specific order, from the least amount (or 0) of associations to the model dependent on the most associations. We wrote similar queries for each model, and ran them all in a Rake task.

Here's the formula for all of the SQL queries which performed this task:

TRUNCATE models; 
INSERT INTO models(name, relationship_id, bson_id, created_at, updated_at)
  SELECT name, tmp_models.name, relation_models.id, _id, COALESCE(tmp_models.created_at, NOW()), COALESCE(tmp_models.updated_at, NOW()) 
    FROM tmp_models 
    LEFT JOIN relation_models ON relation_models.bson_id = tmp_models.relationship_id;

When that task is written, the database will be magically populated with all of the information from Mongo, except instead of tying everything together with the BSON::ObjectId, we are using numerical IDs generated by Postgres. This lets the Mongo database once again "fit in" with our Rails app.

When these Rake tasks are run side by side, they do a complete conversion of the MongoDB collection into a Rails-ready SQL database. Because the API will be live when we're migrating to the new server, we need to make sure we're dropping as little data as possible.

Migrating a Live API

Our API gets a good bit of usage, and what's worse is that we didn't have monitoring tools set up to track when leads were coming through and when they're not (we do now, thanks to @fromonesrc's Graphite/StatsD skillz). So we had to assume a few things about the big API move:

We will probably lose some data in the transition.
There will be clients attempting to connect to the old server when we make the call to move DNS to the new box.
There will be bugs when moving to the new infrastructure.

Putting The Pieces Back Together

To effectively launch the new API, I had to sit with Chef for about 2 and a half weeks and figure that shit out. It's not very simple, but Chef is an incredibly powerful tool for managing server configuration and launching new server instances. With our extensive use of Amazon Web Services, Chef has proven to be a perfect companion in DevOps, like when EC2 instances randomly terminate or an availability zone goes down. But perhaps its most powerful asset is it's introduced us to a whole new realm of knowledge about how all this stuff is being deployed behind-the-scenes. Chef makes us all aware of the way our servers are set up, and may shine some light into potential design flaws or inconsistencies with our network.

With all the complex systems happening here, it's nice to know that someone, somewhere has simplicity in mind. The Amazon Elastic Load Balancer is an incredibly simple means of creating a load balancer, and essentially enables 0-downtime deployments at all times (even in fatal failure situations). Not only that, but the load balancer allows us to experiment with horizontal scaling of our EC2 infrastructure.

So in order to deploy the API, I set up its infrastructure to use a single load balancer and API server. Since we don't get that much API traffic right now, I kept it simple and only put one EC2 instance on the load balancer. With the load balancer and Chef EC2 instance in place, it was time to test the API. To do this, I used /etc/hosts to "trick" my machine into thinking live.api.elocal.com had a DNS configuration that pointed to the load balancer. When I loaded the page locally, I was greeted by the new API, with a version tag number displayed in the footer for good measure.

With Rails' testing framework, Cucumber and RSpec, I was able to effectively make sure that our clients would not notice any sort of downtime or difference when posting their leads to our backend.

They'll Never Know What Hit Them

Losing data is never a good thing. So when moving the API over, we wanted to make sure that ideally 0% of the data gets lost. That's (probably) not going to happen, but it's still a good idea to shoot for that. Our data loss was pretty much negligable because of our deployment speed and the fact that everything was "set in place" before the actual movement happened, so basically all we needed to do was to "flip a switch". But there's still an elephant in the room: DNS.

DNS is a tricky thing. It's never quite certain how long it will take for all of the nameservers to propagate changes across the Internet, and different locations may yield different results. To get around this problem, we used Apache's mod_proxy to silently forward requests from api.elocal.com to the new EC2 load balancer box. Simultaneously, we wrote a new CNAME record for the api subdomain that points to the new load balancer. When both are in place, all API requests, new and old, are forwarded to the new load-balanced API server, which is running the new codebase.

What's Next?

After the widget rewrite, we're going to take a second look at the email parser. We may even implement some kind of Bayesian filtering to increase our chances of the parser mapping legal need to the proper fields in our backend. But perhaps the most integral of all this is to deprecate a direct relationship between the IMAP inbox and the Rails app. By taking advantage of Sendgrid's Parse API, we can simply create an EmailLeadsController to handle incoming emails, which will pass off the Hash to a Mail::Message that the EmailParser can accept.

Conclusions

This process, while kinda scary, was a complete success. In fact, everything was done in the moment, so the least amount of data was lost in transit. Meaning that, all in one big command, we imported Mongo to Postgres, converted the schema to something Rails could understand, and restarted the Nginx server with the new configuration. (Thanks Capistrano) At the same time, the COO updated the DNS configuration to point to the ELB, and the conversion was complete.

All in all, I learned a LOT about how to properly do maintenence on systems while they're still operational, and I learned a bunch of what NOT to do with a Rails app. Namely...

Don't rewrite anything until it becomes a last-resort option (i.e., debugging would take more time than rewriting)
Don't fight the framework. If you're hacking too much at Rails to get your code working, you're probably doing it wrong.
Chef is annoying sometimes, and difficult to debug, but it's an indispensable tool for managing server configuration.
There are companies that actually sell C++ libraries. Libraries! And what's worse, there are companies that BUY these libraries!

meet blue_velvet

2 Aug 2012

use your facebook page as a cms

It's certainly been a while, hasn't it? I've been busy with gigs and vacation, so there's been little time to blog. I have been working on stuff...including the foundation of a record label (which I'll talk about in a future post) and maintaining the Wonder Bars' official website.

In the time it's taken me to maintain that site, I've been quietly developing a solution to a problem I think most small-scale developers have, content duplication. In other words, we've all run into the problem of a company's Facebook page saying one thing and their website another. Who wants to log into 2 different accounts just to paste the same paragraph of text? Furthermore, what if each CMS has its own input format...say one lets you do Markdown but the others force you to be either text-only or HTML-only. Now you're looking at converting content, and for non-programmers this may just not be a reality, or if it is it's a nightmare at best.

Something needs to be done. This is the 21st century, after all. With the power of Ruby on Rails and Rails engines, however, there is a solution.

introducing blue_velvet

blue_velvet is a gem I extracted from working code on TheWonderBars.com. The live production codebase uses the gem with all of its default settings. You probably didn't notice, which was the whole point. The gem code is an exact duplicate of the code that previously powered this functionality on the site, so you're getting what I was working with; a powerful, convention-over-configuration engine for displaying Facebook attributes on your own, hosted, well-designed site.

making it work

Sound fun? It's real easy to get started!

Just add

gem 'blue_velvet'

to your Gemfile, run

bundle

and generate the configuration using

rails g facebook:config

Generate a new Facebook app on http://developers.facebook.com. Enter in your App ID and App Secret keys to the proper YAML attributes, then find the Facebook page you want to look for and look for the number after your page's hyphenated name in the URL. That's your page_id, which you need to put into YAML as well.

Now that you're set up to access a Facebook page, you can build routes to your Facebook page's attributes like so:

(in routes.rb)

get "/about" => 'facebook/page#description'

Now type rails server and visit http://localhost:3000/about, and you should see the content of your page's "Description" field. Since this was designed to be used in a [PJAX][pj]-style environment, the attributes are served as partials.

advanced usage

You can also do some cool stuff on the Ruby side to create better views of your Facebook content. For example, The Wonder Bars' /about page uses the following Facebook page information to get all of its content together:

class FacebookController < Facebook::PageController
  # Accesses 'about', 'description' and 'bio' for a complete "About Us" page.
  def about_us
    @tagline = facebook.about
    @elevator_pitch = facebook.description
    @long_form_bio = facebook.bio
    render layout: false if request.xhr?
  end
end

You can have any controller access that facebook object by simply inheriting from Facebook::PageController. You must cache values from the Facebook page into instance variables for them to become available in the view layer. Any attribute that can be obtained via the Page Graph API can be referenced here in some way. Use the rails console to explore more about the facebook object.

As described above, since the default templates use layout-less partials to serve content, you can create static pages by extending Facebook::PageController and defining custom actions, complete with static views, to serve static Facebook page content on your hosted site.

missing something?

Did you expect blue_velvet to do something that I missed? If so, let me know in the GitHub Issues tracker. Or, you can fork the code and contribute a pull request, I welcome all kinds!

research and development

1 May 2012

I feel like I've been posting a lot about code on this blog. It's mostly what I find myself writing about inside my text editor, but today I feel like I should change the subject a bit. I've been feverishly working on both The Wonder Bars' debut album as well as side projects with some of my friends here in the Philadelphia electronic music scene (in its various forms). One of the most fruitful collaborations has been with The Architech, a DJ I've known about for quite a while but which Philly is just beginning to get introduced to. He was one of the core guys behind the early raves at The Vaudeville Theater, where influential rave DJs like Brak (now Udachi) and Jen Mas performed in what seemed like uninhabitable conditions. This was all before my time as a raver, though, so if you really want to hear the full story you'll just have to come out to one of his shows and ask him yourself. He's a resident at Propaganda and the Bangarang! 4th Fridays monthly parties at The HeadHouse, and frequently plays the legendary PEX parties, including the most recent Heartburn at the Electric Factory.

We've been making tracks together since the start of 2012 as Research and Development. The tracks are harder edged than my work with the Wonder Bars, and that's intentional. I definitely wanted to do something different than what I've been doing with the 'Bars, especially musically. This is a chance for both of us to break out of our shells, and explore some of these less-travelled paths in dance music.

But enough about that. Here's what we've been working on...

Beardyman - Vampire Skank (R&D remix)

An entry for the Beardyman remix competition on JunoDownload, this track was made almost entirely using Beardyman's vocal samples. Our initial goal was to use as much of the sample pack as we could, but we ended up adding some bass stabs and drums to "beef it up" a bit.

I'll Be Your Baby

This was an entry for the Beyonce End of Time remix competition. We had already been working on a remix that had a harder edge and was more "bumpy". But after experimenting one weekend like good little scientists, we came up with this rhythmic, pulsating tune that emphasizes the bass and completely changes the vibe of the original track.

What's next?

We have a few new ones in the works, including AN ORIGINAL as well as a remix of Ross D's Disappear. We're trying some new (old?) tricks in these next couple tracks, so expect them to sound radically different from the music you just heard!

kick out the jams, motherfuckers

30 Apr 2012

Well, I finally broke down and integrated Twitter Bootstrap with the existing UI of my blog. To be fair, it DEFINITELY needed some work, and I just didn't have the time to get into the existing Sass SCSS framework and figure everything out. But as Bootstrap turned 2.0, I found myself longing for a responsive UI (so I wouldn't have to "deal with" mobile), a standard reset, great typography defaults, and nice buttons/form elements to play with should I need them. The answer was simple. I needed Bootstrap.

Bootstrap is written in the LESS framework, which kinda sucks because I prefer SCSS since I work with Rails apps all day. Thankfully, someone made a gem that ports the Bootstrap framework over to Sass for easy modification. Now, if I want to modify the very variables and mixins that power the framework, I can do so. That's really cool.

Another sort-of invisible addition that I made that actually makes all this possible is I am now compiling all assets on the server side automatically using Sprockets, better known as "the asset pipeline" in Rails. It has all the same benefits as the asset pipeline...and all of its shortcomings as well (and then some!). My config.ru has gone through a heavy transformation based on this gist I found for loading Sprockets and Bootstrap onto a non-Rails application. Because I'm using Sprockets to dynamically load stylesheet, JavaScript and image assets, I can develop without recompiling the stylesheets every time. My JavaScript files are minified and compressed before going into production for optimal performance. All of this is happening with the power of Ruby and Rack. Since this static site is being served as a small Rack application, I can take advantage of the stack for logging purposes, compiling static assets, and possibly other things coming soon down the pipeline...

Here's a more detailed version of what was changed...

Bootstrap in Sass provides a number of advantages over the previous
buggy UI. A global, standard reset is in place as well as new
typographic changes and spacing. We're also taking advantage of
Bootstrap-2.0's "responsive" grid, and the site looks great in
both desktop and mobile browsers of practically any size.

I've also completely refactored the aging CSS framework into one
based more on elements rather than roles. It is not complete yet.
There's an additional file called "support" which is responsible
for holding all mixins and variables.

We are also now loading Bootstrap through Sprockets, thanks to a
config.ru I ripped off of https://gist.github.com/2143990.

CHANGELOG

- fixed call to nonexistant task
- refactored assets group
- refactored rackup file for bootstrap-sass
- added assets group to bundler setup
- got rid of :framework and status_exchange in the rackup, using :default now
  for framework gems
- integrated new css framework emphasizing element "classes"
  rather than roles into the markup, which has been modified to
  suit the new responsive bootstrap grid
- removed the octopress folder
- support partial for mixins and variables/colors
- removed static bootstrap CSS
- element based file name and changes to aside

Gotta give some shoutouts @mdo and @fat for building such an awesome CSS framework, Thomas McDonald for porting it to Sass, and @sstephenson for the amazingly useful Sprockets framework. Much love.

weaning off oh my zsh

18 Mar 2012

Since December 2011, I've been working with ZSH, an alternate shell for *nix-based operating systems which has conventions similar to BASH, ZSH adds a number of more modern features to the standard shell fare, such as more customization options for your prompt and extendable autocompletion. To get started on it, I've employed the lovely Oh My ZSH framework which helps to organize your ZSH settings (.zshrc) and use functionality developed by others so you can explore the wonderful possibilities of this awesome shell.

Oh My ZSH was designed for new users, and is meant to be replaced at some point in the future by a .zshrc that handles everything. This was also my eventual goal, but after getting more comfortable with Oh My ZSH, I began realizing that I would be missing out on a number of enhancements it gives my shell experience:

Plugins: A big part of my love for Oh My ZSH is its plugin architecture. With a shitload of premade plugins and the ability to easily add your own, incorporating new programs/functionality into my current shell workflow is trivial. For example, I began using git flow yesterday, a plugin based on Nvie's "A successful git branching model" blog post and after enabling the plugin found that it was incredibly easy to work with. Through subsequent trial and error, this same functionality would have taken days to implement (making small changes/additions here and there), but with the plugin I was able to just start using git-flow using conventional 3-character aliases, just like my interface to git, with the same level of autocompletion I've come to expect from ZSH's powerful engine.
Concurrency: Oh My ZSH has an update function that it fires each week to keep the plugins directory up to date. When new technologies are made available, it doesn't take very long before a plugin for that software is added to the Oh My ZSH repo.
Built-in functionality: In addition to its massive plugin library, Oh My ZSH also comes with a collection of aliases, functions and variables for use in your scripts and everyday use. This useful built-in functionality is stuff I use on an everyday basis for prompt display, general hackery, and other fun shell tasks.

I need a framework for ZSH that can accomplish these goals, but doesn't bother with the theming or tricks designed to enable newbies to use ZSH. As someone mildly comfortable with this shell already, I want something a little more powerful but maybe not so easy to use.

Introducing DOTS

The features of Oh My ZSH I highlighted above make it a possibility to use as a power user of ZSH. But as the aim of the Oh My ZSH project is to help newer users get acquainted with the shell, I've decided to start my own project, forked from Oh My ZSH but without the newbie sugar like themes and installation management tasks. I figure everyone can figure that out on their own. The idea behind this framework is for you to fork it, and maintain that fork with your own "dot files", scripts and other home-dir tools you need for general hackery. Once again, it's meant as a starting point but it strips away unnecessary shit that you're never, ever going to use.

Git it!

So you wanna try it out? First, uninstall Oh My ZSH! (rm -rf ~/.oh-my-zsh) and run this little diddy:

curl -L https://github.com/tubbo/dots/raw/master/tools/install.sh | sh

Persisting your dot files

Not only is DOTS designed to take care of your basic ZSH tasks, it also persists your home directory configuration files ("dot files") across your machines. To invoke this feature, simply type the following in your home directory:

persist .gemrc

(or whatever file you want persisted)

This will move the file to ~/.dots/config/gemrc, add it to the git repository, and symlink ~/.gemrc to ~/.dots/config/gemrc. You can now commit this file into Git and synchronize it across machines when you update dots!

To stop persisting a file, type:

forget .gemrc

This will replace the symlink with the file it's linked to and remove it from the Git repository. You will have to make a commit to confirm this change.

the role of the ajax call

11 Dec 2011

I wanted to touch a little on this topic in public because I feel it's something many Rails developers don't get about JavaScript. This was inspired by an email I got, as well as countless discussions I've had with both my co-workers and fellow programmers I've helped out on IRC. I think a lot of programmers could benefit from this information so you don't run into this issue yourself.

The Ajax request.

It's really nice, isn't it? You can do a lot of things with it, you can even download other JavaScript code asynchronously and execute it. This has proven, however, to be somewhat dangerous. While reviewing my co-workers' CMS code to find out why it was spitting errors out all the time, I noticed that he was constantly passing JavaScript in Ajax requests back and forth and expecting them to execute properly. While at one time they may have, for some reason now they get errors like $ is not defined, even though jquery.js is right there in the Net panel of Firebug. What it boiled down to was that our browser (Firefox) was refusing to allow access to previously loaded libraries on the page to Ajax-requested JavaScript.

This was an example in PHP, but Ruby on Rails also suffers from this potential disaster. Rails developers can create .js files that their views respond Ajax requests to. Typically, these files are JavaScript code segments that are supposed to be executed when they run. But they don't always get executed like that. Sometimes, the browser refuses to load the library, and other times the JavaScript simply fails to run, even though it's in the page. It's as if it was plain text, but not being displayed because it's in a <script> tag. This entire post was inspired by an email I got from a Rails developer attempting to pass JavaScript code and HTML simultaneously in a modal dialog. He was using the Rails JavaScript view system, a now deprecated means of transferring data between the Rails application's controller and helper layers and the JavaScript logic on the view layer.

Our Solution

We decided to solve this by changing the way we wrote Ajax requests in the CMS. It was much more efficient to use Ajax as a static data transfer mechanism, rather than a means of transferring logic code between files. So instead of getting back <script> tags with code in it, we got back JSON data and used that to execute that segment of JavaScript code. PHP makes this easy, as it allows any Array to be encoded into valid JSON. So what we did was respond with something like this

<?php
  header('Cache-Control: no-cache, must-revalidate');
  header('Content-type: application/json');
  $response = array(
    'status' => "success",
    'markup' => '<div class="success">Successfully saved changes to the file.</div>'
  );
  echo(json_encode($response));
?>

Instead of merely echoing that HTML in a String, we are wrapping it with JSON and returning that. Notice that you need special HTTP headers to return a valid JSON response, some browsers attempt to guess based on the content but you can't really rely on that. To ensure that the browser is interpreting this response as JSON, you must specify the HTTP Content-Type as application/json. I've also included a Cache-Control directive which forces the browser to reload the content every time it loads. This ensures that browsers won't cache these requests, because your data is always changing.

So now, we can write some JavaScript to process that data. We like to use the jQuery.form plugin at the office...

<script>
    $('#saveForm').ajaxSubmit({
        url: 'ajax/saveChanges.php',
        dataType: 'json',
        success: function(response, xhr, status) {
            $('#status').append(response.markup).slideDown(400);
            setTimeout(function() { $('#status').slideUp(400); }, 5000);
        }
    });
</script>

That extension to the jQuery library simply submits a form asynchronously, and its options Object lets you bind actions to the various jQuery.ajax callbacks.

A little philosophy

So now you know the code, but it's time to learn about why exactly I chose to do it this way.

Ajax is for transferring data, not logic

You can boil every Ajax call down to one formula: send request, receive response, execute callback(s). This is, after all, the basic format of an HTTP request. Knowing this, you can design your Ajax requests to simply be informational, then when a response is received you execute the JavaScript inside its success callback. This may be a bit confusing to debug, but it's a little easier if you extrapolate every call to a function in the same external .js file, and name your functions by the action they perform. This ensures an application that performs exactly as you expect it. You'll never have to hunt through HTML code in an Ajax response, nor will you ever have a question as to what type of content you're getting from this particular HTTP request.

Standardize your JSON response

Once you've gotten this pattern down, it's important to keep a standard practice when creating your JSON responses. Lets say you want to GET an object, like a Post, from your blog. Fortunately, you've written a small PHP script that will return it from the database. Let's just say, for shits and giggles (and because it's almost 2:00am and I don't feel like writing it out), that this PHP script works and returns the following JSON response:

<?php
    $id = $_GET['id'];
    $sql = "SELECT title, body FROM `posts` WHERE `id` = '$id'"
    $response = mysql_query($sql, mysql_connect('localhost', 'user', 'pass', 'database'));

    if (count($response)) {
        echo json_encode($response);
    } else {
        echo json_encode(array('message' => "Error: Shit don't work!"));
    }
?>

Right now, this doesn't follow a standard practice. response.message would have to be tested on the client side to see if this is an error or not. That's no way to do business, especially if you may need multiple messages or styles for each message. A better way to create this JSON would be to do

<?php
    $id = $_GET['id'];
    $sql = "SELECT title, body FROM `posts` WHERE `id` = '$id'"
    $post = mysql_query($sql, mysql_connect('localhost', 'user', 'pass', 'database'));

    if ($id && count($post)) {
        $response = array(
            'type' => 'success',
            'post' => $post
        );
    } else if (count($post)) {
        $response = array(
            'type' => 'warning',
            'message' => "Post not found"
        );
    } else {
        $response = array(
            'type' => 'error',
            'message' => "No ID passed"
        );
    }

    header('Cache-Control: no-cache, must-revalidate');
    header('Content-type: application/json');

    echo json_encode($response);
?>

We know that if response.type is "success", then response.post will be defined and we can work with it. But if response.type is "error", we will know that response.message is defined and we need to display it. The JavaScript can use a switch statement to parse other types of messages like "warning" or "info", which we may use in the future. Altering your JSON responses to be standard now will save you much testing and coding time in the future. Observe how much more informative these error messages are, and how much more flexible your code is after this alteration.

innovators with the biggest impact on society

24 Nov 2011

A response I made to a thread on the Something Awful forums:

Tim Berners-Lee is the innovator who made the biggest impact on society. He created the Web and laid out guidelines to ensure that it would never become the capitalist nickel-and-dime cesspool that the Mobile Web was created to be. Instead, he built the Web to enforce open-source principles (go ahead, try and hide your JavaScript code from me...), and released it to academia first to ensure it would grow as an information hub rather than a marketing tool. While others have taken the Web and warped it for their own financial interests, Tim Berners-Lee has contented to teach and further the concept of a place where humanity could finally be free to experience anything they wanted, or learn about anything they wanted, in seconds.

In my opinion, the Web is still the only 24/7 place in the world where you can really say and do anything you want. It's the reason why most of the music industry has come to terms with the fact that doing your own thing and doing what you love to do is the ONLY reason to be a musician these days, because the Internet and the principles of the WWW have made it so the labels can't control the industry anymore. It's no longer a legal monopoly. Web connectivity is the major reason why the Arab Spring protests could even happen in the first place, without government control they were able to access new ideas and organize effectively. The anger has been in their blood since the 60's, but the Web made it all coalesce so it could drive their oppressors out of the country.

Not only that, but Tim continues to contribute to the Web's ecosystem and publishes papers and articles commenting on the state of the Web, how certain companies are furthering its intended purpose and how some of them are actually diminishing it.

So yeah. Have any of the people mentioned thus far built technology that allowed a revolution to happen? Not a revolution involving the exchange of money, but a REAL one?

Interesting how much advertising affects our life that we miss some of the great innovators simply because they didn't want to/have to market their product.

a new beginning

20 Aug 2011

So here we go, another new layout and another re-hash of this blog's content. I threw away the last two iterations (still have the data, though) of this blog, because I was unsatisfied with the poor content on it. So let's start again.

This will be a blog about programming, music, philosophy, and everything in between.

first of all, the blog

This blog is not in its final design. I just felt like I needed a place online to express my feelings and my experiences throughout life, and regardless of what it looks like I feel as though it needs to be on my domain, at all costs. That's why I've made deployment as easy as possible. All I have to do is a git push to my remote server, and the hook on the remote will generate the Jekyll site on http://psychedeli.ca, which will then push the update directly to the userbase. Pretty nifty, I think!

burning man

I'll be attending Burning Man this year, but I will not be performing. Instead of gifting music to the playa, I've decided to make this year my vacation. Sometimes I tend to get rather stressed out before and after performing, and I don't want that to encroach on my good time this year on the playa. This year, I'll be gifting something very different to the denizens of Black Rock City. In exchange for a free ride to and from the playa, I'll be driving a 24' Budget truck for another camp, Club Verboten, out of Reno. This is interesting on two levels: First of all, while I have driven rental trucks I've never driven one this big, so that's going to be an adventure in itself getting onto the desert in this thing. Second, I'm going to arrive in Black Rock City about two days before the festival starts. I hear it's a great time, and a unique experience in and of itself, to see the playa before it's really built up.

the wonder bars

We're currently recording our first album! Aaron and Jules and I are prepared to do live recordings of the instruments and finally realize a project that's been literally two years in the making. We have not decided on how we are going to distribute this album. We also have not determined the length of the album just yet, as I've begun to collaborate with other members of the band on a few more tracks than the initial planned 12 that I've written entirely. A full post about that is coming soon.

i'll be seeing you

Well, that's about it. I'm committed to writing more in this thing about my future experiences, as I think it would be a good idea for my own mental health if I could get a lot of these feelings off my chest.

incorporating ultrabeat into my workflow

26 Jun 2011

Thanks to The Architech coming over and showing me some of the more interesting features of Ultrabeat, my drum composition has changed tremendously. I used to use ReDrum, a noble and wonderful little drum machine packaged into Reason, because of its familiar look-and-feel (its UI is modeled after the TR-909) and dead-simple feature set that was not only easy to comprehend, but easy to manipulate in a live setting. But composing in ReDrum has its drawbacks, and composing in Ultrabeat works better with the way I think drums ought to be composed in electronic music. However, this presents an interesting problem: since my live setup relies on Reason for drum beats, how am I going to get Ultrabeat patterns and sounds in Reason?

a little evangelism

Starkey turned me onto Ultrabeat at my last semester of UArts during a L.A.C.E. (Laptop And Controller Ensemble, a short-lived ensemble group established by Starkey and composed of UArts students who had an interest in live electronic performance) rehearsal one Tuesday night. He builds all his beats with the plugin, but the way he introduced it to me was describing a bug he had where it CRASHED Logic! Not a really good place to start...leaving me a little prejudice against its reliability until I had a conversation with Architech a few months ago at an Arts Garage party. It eventually led to him coming over to check out the empty room in my house (which he didn't take) and us educating each other about music. One of the most interesting things he taught me was some of the concepts of Ultrabeat that are really, really cool, and basically devalue my earlier reasons for not using the plugin.

the old-fashioned way

I'd like to illustrate my former workflow for dealing with drums. The major problem with drums, and this is why they can't just be bounced and used like any other track, is that the sound of all the other tracks revolve around the sidechain compression of the kick and/or snare. So these tracks need to be extrapolated at all times so the compressors can use them as keys for the sidechain. This is a major component of the house sound in general (plus, it sounds cool), and thus needs to be carried over into live performance from the studio. While screwing around during a rehearsal, I discovered that Reason files can be opened/closed at will without affecting the sequencer, so when I have one .rns file open and open another one simultaneously, I can close the first one without affecting the sound of the sequencer. Hopefully. There have been times where the computer simply gets overloaded and can't handle all that instruction at once, wherein the sound "hiccups" a little. To date, this has never happened to me in a live performance, but I still don't want to risk it. There must be a cleaner way to do this...

the new hotness

Ultrabeat is like any other drum machine. It's got a 32-square pattern sequencer and 16 separate sample tracks, each with their own dynamics control (as if they were just normal Logic tracks), and additional options for crafting the drum sound like a drum synth, filters and envelope definition. There can be an unlimited amount of patterns stored inside the plugin itself, but I almost never save my patterns, instead opting to click the grid icon next to the pattern chooser and dragging the current pattern into Logic's arrangement window, where it is added to the Ultrabeat track as MIDI. This MIDI controls the entire drum machine, allowing you to create "evolving" beats that can change every 2 bars (or longer), and so you can easily see which patterns are being invoked on each measure. By using Ultrabeat, and adopting a more MIDI-centric workflow, my new songs have taken on quite a different character in the drum track. Since it's so much easier now to create different types of beats in my drum machine, I'm able to push further and explore more variants of beats.

One of the things I like about Ultrabeat's UI is that it can be altered to fit your current workflow. What if you have the sounds you're looking for, but you want to actually sit down and write the pattern? Hit "Pattern view" and the sound-editing options will disappear, leaving you with a huge grid to play with. But what if that kick is filtered a little too high? No problem, just click the track and hit "Full view" to bring up the sound editing options, where you can alter the filter. Notice that below the editor, you can edit the pattern for just this track. Reason users may find this interface to be a little more at-home ( I know I did) because you're used to editing one track at a time. But I strongly recommend moving over to the pattern view way of composing, it's much easier to create cleaner drum tracks that don't collide with each other.

So Ultrabeat is a great interface for working with drums, but how about controlling the sound after the drums have been synthesized? 99% of the time, I have to apply some mastering plugins to get the drum sound I want out of Ultrabeat and my TR-808/909 samples. With Reason, I was able to totally control the routing of each drum track and establish compression/filtering for each separate drum sound. This gave each of my tracks a professional, heavy and floor-filling sound that was quite malleable yet easy to deal with. My father once told me that production-wise, the foundation of the track, drums and bass normally, may be the most important elements of the whole song, and I've always strived to make this foundation both interesting and prominent, regardless of the genre I'm working with. It may also be a reason for why I enjoy creating electronic music, and house/dance specifically, because of this attention paid on the foundation. It's just not House without a booming kick & funky bass line!

This is where it gets interesting. Because Ultrabeat is built into Logic, we can actually create Aux tracks from the drum kit itself, and extrapolate some of the drums into their own separate mono tracks. From here, we can EQ, add effects and generally craft each drum sound just the same as we could with Reason. You can even apply separate automation for each drum aux and really get down-and-dirty with controlling the sound. After learning all of this, I decided it was time to finally abandon ReDrum for composing, and totally immerse myself into this new plugin that greatly accelerated my workflow and taught me a LOT about Logic.

bringing it all back home

So now I have these new tunes created in Ultrabeat, but how do I play them live? This question plagued me for weeks after I realized a month ago I would need to somehow extract these drum parts and use them in Ableton. But Ableton doesn't support Ultrabeat, so a major disconnect was taking place. My old workflow used Reason for the drums, so it was easy to just open the file in ReWired Ableton and play it through a couple audio tracks. But now that I no longer use Reason to compose drums in, I need to figure out a way to migrate the drum patterns I come up with into a sequences I can trigger easily and efficiently.

After doing some tests, I discovered that Ultrabeat's MIDI output is standardized General MIDI for drums, and therefore can be imported directly into Ableton Live (each region even becomes its own clip!) and pumped via ReWire to the Reason instance. My Reason files are now simply used for saving the preset sounds of each song's drum kit, and Ableton will now sequence the patterns like it was meant to do originally.

This part is actually a little confusing, but it makes sense in the end. What you have to do is select the MIDI regions you want in Logic (I usually group by song form: verse beats, chorus beats, etc.) and go to File > Export.. > Selection as MIDI File, and then create a new directory called "MIDI Files" in your Logic project folder. Name the file after its form name, so "drums-verse.mid", etc. This allows you to use MIDI in other ways, like for example if you use a VST you can simply open the VST in Ableton and pump in MIDI/presets you created in Logic. This allows you to alter the sound on the fly for your particular PA system (they're all different, I find myself doing a lot of audio engineering work at the beginning of a set to make the speakers react to my music properly).

To import this MIDI into Ableton, create a new MIDI track for holding drum patterns (I called mine "Drums"), and simply track the .mid file into the track. The clips will show up horizontally, simply drag them into subsequent vertical rows in your Live Set to establish a proper order for the drums to go in. By clicking these clips directly, you can invoke the MIDI for each drum pattern.

In Reason, create a new file and remove the mixer. Add in a ReDrum plugin instance and set up the drums in exactly the same order as you have them in Ultrabeat. So for example, if your drum sample order looks like: kick, snare, claps, open hat, closed hat, ride in Ultrabeat, that's what you'd want in Reason. Mute out the tracks that aren't pre-set this way to avoid wayward notes triggering drums that aren't set up properly. Hit "Tab" to get the routing panel, and route the kick & snare into separate aux channels in Reason's hardware interface patchbay, then take those channels as input into Ableton for your kick & snare tracks. This will extract just the kick & snare into separate tracks so you can set sidechains for them in other instruments. Ableton's compressor has sidechain built in now, so no need for extra VSTs taking up your memory!

So now you have Reason set up to read MIDI from Ableton, and to pump drum audio back into the sequencer through 3 separate tracks. This is the way I do things live and at home, and I hope it helps your workflow because it's certainly helped mine!

decisions, decisions

9 Apr 2011

Well I'm stuck in a bind. I can't seem to decide which operating system I want to run my all-powerful file server. A few days ago, my wonderful sister gifted me two computers, the Dell OptiPlex GX270 and GX280. The basic difference between these two tried-and-true workhorses is the graphics card interface-the GX280 includes a PCI Express x16 slot whereas the GX270 is still using AGP-and of course the model number. Other than that, they use the same motherboard, same case design, and same basic layout. PERFECT for general hackery purposes...and running a Boxee...

So my first endeavor with these machines is something I've been needing for a long time: a file server to host all of the house's media. Everyone who lives in my house utilizes the internet or computers mostly for their media needs, and we've been longing for a way to experience it on a bigger screen. But a problem arises: These machines do not have the ability to pump S-Video to my regular TV, so I'm gonna need to wait a couple days before actually going ahead and opening up the boxes. I also need SATA cables and a SATA-to-IDE converter because the Dell OptiPlex motherboards come with IDE by default! By using both the SATA connector and the IDE connector adapted to a SATA connection, I'll be able to run a software RAID-0 array, mirroring the contents of the 1TB of space inside the box and therefore providing a greater level of security and stability to the system as a whole. With all of the hits it will be getting, this is a crucial step in extending the server's lifespan.

This is actually a good thing, because it gives me a bit more time to decide on an operating system. What I need is very simple, I've boiled it down to only a few requirements:

I want ZFS. It's amazing, the technology we have today. ZFS is this wonderful file system developed by the good ol' boys over at Sun Microsystems (now Oracle) that addresses many of the problems of former filesystems and even some of the problems of magnetic hard drives themselves. ZFS even introduces new features like built-in SMB/Netatalk capabilities, making it a lot easier to transfer data from the server to the computers where it will actually be useful.
I want complete Boxee transparency. The Boxee will be the major customer of this server, constantly pulling data from the media drive "/Zappa" to fulfill its users demands of the music library. Now that all music can be accessed from the living room's 3-way stereo system, I'm predicting that my roommates will be hitting the Boxee a lot more, and thus the fileserver.

Right now, my best option is FreeNAS, it's a distribution of FreeBSD (an OS I know and love) that's specially formulated to act as a network storage device. Precisely what I'm looking for! It's even bundled with a DAAP server and NFS so the Boxee can be "tricked" into thinking /Zappa is actually a part of its own local filesystem. The possibilities are endless, allowing for seamless integration of the data on these disks to the entire network here at 1913.

I think my choice has been made.

UPDATE: My FreeNAS file server is running smoothly, fully jailed. Story at 11.

Technorati Claim ID: NFGKYZ5CMSY2