INFO  [2016-09-01 20:41:23,904] io.dropwizard.jersey.DropwizardResourceConfig: The following paths were found for the configured resources:

POST    /eat/lunch (com.tradier.rest.LunchResource)
GET     /fruit/banana (com.tradier.rest.BananaResource)
GET     /vegetable/avocado (com.tradier.rest.AvocadoResource)

But it’s both impractical and slow to startup a server periodically to see it’s endpoints. Of course documentation would be a solution here as well, but ain’t nobody got time for that!. Fortunately, Dropwizard 0.8 added public methods to DropwizardResourceConfig to access the endpoint data in a way that can be leveraged within an application. Not wanting this to be part of a public API, Dropwizard tasks are a great place to add this to our application:

public class EndpointsTask extends Task {

  public EndpointsTask() {
    super("endpoints");
  }

  @Override
  public void execute(ImmutableMultimap<String, String> parameters, PrintWriter output)
      throws Exception {
    DropwizardResourceConfig config = new DropwizardResourceConfig();
    config.packages("com.tradier.rest");
    output.println(config.getEndpointsInfo());
  }
}

Now, when we want to see the endpoints for a given application, we can just access the task!

This post has been cross-posted to the Tradier Developer Blog, for more posts like this, you may want to follow our posts there as well!

Using the Redis Ruby Gem, we first turned to pipelined requests. Pipelined requests return a future, meaning in order to fully query and load, you essentially have to loop twice:

data = {}

$redis.pipelined do
  keys.each do |key|
    data[key] = $redis.hgetall(key)
  end
end

data.each do |key,value|
  data[k] = v.value
end

While this does the job, it’s tedious and with large key-sets not as performant as we’d like it to be. What we’d really like is something closer to Memcached’s multi-get support. As we considered other solutions, we decided to take a look at Redis’ scripting support to see if it could help. Not really knowing much about Lua, we were pretty surprised by how powerful Lua was. Using Lua, we can make a single request to Redis, passing all of the keys as an argument to the Lua script:

local collate = function (key)
  local raw_data = redis.call('HGETALL', key)
  local data = {}

  for idx = 1, #raw_data, 2 do
    data[raw_data[idx]] = raw_data[idx + 1]
  end

  return data;
end

local data = {}

for _, key in ipairs(KEYS) do
  data[key] = collate(key)
end

The code was fairly simple. We can loop through the KEYS and invoke the collate method we defined to load the hash data. The challenge then became passing this data back to our ruby code. We found that while Lua objects will not easily serialize back to a Ruby object, Redis’ Lua implementation offers up some options: namely cjson and cmsgpack. We need just return from the script and we’re now returning data back:

-- return json
return cjson.encode(response)

-- return messagepack
return cmsgpack.pack(data)

What we found is that between pipelined requests, lua + json, and lua + messagepack, messagepack was the best performer of the three solutions. Our final solution ended up something like this:

require 'redis'
require 'msgpack'

keys = %w(FOO BAR BAZ)

lua_msgpack_loader = <<LUA
local collate = function (key)
  local raw_data = redis.call('HGETALL', key)
  local hash_data = {}

  for idx = 1, #raw_data, 2 do
    hash_data[raw_data[idx]] = raw_data[idx + 1]
  end

  return hash_data;
end

local data = {}

for _, key in ipairs(KEYS) do
  data[key] = collate(key)
end

return cmsgpack.pack(data)
LUA

redis = ::Redis.new(:driver => :hiredis)
data = MessagePack.unpack(redis.eval(lua_msgpack_loader, :keys => keys))

Of course, no post like this would be complete without a benchmark (using 10K keys):

                      user     system      total        real
lua + json        0.350000   0.010000   0.360000 (  1.242315)
lua + msgpack     0.260000   0.020000   0.280000 (  1.146377)
redis pipelined   1.070000   0.020000   1.090000 (  1.759858)

Overall, we were pleasantly surprised by Redis and Lua and it’s definitely a solution we’ll turn to in the future as well.

This post has been cross-posted to the Tradier Developer Blog, for more posts like this, you may want to follow our posts there as well!

As we built out our new market data service, we found a need to add a task. Dropwizard’s documentation is fantastic, encompassing everything from application structure to internals and semantics, but tasks aren’t covered particularly well. We’ve submitted a pull request to improve the docs, but opted to document our learnings here as well.

The Task is defined as an abstract class that you must subclass. The main thing to point out is the execute method which is called when the task is invoked. Let’s define a sample task to see how it works. For this example, we’ll create a task called TruncateDatabaseTask. First, we’ll define the class:

package com.myapp.tasks;

import io.dropwizard.servlets.tasks.Task;

public class TruncateDatabaseTask extends Task {
}

Dropwizard requires that each task have a name. The name has a special purpose as it’s also used to compose the path that the task is accessible from within Dropwizard. The Task defines a basic constructor that we can use, but let’s define it explicity via a constructor of our own (more on this later). We’ll also implement the execute method even though we’re not doing anything yet.

package com.myapp.tasks;

import io.dropwizard.servlets.tasks.Task;

public class TruncateDatabaseTask extends Task {
  public TruncateDatabaseTask() {
    super("truncate")
  }

  @Override
  public void execute(ImmutableMultimap<String, String> parameters, PrintWriter output) throws Exception {
  }
}

At this point, we have a fully composed Dropwizard task. Now we just need to add it to our application:

public void run(MyAppConfiguration config, Environment env) throws Exception {

  // tasks
  env.admin().addTask(new TruncateDatabaseTask());
}

Start your application, and you’ll see that the task is now registered alongside the garbage collection task that Dropwizard adds automatically

INFO  [2014-05-16 19:15:35,564] io.dropwizard.setup.AdminEnvironment: tasks =

    POST    /tasks/gc (io.dropwizard.servlets.tasks.GarbageCollectionTask)
    POST    /tasks/truncate (com.myapp.tasks.TruncateDatabaseTask)

Let’s make this a more practical example. The value of defining a constructor is that we can dependency inject anything we need into the task from our Dropwizard application:

package com.myapp.tasks;

import io.dropwizard.servlets.tasks.Task;

public class TruncateDatabaseTask extends Task {
  private Database database;

  public TruncateDatabaseTask(Database database) {
    super("truncate")
    this.database = database;
  }

  @Override
  public void execute(ImmutableMultimap<String, String> parameters, PrintWriter output) throws Exception {
    this.database.truncate();
  }
}

Most of our Dropwizard tasks look a lot like the one above. We found these tasks valuable as schedulable components of our application without building custom scripts to tie back to the current runtime. If you use Sinatra or Express to build service APIs definitely check out Dropwizard.

This post has been cross-posted to the Tradier Developer Blog, for more posts like this, you may want to follow our posts there as well!

We use Sentry to manage our exception handling, so I figured there must be a way to handle these warnings just like exceptions. Since I had to read a little bit of Rails code to figure this out, I figured I’d share what I learned. ActiveSupport allows you to define one or more deprecation behaviors by using ActiveSupport::Deprecation.behavior=. Several options are included by default:

• :raise - raise an exception instead of warning
• :stderr - log them to $stderr
• :log - log them out using Rails.logger
• :notify - publish an alert using ActiveSupport Notifications
• :silence - sweep them under a rug and pretend they never existed, until they are truly deprecated and bite you.

Based on the documentation, you can also pass a custom handler class or a proc, basically, anything that responds to call can be used. When a deprecation warning is issued, the list of behaviors is looped through and passed the deprecation message and callstack:

def warn(message = nil, callstack = nil)
  return if silenced

  callstack ||= caller(2)
  deprecation_message(callstack, message).tap do |m|
    behavior.each { |b| b.call(m, callstack) }
  end
end

By default, Rails uses :stderr. I wanted to keep that behavior, but also include Sentry. ActiveSupport Notifications are a perfect way to do that. Using the raven gem to hook into Sentry, here’s where I landed:

ActiveSupport::Deprecation.behavior = [:stderr, :notify]
ActiveSupport::Notifications.subscribe('deprecation.rails') do |name, start, finish, id, payload|
  Raven.capture_message("DEPRECATION WARNING", {
    :logger => 'logger',
    :extra  => {
      'message'   => payload[:message],
      'backtrace' => payload[:callstack].join("\n  ")
    },
    :tags   => {
      'environment' => Rails.env
    }
  })
end

class Devise::Mailer < Devise.parent_mailer.constantize
  include Devise::Mailers::Helpers

  # there's more stuff in here, you should read the source!
end

Why is this useful? There are times when it’s convenient to be able to change this class. For instance, let’s suppose we want to automatically bcc an email address on new emails delivered from Devise. I needed to do this recently and found it to be quite easy using the parent_mailer configuration.

First, create a new mailer class that inherits from ActionMailer::Base and put it in app/mailers, let’s call it BaseMailer. The simplest example would be an empty class:

class BaseMailer < ActionMailer::Base
end

Then, in the initializer (typically config/initializers/devise.rb), instruct Devise to use this class as the parent_mailer:

Devise.setup do |config|
  config.parent_mailer = 'BaseMailer'
end

For the sake of brevity, I’ve omitted the rest of the Devise configuration, but you’ll want to add the parent_mailer alongside any other configuration items. Now DeviseMailer will inherit from our mailer class. Now, it’s just a matter of hooking in our auto-bcc logic:

class BaseMailer < ActionMailer::Base

  protected

  def mail_with_bcc(headers={}, &block)
    headers.merge!(:bcc => 'me@example.com')
    mail_without_bcc(headers, &block)
  end
  alias_method_chain :mail, :bcc
end

As you can tell, I really enjoy reading other people’s code. Especially when you are pleasantly surprised by a feature that solves a problem you are trying to solve!

NoMethodError: undefined method 'authenticate' for nil:NilClass
  ~/Projects/myapp/.bundle/gems/devise-3.0.1/lib/devise/controllers/helpers.rb:56:in 'current_user'
  ~/Projects/myapp/.bundle/gems/paper_trail-2.7.2/lib/paper_trail/controller.rb:17:in 'user_for_paper_trail'
  ~/Projects/myapp/.bundle/gems/paper_trail-2.7.2/lib/paper_trail/controller.rb:59:in 'set_paper_trail_whodunnit'

What’s going on here? Konacha’s SpecsController inherits from ActionController::Base so that it can render ERB templates and layouts which setup a browser environment to run the test suite. Pretty standard stuff for a Rails Engine. That brings us to PaperTrail and it’s controller callbacks. By default, PaperTrail enables itself for all controllers, SpecsController included. If we look at user_for_paper_trail, we see the following:

def user_for_paper_trail
  current_user if defined?(current_user)
end

It’s triggering current_user which invoked Devise, causing the exception above. Clearly, PaperTrail’s user fingerprinting are interfering with Konacha. To fix this, we simply need to disable PaperTrail in the SpecsController. In addition to your Konacha configuration, you’ll want to have something like this in config/initializers/konacha.rb:

if defined?(Konacha)
  require 'capybara/poltergeist'
  Konacha.configure do |config|
    config.spec_dir    = "spec/javascripts"
    config.driver      = :poltergeist
    config.stylesheets = %w(application)
  end

  Konacha::SpecsController.class_eval do
    def paper_trail_enabled_for_controller
      false
    end
  end
end

With that in place, our tests started to work again. Konacha and Mocha are a nice combination for testing JavaScript, be sure to check them out. Happy testing!

• app1 deployed to /apps/app1/current at mydomain.com
• app2 deployed to /apps/app2/current at mydomain.com/app2

This is easily accomplished with Passenger, but took some tinkering to figure out. Here’s what I ended up with:

<VirtualHost *:80>
  ServerName mydomain.com
  DocumentRoot "/apps/app1/current/public"
  RailsEnv production

  <Directory "/apps/app1/current/public">
    Options Indexes FollowSymLinks -MultiViews
    AllowOverride All
    Order allow,deny
    Allow from all
  </Directory>

  <Directory /apps/app1/current/public/app2>
    Options Indexes FollowSymLinks -MultiViews
    AllowOverride All
    Order allow,deny
    Allow from all
  </Directory>

  RackBaseURI /app2
</VirtualHost>

Apache is now properly configured. All that remains is a little setup in the apps themselves. Since the Apache config expects a Rack/Rails app at /apps/app1/current/public/app2, you just need to create a symlink in /apps/app1/current/public to app2:

ln -s /apps/app2/current/public app2

Depending upon your deployment strategy, you may need to create the symlink each time you deploy. I setup a symlink as part of the capistrano deploy and everything just worked™.

Bonus: If you use a common datastore and session_store key within your Rails applications, you can share authentication between the two apps to create a cohesive “logged-in” interface.

CarrierWave.configure do |config|
  config.fog_credentials = {
    :provider               => 'Rackspace',
    :rackspace_username     => ENV['RACKSPACE_USERNAME'],
    :rackspace_api_key      => ENV['RACKSPACE_API_KEY'],
    :rackspace_servicenet   => Rails.env.production?,
    :rackspace_temp_url_key => ENV['RACKSPACE_URL_KEY']
  }
  config.fog_directory = 'mp3s'
  config.fog_public = false
end

Add an Uploader class to your model and you are nearly all the way there:

class RecordingUploader < CarrierWave::Uploader::Base
  storage :fog
end

class Recording < ActiveRecord::Base
  mount_uploader :file, RecordingUploader
end

But, I hit a snag. Not wanting the files I was uploading to be publicly available, I had configured fog_public = false in the CarrierWave configuration. But when I tried accessing the files from my AR model, I kept getting errors. At time of writing, the latest version of CarrierWave (0.7.1) did not support private Cloud Files. Fortunately for me, CarrierWave master did. Assuming that was all I needed, I was hopeful that my problem was fully solved but I was still having issues.

What I had missed was the rackspace_temp_url_key. This is something you need to established before use. I couldn’t find a way to do this on Rackspace’s website, but fortunately, Fog exposes an easy way to set this via API. I stuck a little bin script in my scripts directory and ran it to set a key:

#!/usr/bin/env ruby

require 'rubygems'
require 'bundler/setup'
require 'fog'

storage = Fog::Storage.new(:rackspace_username => ENV['RACKSPACE_USERNAME'],
                           :rackspace_api_key  => ENV['RACKSPACE_API_KEY'],
                           :provider           => 'Rackspace')
storage.post_set_meta_temp_url_key(ENV['URL_KEY'])

Once the rackspace_temp_url_key was properly setup, I was all set. Private Cloud Files accessible using CarrierWave.

Why was this so hard, you ask? For starters, Facebook’s documentation is particularly lacking. Most of the documentation leads one to believe it’s really only good for reading from the API. It’s difficult to understand how you can create anything. Further muddying the waters is the amount of irrelevant information out there since the release of the Graph API. They’ve changed their product so much is such a short period of time that it’s hard to tell what’s relevant and what isn’t. Many tutorials you’ll find won’t work because they speak to the old REST API that is now deprecated. In other words, separating old from new is a bit of a needle/haystack problem.

So where to start, first read up on Facebook API authentication. So we’re going to authenticate via OAuth and once authenticated, make calls against the API. But how do we do this as a page instead of as ourself? Facebook devotes 2 paragraphs to it in their documentation under a section titled “Page impersonation”. I highly recommend you read it, especially this: “Once a user has granted your application the “manage_pages” permission, the “accounts” connection will yield an additional access_token property for every page administrated by the current user. These access_tokens can be used to make calls on behalf of a page. The permissions granted by a user to your application will now also be applicable to their pages.”

They’ve laid it out for us:

• grant your application the ‘manage_pages’ permission
• use ‘accounts’ to yield an access_token for the page
• use the access_token to make calls on behalf of the page

I’m going to show you how to do it. Let’s assume you’ve done the following:

• created a Facebook Page
• created a Facebook app
• added the app to your profile

If you haven’t done that yet, go right ahead, we’ll wait for you to come back. Make note of the ids of your app and your page, we’ll need those shortly. Now, we’ll need to retrieve an access token for our app so that we can use it to make Graph API calls. The easiest way to do this is to go to the browser and enter a URL that looks like the following:

https://graph.facebook.com/oauth/authorize?type=user_agent&client_id=<APPID>&redirect_uri=<CALLBACK_URL>&scope=manage_pages,offline_access,publish_stream

The comma delimited options we are passing to the scope argument specify what permissions we are granting to the app. offline_access allows us to use the access_token outside of a user session. Without this, the token will only be valid during your user session, when you logout, the token expires. We also need to allow the app to manage_pages that we administrate. Lastly, publish_stream is what allows us to create content via the API.

Visiting that link will bring you to an authorization page where Facebook will prompt you to verify that you want to allow your app access to the privileges you specified above. When you click approve, you’ll be redirected to your callback URL and Facebook will pass your access token in the URL:

your_callback_url?access_token=<YOUR_TOKEN_HERE>&expires_in=0

Make sure to record the access token. Armed with that, you can now make authenticated requests against the Graph API. At this point, we should have all we need to start working with Ruby. There are a number of Rubygems out there in varying stages of development that support the Graph API. I was previously familiar with Mike Mangino’s facebooker rubygem, but it’s meant to work against the deprecated REST API. I was then turned on to Facebooker2 which is a revamp of the original Facebooker gem meant to work with Facebook connect. That’s not really what we’re looking for, but I tried working with Facebooker2 for a while before realizing it wasn’t the perfect fit for what I was trying to do. Inevitably, with this sort of thing, I prefer to dig down into to the code and the tests to see how the gem works. As it turns out, the bulk of Facebooker2 is really a Rails-friendly wrapper around another gem by Mike Mangino called mogli.

With Mogli, we can very easily obtain a client that we’ll use to engage with the API:

access_token = 'my_facebook_access_token'
page_id = 'my_facebook_page_id'

client = Mogli::Client.new(access_token)
user = Mogli::User.find("me",client)
page = user.accounts.select do |account|
  account.id.to_s == page_id
end.first

page = Mogli::Page.new(:access_token => page.access_token)
page_client = page.client_for_page

user.accounts will return an array of accounts which will list any pages and apps you own. I added a select because we want to manage a specific page so we need to retrieve the page based on the page_id. Then we use that page’s access token to create a new facebook client that we can use to work on behalf of the page. Now, we just need to use the client to create our wall post:

post_data = {}
post_data[:name]    = 'Title for my link'
post_data[:link]    = 'http://path.to/my/link'
post_data[:caption] = 'A caption'
post_data[:description] = 'A description'
post_data[:picture] = 'http://path.to/myimage.jpg'
post_data[:actions] = {
  :name => 'My site name',
  :link => 'http://link.to/my/site'
}.to_json

client.post("feed", nil, post_data)

What I’ve demonstrated above is a use-case where you’d post a link with a thumbnail image. Facebook specifies quite a few other things that you can include with your post. A few things are not relevant to page publishing, for instance you cannot set privacy on a page’s wall post as it’s visible to all people that like that page.

Mogli does include a Mogli::Post class but I wasn’t able to get it to work without getting an error response from the API. Fortunately it also supports passing a simple hash of attributes. Notice that I’m serializing the actions hash to JSON before passing it to the hash. That threw me for a real loop, but Facebook very literally wants JSON objects passed as strings.

Executing the above code should result in your post appearing on your pages wall. Congratulations - you made it. Happy wall posting!

(I realized after the fact that woot already has a twitter account strictly for wootoffs, but it was far from complete enough to replace what I’d built. For starters, I would have had to parse the text of the tweet and even then, it didn’t include a direct link to the product or a link to an image.)

What dawned on me was what a poor job RSS does for this sort of thing. I had to continually poll (every 30 seconds) to see if anything had been updated, at some points nothing changed for nearly an hour. I see why technologies like pubsubhubbub and rssCloud are a potential replacements. But why haven’t they taken off? Well, for one, the name pubsubhubbub is a little obtuse, but moreso, it just seems a little complicated. When you have to create a youtube video to explain your technology in overly simplified terms like the pubsubhubbub team did, you aren’t off to all that great of a start. Wordpress adopted rssCloud but has it made any difference? We’re over a year in and neither technology seems to have taken hold in any significant way.

Not foreign to continual polling, Twitter took a different approach, introducing a streaming api. I can’t help but think this is a much easier and more dynamic means of delivery. But who wants to consume a bunch of different streams? Well, pretty soon, we may not have to. Twitter has also been working on annotations. Woot could simply annotate their tweets and anyone writing a service around it just subscribes to the stream. Awesome. Twitter annotations is an amazingly powerful idea wrapped up in the simplicity of their service and I can’t wait to see what people do with it.