flouri.sh - Blog

Ruby Sandboxing Resources

2009-04-11T18:28:00Z

A complete list of sandboxing resources that I've been able to find, from articles on the sandbox itself to examples of the sandbox in action.

The sandbox itself

Freaky Freaky Sandbox - A sandboxing gem based on the MRI ruby interpreter. Written in C, it hacks the VM to allow safe execution of untrusted code
Why-the-lucky-stiff talking about the freaky freaky sandbox - He explains it as an alternative to $SAFE in a discussion with ruby team.
How to set up the C-Ruby sandbox
JavaSand - A sandboxing gem for JRuby. It provides the same API as the C-based-ruby sandbox.
How to set up the JRuby sandbox

Sandbox Support

acts_as_wrapped_class - A gem that adds easy class-wrapping for safely exposing an API to the sandbox code.
acts_as_runnable_code - A gem that makes creation of sandboxes and evaluation of uploaded code easier.
Safely Exposing your App to a ruby sandbox - My article on setting up a sandbox.

Sandbox Examples

How to set up the JRuby sandbox

2009-04-04T21:15:00Z

The JRuby Sandbox is simply a rewrite of why's original sandbox gem in JRuby. It's much less of a hack than the C implementation, and generally considered to be more safe. Here's how I set it up:

Download and install the latest JRuby binaries from CodeHaus (I tested with 1.1.5).
Download the source of the javasand jruby gem from the JRuby addons project
svn checkout http://jruby-extras.rubyforge.org/svn/trunk/javasand
Compile the gem:
ant
BUILD SUCCESSFUL
If the build fails, it might be because it can't find the JRuby classes. You'll need to find jruby.jar and then add a line to build.xml inside the "build.classpath" path:
<fileset dir="/path/to/jruby/jars" includes="*.jar" />
Package up the gem:
jgem build javasand.gemspec
Install the gem:
sudo jgem install javasand-0.0.2.gem

Test the sandbox with jirb -rubygems

require "sandbox"
Sandbox.safe.eval("2+2")
# yields 4

As you can see above, I had to compile the gem from source. The binary gem of javasand from rubyforge failed with the following exception:

irb(main):001:0> require "sandbox"
=> true
irb(main):002:0> Sandbox.safe
org.jruby.ext.sandbox.Sandkit:714:in `removeMethods': java.lang.NoSuchMethodError: org.jruby.RubyModule.removeMethod(Ljava/lang/String;)V

How to set up the ruby sandbox

2009-04-04T20:43:00Z

There's very little recent work on the MRI ruby sandbox, so here's a quick guide to getting the sandbox installed and running. Unfortunately, the sandbox requires a patched ruby, but luckily it's not that hard to set up.

Download the latest version of ruby 1.8.6 from ftp://ftp.ruby-lang.org/pub/ruby/1.8 (does not work with 1.8.7 or 1.9, sorry)
Download the sandbox gem source from git://github.com/why/sandbox.git
Patch ruby:
patch -p1 < ../sandbox_gem/patch/ruby-1.8.6-sandbox_needs.patch
patching file error.c
Compile and install the patched ruby:
./configure make sudo make install
Download and install rubygems from RubyForge
Install the sandbox gem:
cd sandbox_gem && sudo ruby setup.rb

Test the sandbox:

require "sandbox"
Sandbox.safe.eval("2+2")
# yields 4

Now that you've got the sandbox running, read more about it in my article on Advanced Sandboxing, or my Sandbox Introduction.

I'm speaking at Golden Gate Ruby Conf

2009-04-04T19:29:00Z

The title of my talk is: Playing with Fire: Running untrusted code in a sandbox. I expect to be posting some more articles related to the sandbox before my talk on April 17th.

Duplicate joins merge in rails 2.2

2008-11-22T19:25:00Z

If you've ever suffered from duplicate table aliasing problems in rails, there's a new feature in 2.2 that will help some of these situations go away. This seems to come up mostly in named_scope, where you want to define a list of scopes that can work by themselves or with any other scopes. This often means having identical or similar joins in multiple scopes. Take this example:

class User
  has_one :profile
  named_scope :male, {
    :join => "INNER JOIN profiles ON profiles.user_id = users.id",
    :conditions => "profiles.gender = 'male'" 
  }
  named_scope :recently_updated, {
    :join => "INNER JOIN profiles ON profiles.user_id = users.id",
    :conditions => ["profiles.updated_at > ?", 1.week.ago]	
  }
  named_scope :admin, {
    :join => "INNER JOIN profiles ON profiles.user_id = users.id AND profiles.admin = 1
              INNER JOIN emails ON emails.profile_id = profiles.id"
  }
  named_scope :with_profiles, {
    :join => :profile
  }
end

Before 2.2, calling User.male.recently_updated results in a table aliasing problem, because rails joins in the profiles table twice. Three features in 2.2 make this better:

:join with hash/symbol syntax merges hashes when combining scope (Andrew White)
:join now can take an array of strings (David Stevenson & Joseph Palermo)
string identical joins are uniqued when combining scopes (David Stevenson & Joseph Palermo).

After 2.2, you can call User.male.recently_updated because there are two string identical joins combined in different scopes. You can't call User.male.admin without making some modifications, because the two joins involved are not string identical. Here's how I'd modify the :admin scope:

named_scope :admin, {
  :join => ["INNER JOIN profiles ON profiles.user_id = users.id",
            "INNER JOIN emails ON emails.profile_id = profiles.id"],
  :conditions => "profiles.admin = 1"
}

By using the array of strings :join syntax, I can let rails know that those are two separate joins, each of which can be merged if an identical join comes up in another scope. I also moved the extra join condition (profiles.admin = 1)to :conditions, so that the INNER JOIN statement would not have any specific logic for that particular named_scope in it.

I still can't call User.male.with_profiles because the string representation of the join from with_profiles probably doesn't match the join from male. It will be off due to the way rails generates join using different whitespace and escape characters than I originally wrote. This can be easily fixed by copying the exact string rails generates from :join => :profileand pasting it into male.

acts_as_solr has a new home

2008-11-18T20:30:00Z

Developers finally set up master repo for acts_as_solr

After a long time of no development, a new acts_as_solr plugin has emerged on GitHub. Luke Francl & Mathias Meyer have adding a lot to the old plugin and created a new "master". Mathias has merged several branches on github including changes from David Palm, kengruven, and myself. This is great news for people running/modifying acts_as_solr, as Mathias is a real person who will actually respond to your pull requests. Some new features include:

SOLR 1.3 installed
Ruby 1.9 support
LibXML 0.7+ support
Store types as string not text fields in the index
JVM options in solr.yml
Completely rewritten test suite
New solr:reindex task that automatically finds and indexes your solr models

QueryReviewer now aggregates identical queries

2008-09-10T01:18:00Z

I've been using query reviewer exensively at pivotal, and we've noticed that it is ridiculously slow when you have a lot of queries. This is due to the large number of partials rendered (5 partials) for each query in the reviewer box.

A solution to this problem, that also brings a huge advantage, was to report aggregate statistics for near-identical queries. Similar to way mysqlslowdump groups near-identical queries, query_reviewer now also reports only 1 line for X identical queries (identical means the same structure and same stack trace). Consequently, it's a lot easier to find places where you're missing includes . Here's what it looks like now:

The time reported for a group is the total time of all the queries in that group. In addition, I add up all the times for all the queries and report that number in the header. From this, I've noticed that a lot of time is spent in rails outside of waiting for mysql to return result. I wonder where that time goes...

query_reviewer can be downloaded from github!

QueryReviewer is now rails 2.1 safe

2008-06-25T03:50:00Z

Thanks to a nice suggestion by the community which I pretty much used verbatim. Still works with rails 2.0 and 1.2.3. Check out the project homepage.

http://code.google.com/p/query-reviewer/

I'm working at Pivotal

2008-04-30T16:05:00Z

After a 4 month interlude at SpongeFish, I'm now working at Pivotal Labs. Pivotal's a really fun place to do agile rails consulting and I'm loving it so far...

MySQL Query Reviewer - now with AJAX and Profiling

2008-04-10T21:05:00Z

Kevin Hall and I have released a new version of the query_reviewer plugin. You should start by looking at my first post, to see what the basic premise is before reading this article. The single largest improvment is the ability to analyze the database requests of AJAX requests, which is accomplished by piggy backing javascript or HTML into the ends of AJAX responses. Here's what it currently looks like, analyzing my project on a page that does lots of database requests:

The improvements are:

View query analysis for AJAX requests
Take into account the duration of a query (if production data)
Show PROFILE information from mysql
Warnings for long key lengths (which can be bad even when you hit an index)

Lots of information is now available for a single slow query:

I'm really excited that this plugin is getting attention, and welcome feedback, suggestions, and bug reports on the Google Code project homepage.

To install it, simply visit the project homepage.

Flourish is back from the dead

2008-03-30T18:07:00Z

Things started going very badly for flouri.sh when my home server wouldn't boot up. I removed almost all the hardware, but could not even get a POST screen or any beeps. I even reseated the RAM and CPU, but no luck. Finally, I grabbed an old media PC shuttle cube and attempted the craziest idea:

The 4 x RAID5 disks are actually being powered by my old server, but the PCI raid card is inserted into my old media PC. The shuttle doesn't have enough power or room to run all those disks so I'm using the old server as a lifeboat with 4 IDE cables running between them!

Writing data migrations in rails

2008-02-18T19:11:00Z

Using ActiveRecord can be tricky, here's how

Data migrations are a major headache in every rails project I've worked on. Developers typically write straight SQL migrations which take much longer to create and test, or they use ActiveRecord and run into problems. I recent wrote a data migration which took 7 existing tables and compressed them into 4. Some of the new 4 had the same names as the existing models, so I finally figured out how to do this safely with ActiveRecord. Here's my advice:

When changing the schema of an existing set of tables, create new tables and then rename them.
Include active record fragments inside the migration class at the top of your migrations.
Put your data migrations inside transactions.
Make your data migrations completely reversible (which is a lot easier when you follow the first rule)

Here's an example of an ActiveRecord class fragment at the top of a migration:

class MarkPrimaryBits < ActiveRecord::Migration
  class LessonVersion < ActiveRecord::Base
    belongs_to :lesson
  end

  class Lesson < ActiveRecord::Base
    has_many :lesson_versions
  end
 
  def self.up
    ...

By putting the Lesson and LessonVersion classes at the top of the migration, I'm allowing myself to use those classes inside my migration and be completely independent of any changes made to the real model from then on (including the deletion of the class itself). Furthermore, I can have another migration which uses those same class names with completely different meanings and they won't conflict with each other.

Extending has_many associations correctly

2008-02-15T18:11:00Z

How to build custom methods on associations that aren't slow

The "has_many do" syntax has been widely adopted in rails, but I often see it going wrong. Consider the following association extension:

has_many :versions do
  def primary
    find(:first, :conditions => {:primary => true})
  end
end

What's wrong with this code is that the finder has to execute a database request every time it's invoked. True, rails has a query cache, but it results in more database requests then one should need. We can make it better by caching the result in an instance variable attached to the association:

has_many :versions do
  def primary
    @primary ||= find(:first, :conditions => {:primary => true})
  end
end

But what if you already have the association loaded? Inside the block, you can access various methods of the AssociationCollection and AssociationProxy classes. Notable methods are the following:

proxy_owner - the module that contains the association
proxy_reflection - the Reflection object that contains the association options (FK, :dependent, etc)
proxy_target - the cached association data, if the association has been loaded
loaded? - returns true if the association has been loaded
reset - delete the cached association data and forget it has been loaded

Using these methods, we can rewrite our method to be the most efficient possible, by making use of loaded association data when present:

has_many :versions do
  def primary
    if loaded?
      @primary ||= proxy_target.detect { |ver| ver.primary? }
    else
      @primary ||= find(:first, :conditions => {:primary => true})
    end
  end
end

Now our method is guarenteed to make only 1 database call, no matter how many times invoked. It also will make zero database calls if the entire association has already been loaded. The only downside is that "how to determine if primary" logic has to be written once in rails_sql and once in pure ruby.

MySQL query_reviewer plugin

2008-02-13T18:46:00Z

query_analyzer on steriods!

Features

QueryReviewer is an advanced SQL query analyzer. It accomplishes the following goals:

View all EXPLAIN output for all SELECT queries to generate a page
Rate a page's SQL usage into one of three categories: OK, WARNING, CRITICAL
Attach meaningful warnings to individual queries
Find out where the query was executed with a stack trace
Display advanced interactive summary on page

It accomplishes this by injecting an absolutely positioned div into your HTML markup before it's sent out of rails. View injection can be turned off, if necessary and replaced with <%= query_review_output %> somewhere in your view.

Screenshot

View larger

Project homepage

http://code.google.com/p/query-reviewer/

Installing

svn export http://query-reviewer.googlecode.com/svn/trunk/ vendor/plugins/query_reviewer

Then optionally run "rake query_reviewer:setup" and edit config/query_reviewer.yml to change the default settings.

mac.com "Web Gallery" Photos in Mephisto

2008-01-02T21:24:00Z

All the flickr functionality with better iPhoto integration!

I just finished a conversion of my existing flickr ajax mephisto plugin to use mac.com web galleries! I finally ended up paying for mac.com, because of the awesome integration with everything.

Features:

Pull photos from a specific album or your entire gallery
Uses AJAX to pull feed data, so it won't bottleneck your main page rendering
Randomly chooses X photos to display every time
A variety of supported images sizes: square, small, medium, and large
Images automatically link to Apple's awesome web gallery interface

Issues (depending on size of gallery, mine has 400 images at the time):

The RSS feed takes between 2 and 10 seconds to deliver from apple.
The downloaded XML file takes 2 to 5 seconds process.

Usage

How I used it in my site:

<p>Pictures:</p>
{% macgallery feed: http://gallery.mac.com/dsboulder/?webdav-method=truthget&feedfmt=photocastrss  count: 4  format: square  width: 75  height: 75 %} 
{% endmacgallery %}

Supported options to the "macgallery" liquid tag:

feed: REQUIRED. The url of the RSS feed to your gallery, or to a specific album.
count: OPTIONAL (default: 6). The number of images to display.
format: OPTIONAL (default: square). Also valid are "small", "medium", and "large".
width/height: OPTIONAL (default: none). If specified, will add width and height attributes to the IMG tag. Useful for resizing the square images from their 160x160 default size.

Where to get the plugin


svn checkout http://mephisto-mac-gallery-ajax.googlecode.com/svn/trunk/ vendor/plugins/mephisto_mac_gallery_ajax