assay depot - Home

Tanning Bed

2008-12-14T03:25:00Z

We've been working on our search engine lately. We've tried acts_as_solr in the past, and have been using UltraSphinx lately. For reasons I'll detail in a later post, we've considered moving back to solr. Rob Kaufman of Notch 8 has done a lot of great work writing Tanning Bed, a ORM agnostic interface for Solr. From his github page:

Tanning Bed is a generic Solr interface for models, because models are too sexy to get the their tans from the sun directly.

We'll be moving back to Solr in the next few months, and we'll be using Tanning Bed. Check it out.

Merb Camp

2008-10-10T22:40:00Z

I just wanted to write a quick blog post about MerbCamp this weekend. MerbCamp is the first Merb conference. So what is Merb? Well, from the Merb website:

Merb is an MVC framework that is ORM-agnostic, JavaScript library agnostic, and template language agnostic, preferring plugins that add in support for a particular feature rather than trying to produce a monolithic library with everything in the core.

I've been playing with Merb a lot lately, and I'm really excited it.

The Merb sprint is under way right now and Matt, Rob and Ryan tell me that 1.0 RC1 will be released this weekend at the conference.

It looks like it will be a great conference, so go install Merb, and hopefully I'll see you there!

Upgrading Ruby Gems to 1.2.0

2008-08-04T00:37:00Z

I just finished deploying the latest version of the website, and I ran into some trouble related to the gems we use, specifically the version of rubygems we use. I'm posting this here in case anyone runs across a similar problem.

It started when I tried to do the following

cap deploy:migrations

I got the following error:

  * executing "cd /mnt/apps/web/releases/20080802232104; rake RAILS_ENV=production  db:migrate"
    servers: ["ec2-67-202-49-9.compute-1.amazonaws.com"]
    [ec2-67-202-49-9.compute-1.amazonaws.com] executing command
 ** [out :: ec2-67-202-49-9.compute-1.amazonaws.com] (in /mnt/apps/web/releases/20080802232104)
 ** [out :: ec2-67-202-49-9.compute-1.amazonaws.com] rake aborted!
 ** [out :: ec2-67-202-49-9.compute-1.amazonaws.com] undefined method `collect' for #<Gem::Version::Requirement:0xb77cdcf8>
 ** [out :: ec2-67-202-49-9.compute-1.amazonaws.com] 
 ** [out :: ec2-67-202-49-9.compute-1.amazonaws.com] (See full trace by running task with --trace)
    command finished
    command "cd /mnt/apps/web/releases/20080802232104; rake RAILS_ENV=production  db:migrate" failed on ec2-67-202-49-9.compute-1.amazonaws.com

We use a number of gems in our project, and since the error included

undefined method `collect' for #<Gem::Version::Requirement:0xb77cdcf8>

I figured the configuration of those gems was a good place to start. We have a block in our environment.rb that looks like:

config.gem "acts_as_versioned"
config.gem "chronic", :version => "0.2.3"
config.gem "pdf-labels", :version => "2.0.0", :lib => "pdf/label"
config.gem "shipping", :version => "1.5.1"
config.gem "rubyist-aasm", :lib => "aasm"

As an aside, if I removed all the ":version => x.y.x" I got a different error:

undefined method `collect' for nil:NilClass

I guess that makes sense. After getting nowhere for a while, I emailed Rob from Notch8 who out of the goodness of his heart emailed me back even though he was on vacation. He suggested upgrading ruby gems itself.

sudo gem update --system

Interestingly, after the update, /usr/bin/gem didn't work because it was pointing to the old gem libraries. I found the following fixed that.

sudo rm /usr/bin/gem
sudo ln -s /usr/bin/gem1.8 /usr/bin/gem

At this point, my deploy is almost working, however, when it tries restart mongrel I get the following error.

ERROR RUNNING 'cluster::restart': Plugin /cluster::restart does not exist in category /commands

Since upgrading got me this far, I tried upgrading mongrel_cluster

sudo gem install mongrel_cluster

Which worked, and I was able to successfully deploy the application. Thank goodness we have staging server, doing all of that on production would have been a non-starter.

Powermed

2008-05-29T21:14:00Z

By now I'm sure you've tried or at least heard of Powerset. In their own words:

Powerset's goal is to change the way people interact with technology by enabling computers to understand our language.

What that means to you is; instead of just doing a keyword search, Powerset understands what you ask it and returns results based on what you mean. Take the following query as an example: "who wrote mongrel". When you ask Google, you get the following:

Which gives you the information you are looking for, but when you ask Powerset the same question you get:

Interestingly Powerset tells us, right at the top of the page, "Zed A. Shaw wrote Mongrel". Pretty cool.

Limitations

Currently Powerset only searches Wikipedia and Freebase. The reason, according to their website is:

Wikipedia is a primary destination for millions of people worldwide who want to find high-quality, detailed information on a wide array of topics. By initially focusing on the Wikipedia collection, Powerset showcases how our technology not only improves search results, but provides new ways to aggregate, summarize and navigate information. Of course Wikipedia is only a starting point. In the coming months, Powerset will expand our product offerings with additional premium content and exciting new features.

I suspect another reason for starting with Wikipedia and Freebase is, they are both clean datasets. Basically they have a lot of information in a standard format.

Enter PubMed

Another dataset I can think of that is very clean and has lot of useful information is Pubmed. I think if Powerset would index Pubmed it would enable some very interesting searches. For that matter you could index the USPTO database or all the open access journals. There are a lot of possibilities.

Conclusion

I have to say, I've never found anything with Powerset that I couldn't find with Google. That being said, I still like the idea and see the power of it. However, it is possible that if you are asking a question that is so specific or semantically tricky that a keyword search won't work, you probably already have a good idea of the type of information you are looking for. My point is, vertical search is an interesting niche for Powerset. That doesn't preclude them from eventually indexing the entire internet, but I'd love to see a Powermed.com.

Ok, back to RailsConf.

Update Deepak Singh over at bbgm often writes about deriving value from structured data.

Upgrade to Rails 2.1.0_RC1

2008-05-23T05:04:00Z

upgrade (with 2 Ds)

Today seemed like a good day for an upgrade, so I decided to update our app to Rails 2.1.0_RC1. (The real reason is, I wanted to use has_one :through, but it wasn't in my version of Rails!)

I've detailed the steps I took to get our app up and running, this isn't an exhaustive list by any means, but I hope it helps.

Download Rails 2.1.0_RC1

Rail is on GitHub now, which is a great move. Unfortunately, our app is still under subversion and piston doesn't yet support Git, so I downloaded RC1 from http://github.com/rails/rails/commits/v2.1.0_RC1

Since I can't manage it with Piston anymore, I completely removed rails from my vendor directory and replaced it with the downloaded version.

Now its time to run

  rake spec

and see where we are.

STATUS: Dead in the water

ActionView::Base to ActionView::Template

At this point I can't even run rake, I just get the following error:

 
  ERROR
  > rake spec
  (in /Users/cpetersen/project)
  rake aborted!
  undefined method `register_template_handler' for ActionView::Base:Class

  (See full trace by running task with --trace)

It turns out we are registering a template handler for PDFs in our environment. Looks like they moved that functionality out of ActionView::Base into ActionView::Template, to fix all we needed to do is change the following line your config/environment.rb:

12	ActionView::Base.register_template_handler 'pdf', ActionView::PDFRender

to:

12	ActionView::Template.register_template_handler 'pdf', ActionView::PDFRender

and you can at least start running your specs again.

STATUS: LOTS of errors, 706 failures in 1014 tests

will_paginate

We use the wonderful will_paginate plugin to handle pagination in our app. However, when I run my specs, I now get the following:

  ERROR
  705)
  ActionView::TemplateError in 'CatalogController bin should show a compound'
  stack level too deep
  In catalog/compound.html.erb


      vendor/plugins/will_paginate/lib/will_paginate/finder.rb:138:in `method_missing_without_paginate'
      vendor/plugins/will_paginate/lib/will_paginate/finder.rb:139:in `method_missing'
      app/models/ware.rb:67:in `suggested_bins'
      spec/models/bin_spec.rb:32

It turns out, we weren't on the latest release. You can get the latest release here. Once again, its from the hub. The same process I used for installing Rails from github applies here.

STATUS: fewer errors, 30 failures in 1014 tests

acts_as_versioned

Not bad, the tests run, and only about 3% are failing. What next?

Almost all of the remaining errors look like this one:

  ERROR
  30)
  NoMethodError in '/survey_templates/edit should render survey template edit page'
  You have a nil object when you didn't expect it!
  You might have expected an instance of Array.
  The error occurred while evaluating nil.include?
  ./spec/views/survey_templates/edit.html.erb_spec.rb:6:in `new'
  ./spec/views/survey_templates/edit.html.erb_spec.rb:6:

We are using the acts_as_versioned plugin for versioning a lot of our content.

It turns out the core team has done a lot of work on "dirty" objects for this release. Ryan Daigle does a good job explaining it. It turns out this functionality breaks acts_as_versioned.

Luckily for us, codafoo is able to help. He forked acts_as_versioned and made it work with RC1 and dirty objects. You can get his version here. Since I had frozen acts_as_versioned into my vendor/gems directory, I just had to replace it with codafoo's version. Your milage may vary.

You'll notice, its GitHub once again. Note to self, switch to git.

STATUS: almost there, 2 failures in 1014 tests

Proper has_one Validation

Now we are getting down to some nitty gritty errors. It turns out my last two errors were only working before by accident.

Prior to RC1, has_one relationships weren't being validated properly on save. I apparently had my fixtures in an inconsistent state, causing my test to fail. My fault, RC1 helped me find a problem with my fixtures. Here is the ticket. I fixed my fixtures, and on to the next/last.

STATUS: last one, 1 failure in 1014 tests

Minor ActiveRecord::Base.find Change

Finally, there was a change to ActiveRecord::Base.find. I have a has_many :through relationship and I am doing a non-trivial find. I was including the table order_states and referencing its order_status_id column in my conditions. However, I wasn't including the table name in my conditions. Here's the code:

12	@orders = Order.find(:all, :conditions => "order_status_id > 1", :include => "order_states", :order => "orders.created_at DESC")

Prior to the RC1 upgrade that worked, but I should be using the table name. One more case of RC1 finding potential problems in my code. I changed it to:

12	@orders = Order.find(:all, :conditions => "order_states.order_status_id > 1", :include => "order_states", :order => "orders.created_at DESC")

and it worked like a charm. I'm not sure what caused this change, but it may have been part of this change

STATUS: finished.

Thanks to Rob and Ryan for helping my through this process.

See you at RailsConf!

Update - attachment_fu

After I wrote this post, I realized that attachment_fu had stopped working (yet all the tests passed, troublesome). When I tried to upload something, I received the following stack trace:


NoMethodError (undefined method `after_attachment_saved_callback_chain' for #<class:0x52030dc>):
    /vendor/rails/activerecord/lib/active_record/base.rb:1645:in `method_missing_without_paginate'
    /vendor/plugins/will_paginate/lib/will_paginate/finder.rb:164:in `method_missing'
    /vendor/rails/activesupport/lib/active_support/callbacks.rb:272:in `send'
    /vendor/rails/activesupport/lib/active_support/callbacks.rb:272:in `run_callbacks'
    /vendor/rails/activerecord/lib/active_record/callbacks.rb:298:in `callback'
    /vendor/plugins/attachment_fu/lib/technoweenie/attachment_fu.rb:377:in `after_process_attachment'
    /vendor/rails/activesupport/lib/active_support/callbacks.rb:173:in `send'
    /vendor/rails/activesupport/lib/active_support/callbacks.rb:173:in `evaluate_method'
    /vendor/rails/activesupport/lib/active_support/callbacks.rb:161:in `call'
    /vendor/rails/activesupport/lib/active_support/callbacks.rb:93:in `run'
    /vendor/rails/activesupport/lib/active_support/callbacks.rb:92:in `each'
    /vendor/rails/activesupport/lib/active_support/callbacks.rb:92:in `send'
    /vendor/rails/activesupport/lib/active_support/callbacks.rb:92:in `run'
    /vendor/rails/activesupport/lib/active_support/callbacks.rb:272:in `run_callbacks'
    /vendor/rails/activerecord/lib/active_record/callbacks.rb:298:in `callback'
    /vendor/rails/activerecord/lib/active_record/callbacks.rb:208:in `create_or_update'
    /vendor/rails/activerecord/lib/active_record/base.rb:2176:in `save_without_validation'
    /vendor/rails/activerecord/lib/active_record/validations.rb:901:in `save_without_dirty'
    /vendor/rails/activerecord/lib/active_record/dirty.rb:75:in `save_without_transactions'
    /vendor/rails/activerecord/lib/active_record/transactions.rb:106:in `save'
    /vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:66:in `transaction'
    /vendor/rails/activerecord/lib/active_record/transactions.rb:79:in `transaction'
    /vendor/rails/activerecord/lib/active_record/transactions.rb:98:in `transaction'
    /vendor/rails/activerecord/lib/active_record/transactions.rb:106:in `save'
    /vendor/rails/activerecord/lib/active_record/transactions.rb:118:in `rollback_active_record_state!'
    /vendor/rails/activerecord/lib/active_record/transactions.rb:106:in `save'
    /app/controllers/assets_controller.rb:54:in `create'
    /vendor/rails/actionpack/lib/action_controller/mime_responds.rb:106:in `call'
    /vendor/rails/actionpack/lib/action_controller/mime_responds.rb:106:in `respond_to'
    /app/controllers/assets_controller.rb:53:in `create'
    /vendor/rails/actionpack/lib/action_controller/base.rb:1161:in `send'
    /vendor/rails/actionpack/lib/action_controller/base.rb:1161:in `perform_action_without_filters'
    /vendor/rails/actionpack/lib/action_controller/filters.rb:580:in `call_filters'
    /vendor/rails/actionpack/lib/action_controller/filters.rb:601:in `run_before_filters'
    /app/controllers/application.rb:17:in `set_current'
    /vendor/rails/activesupport/lib/active_support/callbacks.rb:173:in `send'
    /vendor/rails/activesupport/lib/active_support/callbacks.rb:173:in `evaluate_method'
    /vendor/rails/actionpack/lib/action_controller/filters.rb:395:in `call'
    /vendor/rails/actionpack/lib/action_controller/filters.rb:598:in `run_before_filters'
    /vendor/rails/actionpack/lib/action_controller/filters.rb:578:in `call_filters'
    /vendor/rails/actionpack/lib/action_controller/filters.rb:573:in `perform_action_without_benchmark'
    /vendor/rails/actionpack/lib/action_controller/benchmarking.rb:68:in `perform_action_without_rescue'
    /usr/local/lib/ruby/1.8/benchmark.rb:293:in `measure'
    /vendor/rails/actionpack/lib/action_controller/benchmarking.rb:68:in `perform_action_without_rescue'
    /vendor/rails/actionpack/lib/action_controller/rescue.rb:201:in `perform_action_without_caching'
    /vendor/rails/actionpack/lib/action_controller/caching/sql_cache.rb:13:in `perform_action'

The solution for us was just to update attachment_fu to trunk. Voila, everything started working again.

Great Visualizations

2008-05-17T01:33:00Z

Visualizing complex data can be hard. How do you get your point across when the data you have to present doesn't fit nicely into a bar chart, pie chart, line graph or histogram?

Over the last couple of weeks, I've come across a number of great visualizations. They cover a range of topics, but they all effectively convey complex data in a very accessible way. Edward Tufte would be proud.

Water Distribution

This is a great visualization, it shows where water is located on earth. It could have been displayed in a bar chart, however, it would have effectively gone to zero by the time you got to fresh water. This method effectively conveys relative distribution down to rivers and plants and animals.

Early Adopters

The following graphic shows us the price of a piece of technology relative to its market penetration over time. As if three variables wasn't enough, it shows all that for 6 separate pieces of technology relative to each other.

Global GDPs

The final visualization shows the GDP of the world's four largest economies after the United States, relative to the US. There are a lot of unanswered questions about this visualization, like is the US economy exactly as big as these four combined? The visualization implies so, but I don't know for sure. Additionally, are the economies distributed on a dollar = geographic size only, or does it imply that the UK has the same GDP as the Southeast United States? Again, I assume the former, but the latter is a possibility. At any rate, the graphic is very effective at conveying the relative size of the worlds top five economies.

Torrey Pines Hike

2008-05-05T19:52:00Z

On Thursday, Assay Depot took a hike through the Torrey Pines State Reserve. Our three conferences were marketing successes and yielded what seemed like nonstop mornings and nights of hard work, so we decided to take an afternoon off. The reserve is home to one of the rarest pines in the world, the Torrey Pine.

Torrey Pines perched on the hill

It is probably one of the few natural reserves in the state where you can also get superior mobile phone reception.

There is a path from the frontyard of our office that leads right to the coast. It was an enjoyable two mile walk on what was an unmatched spring day. However, mild treachery did arise. It seemed that some parts of the path were constructed for hobbits. Had Frodo only gone through this path while being chased by the henchmen of Sauron, he would have guaranteed his safety.

A three foot high tunnel. No horses allowed.

Thankfully, this treachery only lasted for about one minute and paled in comparison to the blooming shrubbery, cactus, and black beetles of Mother Earth.

After an hour, we made it to the coast.

We are seen here relaxing around refreshments and string cheese.

Sorry, but no boogie boarding, wakeboarding, or surfing. However, if you know us, you know that anything can happen. Stay tuned!

SDRuby EC2

2008-05-02T21:48:00Z

Last night I had the opportunity to give a presentation on EC2 to San Diego Ruby User Group. In an effort to keep the talk accessible, I intentionally kept the slides simple and promised to supplement them with a EC2 cheat sheet. Well here it is:

Like I mentioned in my talk, this was intended to be a 1-2 page, maybe 3 page cheat sheet. As I was writing it quickly became a 20 page eBook. I hope you find it useful.

Many thanks to Patrick Crowley for organizing it and to Kevin Clark who also presented that evening. I will update this page when the Podcast is available.

Life Scientists and Social Media

2008-03-27T15:37:00Z

I just got an email from BioInformatics LLC advertising their new new report, The New Collaboration: Social Media and the Life Science Opportunity. It is a great report, and is closely related to what we are doing here at the Assay Depot. A few things jumped out at me while reading it.

Product Information

According to the report, third party online portals are the least trusted source of product information, while company websites are the most trusted. I can understand that, when I'm looking for a computer, I check out Apple's website before I check Amazon. Now I wouldn't characterize Amazon (or the Assay Depot for that matter) as online portals, but we are both marketplaces that act as third parties in transactions between customers and companies.

To mitigate this trust issue, and to make things as easy as possible for our users, wherever possible, we will link the products on our website to the companies own web page about that product.

Objective Feedback

If social media isn't the most trusted source of product information, it is the most trusted source for objective feedback. 45% of respondents said so. This complements our strategy of providing peer reviews of the services and products we offer.

Influence on Purchasing Decision

Section 8 of the report touches on how social media has influenced purchasing decisions. Interestingly it seems that social media has made them more informed regarding their purchasing decisions, but it hasn't made purchasing faster, or changed the purchasing process.

I strongly believe this is going to change. Currently their are no tools that combine social media and purchasing, it is a bit of a disjointed process. People search for information (often using social media) about what they are purchasing. Then they go back to the old system of calling to negotiate prices and legal contracts and faxing purchase orders back and forth. I'm trying not to make this post too much of a "sales pitch", but that is what we are trying to do. Make the purchasing of research services easy by combining e-commerce with customer reviews, links to the actual supplier, and eventually more community and social media capabilities. We just want to make scientists lives easier.

Finally...

There are two more quotes in the report that I found interesting:

Social Media appeals to the fundamental values of science - communication contribution and collaborating.

I couldn't agree more...

and:

In an environment where the average scientist buys products from a dozen or more vendors, making it easier for them to open a dialog with you can can only strengthen their loyalty.

right on again...

SOT 2008

2008-03-24T13:28:00Z

Last week was the Society of Toxicology (SOT) meeting in Seattle Washington. It the first conference where the Assay Depot had a booth, and in my opinion it was a huge success! We met a lot of great people, started to get the message out and talked to a bunch of service providers many of whom sound like promising leads.

Like I said, we don't launch until next month, and as such SOT was a learning experience for us and we learned a lot... for instance:

Spend the extra money and get the thicker carpet, if you are standing for 2 or 3 days straight, it makes a difference
Post-it notes are great give aways, everyone needs them, and if they use them, they are basically posting your logo/url/name for you
It must be good to be in the conference business, I mean $161 to rent a table for 3 days... seriously?!

We will be applying these lessons and more next month at the Society for Biomolecular Screening meeting (SBS), the American Chemical Society meeting (ACS) and the Experimental Biology meeting (EB). If you are going to any of those meeting, come check us out!

A Picture is Worth a Thousand Words

I brought my camera so we could record the event... unfortunately I miscalculated and only brought my fixed focal length lens, so the photos are a little more zoomed in than I would like, but you can judge for yourselves.

Kevin and myself (as well as the laptop I'm typing on right now) in front of our booth

Mei, Andy and Charles also in front of our booth

While we were there, we met some great people, including our booth neighbors. Here are a few of them:

Srinivas Ventrapragada, the CEO of Clintox in front of his booth at SOT 2008

Mike DeGraaf of PharmOptima in front of his booth at SOT 2008

Colin Burton of MDS-USA in front of his booth at SOT 2008

It was great hanging out with you guys, Pike st. was a blast!

And last but certainly not least, I got a chance to speak with Deepak from Business|Bytes|Genes|Molecules in person which was great.

Thank you everyone for a great conference.

Create Your Own EC2 Image

2008-03-10T19:40:00Z

We have been working day and night preparing for our beta launch in April, which is just one of the many excuses I have for not posting in so long. Part of preparing for that launch is setting up the platform that we are going to deploy to. For numerous reasons, we’ve chosen Amazon’s EC2 for our production system. EC2 is a great service, and getting up and running with it isn’t hard at all. However, when it comes to creating your own image, I ran into some barriers, I thought I’d share how we overcame them. I hope you find it useful.

(Side note: Deepak Singh over at business|byte|genes|molecules blogged about an interview with Jeff Barr, an Amazon Web Services Evangelist which I found interesting).

Assumptions

For the purpose of this post, I am assuming your have a working EC2 account, your rsa key is installed properly, and have the basic tools working, ie:

    ec2din
    ec2dim

Here is a good tutorial on how to get up and running.

Starting with EC2 on Rails

In my search for a deployment platform, I came across the EC2 on Rails project. It has a bunch of nice things built in, but for various reasons, their standard image wouldn’t work for us. Nevertheless, we decided to use their image as our starting point. You can get their current AMI’s by running the following command (assuming you have the ec2onrails gem installed):

    cap ec2onrails:ami_ids

Currently, the 32-bit AMI is ami-e620c58f. Next we start an instance and login:

    ec2run ami-e620c58f --instance-type m1.small -k gsg-keypair
    ec2din
    ssh -i ~/.ssh/id_rsa-gsg-keypair admin@ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com

In order to rebundle the image, you will need your pk and cert on the server. I store mine in the /mnt/aws-config directory, so lets create that directory. From the server run:

    sudo mkdir /mnt/aws-config
    sudo chown admin.admin /mnt/aws-config

Next, upload the files from your local computer

    scp -i ~/.ssh/id_rsa-gsg-keypair ~/.ec2/*.pem admin@ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com:/mnt/aws-config

To make things easier in the coming steps, I add the pk and cert filenames to my environment. On the server, edit your .profile to include the following lines (I add them before ./usr/local/ec2onrails/config, but I’m not sure it matters):

    KEY_FILE_NAME=pk-[your pk filename].pem
    CERT_FILE_NAME=cert-[your cert filename].pem
    AWS_ACCOUNT_ID=[your account id]

Next we have to prepare to rebundle. The EC2 on Rails image comes with a rebundle script. However, I was never able to make it work, so I broke out the necessary steps here:

    curl http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip > /tmp/ec2-api-tools.zip
    sudo unzip /tmp/ec2-api-tools.zip -d /usr/local
    sudo chmod -R go-w /usr/local/ec2-api-tools*
    sudo ln -sf /usr/local/ec2-api-tools-* /usr/local/ec2-api-tools
    sudo aptitude install -y sun-java6-jre

Insert your configuration here

At this point, you are ready to configure the sever however you see fit. For us it was installing some support software, etc. Although you may want to complete the tutorial first and make sure you can rebundle properly before investing too much time!

The /mnt directory

So at this point you have your perfect server, all you need to do is store the image so you can preserve it forever. There is one last item to deal with, the mnt directory. Whenever you ec2kill an instance and start a new one from the image, the /mnt directory get cleared, which makes sense. You need some place to store all the transient data (like logfiles and deployment versions) that don’t belong in the image. However, somethings in there (like directories for log files) need to exists for the image to start properly. So what’s a person to do? Well I decided on a scheme where I tar up a skeleton on the mnt directory, and wrote an install script for it. That way when I start a new instance, I log in, run ./install.sh and I’m up and running.

Create your mnt directory skeleton

Next I copy the /mnt directory to my home directory and get it just the way I like it (at least get rid of the openssh_id.pub and lost+found).

    sudo cp -R /mnt ~
    cd ~/mnt
    sudo rm lost+found
    sudo rm openssh_id.pub

Now tar up your directory:

    sudo tar cvfz ~/mnt.tgz ~/mnt

Next we need to write the install script. Mine looks something like (don’t forget to make it executable):

    #!/bin/sh

    sudo tar xvfz mnt.tgz
    cd mnt
    sudo chown app.app -R app
    sudo chown mysql.mysql -R mysql_data
    cd log
    sudo chown syslog.adm *
    sudo chown www-data.www-data -R apache2
    sudo chown mysql.mysql -R mysql
    sudo chown root.root -R fsck
    cd ..
    sudo mv * /mnt

    sudo /etc/init.d/mysql start
    sudo /etc/init.d/apache2 start

That’s it, your instance is now configured and ready, now for the good stuff.

Finally, its time to bundle

First we create the image:

    sudo ec2-bundle-vol -u $AWS_ACCOUNT_ID -c /mnt/aws-config/$CERT_FILE_NAME -k /mnt/aws-config/$KEY_FILE_NAME

Next, we need to upload the image to S3, (you must create an S3 bucket prior to this step):

  
    ec2-upload-bundle -b [your-bucket] -m /tmp/image.manifest.xml -a [your access key] -s [your secret access key]

The final step is to register your image with Amazon and get your AMI:

  
    ec2-register -K /mnt/aws-config/$KEY_FILE_NAME -C /mnt/aws-config/$CERT_FILE_NAME [your-bucket]/image.manifest.xml

Putting it all together (AKA, the test)

So you’ve now created your bundle. The true test is, can you start it, login, run your install script, and have a fully functional server. From your local machine:

    ec2dim # You should see your new AMI
    ec2run [AMI from previous command] --instance-type m1.small -k gsg-keypair
    ec2din
    ssh -i ~/.ssh/id_rsa-gsg-keypair admin@ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com

Now run the install script from the remote machine:

    ./install.sh

You should now have a fully functional server.

Bonus

I created two aliases on my local machine that i found very helpful, maybe you will too:

    alias ec2ssh="ssh -i ~/.ssh/id_rsa-gsg-keypair" 
    alias ec2scp="scp -i ~/.ssh/id_rsa-gsg-keypair"

Coming Attractions

Like I said earlier, we are launching our beta in April, so expect to see a lot of changes to the site shortly. One of those changes will include moving from WordPress to Mephisto. if your feed reader resets, I apologize in advance.

Geocoding in Ruby

2007-11-21T14:23:00Z

The Geocode Module

Have you ever needed to turn a partial address into a full one? Have you ever wanted to get the latitude or longitude for a particular address? Maybe you want to plot addresses on a Google Map, or maybe, like a colleague of mine, you find it redundant to ask for city, state AND zip when creating a form. If any of these apply to you, you might be interested in geocoding.

Geocoding is the process of turning an address (or in some cases a partial address) into latitude and longitude. Often times during the process you get more information about the address. For instance if you geocode "1000 5th Ave, 92101", not only with you get the latitude and longitude (32.715714, -117.160158), you also find out that it's in San Diego, CA.

There are many services out there that will geocode addresses for you. For instance the Cartographer plugin provides a couple of nice Geocoding modules. One for Geocoder.us and one for Ontok. Both are great alternatives for geocoding. However, I wanted to use Google, and I didn't want to use the full Cartographer plugin. So I wrote the following module that allows you to geocode addresses in Ruby using Google's service.

module Geocode
  def geocode(input)

    uri = "/maps/geo?q=#{input.gsub(/\s/, "%20")}&output=xml&key=<your google map's key>"
    res = Net::HTTP.start("maps.google.com") {|http|
      http.get(uri)
    }

    begin
      doc = REXML::Document.new(res.body)

      if doc && doc.root
        coordinateNode = doc.root.get_elements("//coordinates").first
        addressNode = doc.root.get_elements("//ThoroughfareName").first
        stateNode = doc.root.get_elements("//AdministrativeAreaName").first
        cityNode = doc.root.get_elements("//SubAdministrativeAreaName").first
        cityNode = doc.root.get_elements("//LocalityName").first unless cityNode
        zipcodeNode = doc.root.get_elements("//PostalCodeNumber").first
  
        longitude,latitude = coordinateNode.text.split(",") if coordinateNode
        address = addressNode.text if addressNode
        state = stateNode.text if stateNode
        city = cityNode.text if cityNode
        zipcode = zipcodeNode.text if zipcodeNode

        return {
          :latitude => latitude.to_f,
          :longitude => longitude.to_f,
          :description => nil,
          :original_address => input,
          :score => nil,
          :street_number => nil,
          :prefix => nil,
          :street_name => nil,
          :street_type => nil,
          :suffix => nil,
          :address1 => address,
          :city => city,
          :state => state,
          :zipcode => zipcode
        }
      end
    rescue
      # Do something useful...
    end
    # if nothing has returned yet, then return nil
    return nil
  end
end

You'll notice a few returned items that are always nil, that's because it is a drop in replacement for the Geocoding modules in Cartographer. So even if you use the Cartographer plugin, you can still use this module.

Using the Geocode Module

To use the Geocode module, simply include it, and pass it an address or partial address.

1234	include Geocoderesult = geocode("92121")puts "#{result[:address1]}, #{result[:city]}, #{result[:state]}, #{result[:zipcode]}"puts "#{result[:latitude]}, #{result[:longitude]}"

Stats on Rails

2007-11-18T15:19:00Z

The Stats Module

Have you ever had an array of numbers and needed to know if one was a statistical outlier? Ever needed the standard deviation of those numbers? Well, I was recently working on an app, where I did. I packaged up the functionality as the following Module.

module Stats
  def mean_stdev_n(numbers)
    sum = 0
    sumsq = 0
    n = 0
    numbers.each do |num|
      if num
        sum += num
        sumsq += (num*num)
        n = n+1
      end
    end

    if n==0
      return nil,nil,0
    else
      mean = sum/n
      stddev = Math::sqrt(sumsq/n - mean * mean)
      return mean, stddev, n
    end
  end
 
  def outlier?(value, mean, stddev, n)
    grubbs = grubbs_outlier_test(value, mean, stddev)
    z = critical_z(n)
    return grubbs > z
  end
 
  def grubbs_outlier_test(value, mean, stddev)
    ((mean - value)/stddev).abs
  end
 
  def critical_z(n)
    case(n)
    when 3
      1.15
    when 4
      1.48
    when 5
      1.71
    when 6
      1.89
    when 7
      2.02
    when 8
      2.13
    when 9
      2.21
    when 10
      2.29
    when 11
      2.34
    when 12
      2.41
    when 13
      2.46
    when 14
      2.51
    when 15
      2.55
    when 16
      2.59
    when 17
      2.62
    when 18
      2.65
    when 19
      2.68
    when 20
      2.71
    when 21
      2.73
    when 22
      2.76
    when 23
      2.78
    when 24
      2.80
    when 25
      2.82
    when 26
      2.84
    when 27
      2.86
    when 28
      2.88
    when 29
      2.89
    when 30
      2.91
    when 31
      2.92
    when 32
      2.94
    when 33
      2.95
    when 34
      2.97
    when 35
      2.98
    when 36
      2.99
    when 37
      3.00
    when 38
      3.01
    when 39
      3.03
    when 40..49
      3.04
    when 50..59
      3.13
    when 60..69
      3.20
    when 70..79
      3.26
    when 80..89
      3.31
    when 90..99
      3.35
    when 100..109
      3.38
    when 110..119
      3.42
    when 120..129
      3.44
    when 130..139
      3.47
    else
      3.49
    end
  end
end

If you are unfamiliar with the grubbs test... well you're not alone, neither was I. I found this page to be especially helpful.

Usage

To use the module, simply include it, calculate mean, stddev and n, then use the outlier? funtion.

include(Stats)
numbers = # some array of numbers
mean, stddev, n = mean_stdev_n(numbers)
if outlier?(5, mean, stddev, n)
  puts "5 is an outlier"
else
  puts "5 is not an outlier"
end

Uninterrupted Sirius Stream - Greasemonkey for IE

2007-11-17T11:43:00Z

Introduction

I’ve been a fan of satellite radio for a few years now. When we opened the doors to our lab space, it wasn’t after the first computer was plugged in that we had Sirius radio streaming.

Recently Sirius rolled out an updated version of their web-based player with the introduction of a timeout feature. Every few hours your have to click on the window to continue listening. Let’s see if we can’t get rid of this new feature for a click-free listening experience that will last all day long.

For those already familiar with tools like Greasemonkey this example is trivial; however, if you have never played with client-side javascript tweaks it might be worth a look. If you are a developer writing production javascript, this just reinforces the public nature of content or logic you send client-side.

Greasemonkey was designed as a plugin for Firefox and there are hundreds of scripts available through userscripts.org. The plugin allows you to query/add/modify web content via javascript as you see fit.

We’re pretty much a Firefox shop at assaydepot, but Sirius is something I’ve always played through IE. It used to be the case that only IE was supported although that’s no longer true. One reason I justify leaving Sirius on IE is the frequent restarts I usually make with Firefox during the week. With a few dozen tabs, Firebug, and who knows that else running Firefox has a habit of getting a little out of control with its memory usage. Restarting Firefox doesn’t interrupt my IE listening experience :)

There is a Greasemonkey for IE adaptation that is somewhat different from the Firefox version, but nonetheless effective. That’s what I used for this project.

The Fix

A quick look at Sirius’s javascript shows the use of a timeout

window.setTimeout("isUserStillListening()", stillListeningTimeout);
...
function isUserStillListening() {
...display a div and require user-interaction to continue listening...
}

This callback displays a hidden div that the user then clicks on. My first hack checked the display style attribute of this timeout div, if inline it then clicked on the div continuing playback. I then woke up and just overrode the callback with a null function that leaves the player running. GM4IE scripts are executed at the end of the page – sirius sends a full featured function that GM4IE overwrites with nothing at the end of the page.


function isUserStillListening(){}

That’s it. To install this one line of js I installed GM4IE, created a GM4IE config file and the javascript file with this function.

The GM4IE script installer from File didn’t work right for me so I manually edited the extention.list and whella – uninterrupted tunes once again!

GM4IE Sirius Continue Listening script

Instantly add wiki functionality to your Rails app

2007-10-07T20:56:00Z

Introduction

Its been a while since we've posted. That's because we've been busy preparing for our beta launch. Toward that end, we've hired Rob Kaufman of Notch8 to strengthen our development effort. This post is about one of the plugins that has come out of that work, wiki_column. wiki_column, allows you to add wiki functionality to any (textual) attribute of any model. Wiki links in that text will then refer to the model that you define as your wiki, the wiki model defaults to WikiPage. Any attribute you define as a wiki_column, will also get textile formatting automatically, for this reason wiki_column requires the RedCloth gem.

How to install it

Let's assume you want to add a wiki to your Rails app (why else would you be here?). Let's also assume that you have a Product model that has a description attribute that would like wiki enabled. Given those assumptions, we'll start at the beginning. First install the RedCloth gem:

sudo gem install RedCloth

Next install the wiki_column plugin.

./script/plugin install http://wiki-column.googlecode.com/svn/trunk/wiki_column

Next generate your WikiPage model and your Product model.

./script/generate scaffold_resource WikiPage slug:string body:text created_at:datetime updated_at:datetime
./script/generate scaffold_resource Product name:string description:text price:decimal created_at:datetime updated_at:datetime

Don't forget to migrate.

rake db:migrate

Now we add the wiki functionality to your models, and validate that our slugs are unique. In your WikiPage model, add the following lines.

12	validates_uniqueness_of :slugwiki_column :body

Similarly, add the following line to your Product model.

wiki_column :description

Next we need to change the show views. wiki_column adds a new method to your model called wiki_#{column}. So in your show.rhtml for WikiPage, change:

<%=h @wiki_page.body %>

<%= @wiki_page.wiki_body %>

Do the same for Product descriptions so they will link directly to your new wiki. Lastly, want our url to make sense to users. Our scaffold resource is called WikiPage, because we are adding/editing and viewing individual pages. However, to a user viewing our site, the following url would make more sense.

http://company.com/wiki/1

Thus, the last step is to add a new resource to the routes.rb file

map.resources :wiki, :controller => 'wiki_pages'

That's it, you should now have a functioning wiki and your products should integrate with it nicely.

How to use it

Now that your application is wiki enabled, how do you make use of this new found functionality? You already know you can use the full range of textile commands, they do a better job of explaining their syntax than I could. What wiki_column adds is the ability to specify wiki words. Those wiki words should be surrounded by square braces. For example, if you wanted to link to a page with the slug ProductionInformation, your code would look like

[ProductInformation]

Conclusion

We designed wiki_column to be easily integrated into new or existing Ruby on Rails applications, as well as to be easy for your customers to use. As alway your comments, critiques, suggestions, and especially patches are welcome. Please post below.