Locomotivation

We've Moved!

Thu, 12 Nov 2009 17:06:28 -0500

Hopefully, you’ve already figured out that we’ve moved all our blogging to the main Squeejee blog.

We decided to make this change in an effort to unify our efforts around the Squeejee brand, rather than splitting the tech stuff out into something else.

So head on over to http://squeejee.com/blog for all the latest goodness!

TweetSaver.com Launched on MongoDB, MongoMapper

Tue, 29 Sep 2009 11:41:13 -0400

As you all probably well know by now, we’re big fans of MongoDB here at Squeejee. We’ve written about it, spoken about it, and ported mysql based apps over to it.

We’ve recently launched our first application built from the ground up on MongoDB. TweetSaver also happens to be the first production app that we know of utilizing John Nunemaker’s MongoMapper ORM.

How MongoDB Made Things Easy

Look, Ma, no schema!

Mongo’s schema-less nature made it easy to store all sorts of data from a number of APIs. Grabbing rich, deep data structures was easy because we didn’t have to relationally model the relationships between documents and subdocuments. We could simply ‘stash the hash.’

Upserts!

One of the coolest features of MongoDB is the ‘upsert.’ In ActiveRecord we do this with find_or_create_by_xxx. This performs two calls — a select and insert — for new records. MongoDB allows you to peform fire-and-forget saves by upserting:

myColl.update( { name: "Joe" }, { name: "Joe", age: 20 }, { upsert: true } );

where the name ‘Joe’ is your unique field.

How MongoDB Made Things Hard

Embedded object, foreign key, or DBRef?

One of the toughest things about going NO-SQL is having to think about the best way to model data for your application. Although MongoDB shines in storing deep, nested document structures, it also fully supports more relational database design. This choice means that you need to think through the tradeoffs up front. It’s a good idea to go through the scenarios in which you’ll consume pieces of data and determine how to store it.

Plugins / Gems built on ActiveRecord

Since MongoMapper is not ActiveRecord, all of those Rails gems and plugins that have ActiveRecord dependencies will not work. Your choices here are:

(a) Go with a MySQL hybrid approach to store a subset of data in MySQL. For example, we store our Users table in MySQL to take advantage of the number of Authentication plugins out there. (RestfulAuth / AuthLogic / TwitterAuth/ etc). However, even this is pretty simple to implement with MongoDB. Here is a Gist from John Nunemaker to implement authentication with his MongoMapper ORM: http://gist.github.com/147427

(b) Port over a solution to work with MongoDB. The nice part about this route is that there are no migrations to futz with. Just define your needed fields directly in your app, plugin, or gem.

Major Gotcha’s

It’s easy to play around with Mongo, see its speed and think that the laws of computing have been suspended. Alas, collections with hundreds of thousands of records still require indexing.

So, before you start your next project take a minute to see if MongoDB might be a good fit… it’s been a great addition to the Squeejee toolbox.

Ready, FIGHT! TwitterLand now includes ThumbFight support

Sun, 06 Sep 2009 14:03:09 -0400

Thanks to Ron Evans for contributing ThumbFight support for our TwitterLand gem.

ThumbFight measures how people liked or disliked search terms based on the last one hundred tweets for the term. The results can be shown in a head-to-head matchup like this one for “Ruby vs. Python”.

To use the ThumbFight API, just upgrade to TwitterLand version 0.3.0: sudo gem install twitterland

My favorite of all the new Google Reader features released this...

Fri, 14 Aug 2009 13:00:00 -0400

My favorite of all the new Google Reader features released this week. Now catching up on older posts is not quite all-or-nothing.

— @pengwynn

Do the Monster Mash!

Fri, 14 Aug 2009 09:00:00 -0400

Working on several API wrappers lately, one of my favorite gems is Mash from Michael Bleigh. Much like OpenStruct, Mash is basically a Hash with dot notation.

But unlike OpenStruct, you can pass a Hash to Mash and it will recursively do a deep dive and Mashify it. Now you get pretty accessor methods for all those parsed hashes from your JSON API calls:

# Return tweet referencing a URL
results = Twitterland::BackTweets.search('http://squeejee.com', 'OU812')
results.tweets.size
=> 25
results.tweets.first.from_user
=> "euromarianne"
results.items_per_page
=> 25
results.total_results
=> 3301

i_am_ron_burgandy?

Mash includes a couple of helpful methods for traversing those deep hashes when you’re not certain if a key has been set:

mash = Mash.new
mash.first_name?
=> false
mash.first_name = 'Wynn'
mash.first_name?
=> true

The ! method provides multi-level assignment, kind of like mkdir -p for hashes:

mash = Mash.new
mash.these_droids!.will_do_nicely = true

Install the gem and mash it up

sudo gem install mash

— posted by Wynn Netherland // @pengwynn

Google Ad(makes no)Sense

Wed, 12 Aug 2009 11:57:00 -0400

Today we received an email from Google AdSense letting us know they had disabled our AdSense account. The only explanation we got was that we were a “Significant risk to their AdWords advertisers”. Check out the full email.

Hello,

While going through our records recently, we found that your AdSense
account has posed a significant risk to our AdWords advertisers. Since
keeping your account in our publisher network may financially damage our
advertisers in the future, we’ve decided to disable your account.

Please understand that we consider this a necessary step to protect the
interests of both our advertisers and our other AdSense publishers. We
realize the inconvenience this may cause you, and we thank you in advance
for your understanding and cooperation.

If you have any questions about your account or the actions we’ve taken,
please do not reply to this email. You can find more information by
visiting
https://www.google.com/adsense/support/bin/answer.py?answer=57153.

Sincerely,

The Google AdSense Team

Google’s email is cryptic to say the least. It would have been nice to at least know the specific violations we commited that put their advertisers at risk. Not only did they cancel our account but they also confiscated our AdSense earnings. I just don’t understand the draconian nature of their actions. If we did something that violated the TOS then tell us. If we knew what we did wrong I promise we would make it right. I did file an appeal and will keep you updated on our communication with Google.

TwitterLand - All the best Twitter-related APIs in one gem!

Fri, 07 Aug 2009 18:48:00 -0400

Update - Added TweetBlocker API support
Update - Added BackTweets API support

Here at Squeejee we do a lot of data mashups, and more and more of those are mashups of Twitter and 3rd-party services related to Twitter.

There are a bunch of Twitter-related sites out there that have great APIs to pull in their data. Since we’ve been using so many of them repeatedly in our projects we finally decided to pull them into a single gem. TwitterLand will give all you other Twitter app developers out there easy access to these great APIs in your ruby projects.

So far we’ve included:

Install

 sudo gem install twitterland

Follow Cost Usage

# Get follow cost for specified user
Twitterland::FollowCost.show('bradleyjoyce')

=>

Twitter Grader Usage

# request your api key at [http://twitter.grader.com/accessrequestform](http://twitter.grader.com/accessrequestform)
# get twitter grade for user
api_key = "OU812"
Twitterland::TwitterGrader.grade('bradleyjoyce', api_key)

=> 98.4183

Mr.Tweet Usage

get your api key at http://api.mrtweet.com/newapi

Initialize Mrtweet

api_key = 'OU812'
mt = Twitterland::Mrtweet.new(api_key,'bradleyjoyce')

Is user

# Check whether the given user is a MrTweet user.
mt.is_user

=> true

Profile

# Returns MrTweet statistics of the given user
mt.profile

=> 

mt.profile.links

=> 0.53

Recommendations

# Returns the latest recommendations the given user received on MrTweet
mt.recommendations

=> []

mt.recommendations.first.name

=> "billtrammel"

Most attention towards

# Returns the twitter_id's of 3 users that for the given user pays the most attention to
mt.most_attenion_towards

=> [15049040, 17993906, 22286046]

Recommend

# Creates a recommendation from the given user, to another user (aka "friend_name")
reason = "Wynn is an awesome entrepreneur, rubyist, designer and friend! Follow him for his useful and entertaining tweets!"

friend_name = "pengwynn"

mt.recommend(reason,friend_name)

=> true

Twinfluence

Initialize Twinfluence

username = 'bradleyjoyce'
password = 'mypassword'
t = Twitterland::Twinfluence.new(username,password)

=> #

User

# gets twinfluence data for user
t.user('bradleyjoyce')

t.user('bradleyjoyce')

=> , ]> centralization="24.224443556856" centralization_grade="0.0 Average - Resilient" description="Entrepreneur and Web Developer -- Floxee.com, TweetCongress.org, Sherflock.com, Squeejee.com" followers_count="1315" friends_count="1278" id="14688076" location=nil name="Bradley Joyce" screen_name="bradleyjoyce" second_order="7936664" second_order_grade="Rank: #10,185 (93%)" social_capital="6035.4859315589" social_capital_grade="+1.4 High" statuses_count="3511" unix_timestamp="1249683954" url="http://bradleyjoyce.me" velocity="15535.453112211" velocity_grade="+0.1 Fast Average">>

Twitter Counter

Show

tc = Twitterland::TwitterCounter.show('bradleyjoyce')

tc.rank

=> 37194

# available methods
tomorrow_2w 
followers_2w_ago 
followers_yesterday 
followers_current 
friends_current 
next_month 
growth_since_2w  
started_followers 
rank 
user_id 
growth_since 
follow_days 
tomorrow 
next_month_2w 
average_growth 
average_growth_2w

TweetBlocker

User

# get grade for user
result = Twitterland::TweetBlocker.user('bradleyjoyce')

=>

Spam

# report a user as spammer
Twitterland::TweetBlocker.report_spam('spamtest')

Rate Limit Status

# check your rate limit status
Twitterland::TweetBlocker.rate_limit
=>

BackTweets

Search

# Return tweet referencing a URL
results = Twitterland::BackTweets.search('http://squeejee.com', 'OU812')
results.tweets.size
=> 25
results.tweets.first.from_user
=> "euromarianne"
results.items_per_page
=> 25
results.total_results
=> 3301

Source

http://github.com/squeejee/twitterland/

Documentation

http://rdoc.info/projects/squeejee/twitterland

Copyright

posted by @bradleyjoyce

Remixr - Ruby gem for the BestBuy Remix API

Wed, 05 Aug 2009 13:38:07 -0400

The Ruby BestBuy Remix API gem. Remix is an API that gives you access to BestBuy.com’s product catalog data and more. Remixr is a Ruby wrapper for the Remix API.

Install

sudo gem install remixr

Usage

Remixr.api_key = 'OU812' # get yours from http://remix.bestbuy.com/apps/register
client = Remixr::Client.new

Find stores

# find stores within 50 miles of ZIP 76227
stores = client.stores({:area => ['76227', 50]}).fetch.stores

stores.first

=> {"city"=>"Denton", "longName"=>"Best Buy - Denton", "name"=>"Denton", "region"=>"TX", "address"=>"1800 S Loop 288, Ste 102 Bldg 1 ", "country"=>"US", "lng"=>-97.10067, "postalCode"=>"76205", "phone"=>"940-384-9581", "hours"=>"Mon: 10-9; Tue: 10-9; Wed: 10-9; Thurs: 10-9; Fri: 10-10; Sat: 10-10; Sun: 11-8", "storeId"=>827, "fullPostalCode"=>"76205", "lat"=>33.192524, "distance"=>9.79}

Find products

# fetch first page of products on sale below 20 bucks
products = client.products({:salePrice => {'$lt' => 20.00}}).fetch.products

# fetch only SKU and salePrice 
products = client.products({:salePrice => {'$lt' => 20.00}}).fetch.products

Chaining

You can also chain stores and products to return stores and nested product info or vice versa

# find stores within 50 miles of ZIP 76227 and products over three G's

stores = client.stores({:area => ['76227', 50]}).products({:salePrice => {'$gt' => 3000}}).fetch.stores

stores.first.products.first.shortDescription

#=> "ENERGY STAR Qualified 4 HDMI inputs; gray Touch of Color bezel; 16:9 aspect ratio"

Fetching

All calls terminate in a call to fetch which takes the following options

:page - positive integer for page number
:show - comma delimited string or array of field names to show
:sort - hash or string of sort info {'fieldname' => 'asc|desc'}

Conditional operators

We took a page out of MongoDB’s playbook and mapped conditional operators to text equivalents to avoid having these be keys to hashes:

{:salePrice => {'$gte' => 300.00}}
=> salePrice > 300.00

$gte - greater than or equal to : field > value
$lte - less than or equal to : field > value
$gt - greater than : field > value
$lt - less than : field > value
$ne - not equal to : field != value

Documentation

http://rdoc.info/projects/squeejee/remixr

Copyright

Twitter apps sing on MongoDB

Fri, 24 Jul 2009 17:42:00 -0400

As we’ve previously mentioned, at Squeejee, we’re big fans of MongoDB. We’re excited that some very smart people are starting to take notice of Mongo, too.

In this first post of what we hope will be a helpful introductory series, we’ll cover the use case of using a document store like MongoDB.

Mashups are all the rage

It’s rare that we are presented with a new project from a prospective client these days that doesn’t pull data from some third-party API. Twitter seems to be the most popular choice of late, perhaps because it adds value both as a content provider and a transportation medium. One of the challenges of integrating with Twitter in any large-scale way is creating your own local cache of Twitter data so you can ration those precious API calls and avoid their API rate limiting. When we built our own mashup, Tweet Congress we wanted to integrate with a number of third-party Twitter services that each pulled data from Twitter, analyzed it, and passed that value on via their own APIs. The common thread was that each of these services exposed data as JSON (JavaScript Object Notation). Taking these deep, rich data structures and breaking them out into relational database tables seemed silly (not to mention tedious). With the relational approach, if a service exposed a new value that we wanted to consume without fighting our framework, we’d have to extend the schema on our end to capture it.

Schema-less is more

MongoDB is a high-performance, open source, schema-free document-oriented database. This means we stash those deep data structures we get back from those nifty web services more easily. MongoDB uses BSON to store your documents (or objects). This BSON smells less like a buffalo and more like JSON, which you already know. The main difference is that BSON can serialize binary data (images, and even Word documents, oh the irony!) and dates. This schema-less storage means your app can store all that new data TwitterCounter returns you tomorrow without needing to change your database.

Hey, ever heard of CouchDB?

Certainly CouchDB (and many other tools that predate it), does document storage and does it well. We are still big CouchDB fans, however we found that querying CouchDB dynamically was a lot like getting a question answered at the DMV. CouchDB shines when you know how you’re going to query the data up-front, in static views of schema-less data.

MongoDB provides all that schema-less goodness without giving up any of the rich dynamic querying you get with a relational database. More on this in our next post.

Back to our Twitter mashup

Let’s talk specifically about Twitter for a moment. The Twitter API has methods to retrieve the social graph for a particular user, returning an array of integer user IDs for the user’s friends or followers. With a relational database, you’d normally insert a row for each of those values. So a user with a thousand friends would get a thousand database rows, each with something like user_id and friend_id (and id if you’re using some Rails magic). With MongoDB, we can just stash that array with the object itself — and query it: :conditions => {:friend_ids => current_user.id}

Fields on-the-fly

Another benefit of all this schema-less goodness is your application can define fields at run-time. On our Floxee platform, we support different custom fields on a per-site basis. Due to the magic of MongoDB, we can have custom fields for party, state, and district for all the congressfolk in Tweet Congress and fields like skills and soda_preference on our own Squeejee twitter directory. Just upload your CSV with some custom columns and we’ll create those fields on-the-fly (after sanitizing them of course).

More to come

In our upcoming posts we’ll cover Ruby bindings, performance, deployment, and more. Stay tuned!

TextMate Pro Tip: Use F2 for bookmarks

Tue, 21 Jul 2009 08:54:00 -0400

One of the things I love about TextMate is that it’s not an IDE, and it feels lightning fast. That said, sometimes I forget to use some of the powerful features in TM. Here’s one I decided to start using:

Cmd-F2 to toggle a bookmark

F2 to navigate to the next bookmark

Shift-F2 to navigate to the previous bookmark

How many times have you scrolled back and forth between two (or more) chunks of code using the mouse wheel, Cmd-F, Cmd-G, page up, page down, or the dreaded scrollbar drag which makes your “mouse mileage” go up?

Just use F2 and you can quickly flip between your bookmarks. Note: You can also use the mouse to toggle the “*” that appears on your bookmarked lines of code.

Pro Tip: On a Mac laptop, consider changing this setting: System Preferences -> Keyboard & Mouse -> Use all F1, F2, etc as standard function keys to “checked”. That way you won’t have to hit the “fn” key to use the magical F2!

Carpal tunnel syndrome be damned!!

Global Settings With Mongo and Rails in 15 Lines of Code

Wed, 01 Jul 2009 01:52:00 -0400

There is a nice ActiveRecord Rails plugin called rails-settings that does the following:

Settings is a plugin that makes managing a table of global key, value pairs easy. Think of it like a global Hash stored in you database, that uses simple ActiveRecord like methods for manipulation. Keep track of any global setting that you dont want to hard code into your rails app. You can store any kind of object. Strings, numbers, arrays, or any object.

We have used this plugin in the past for global key / value pairs and it works great. However, for our Floxee app, we are using MongoDB / MongoRecord so this plugin would not work.

Lucky for us, MongoDB is already a document store so I was able to recreate similar code in about 15 lines of code inside of a “method_missing” method.

With this, we are now able to do things like:

>> AppSetting.jim = "is cool"
=> "is cool"

>> AppSetting.arr = ["jim", "was", "here"]
=> ["jim", "was", "here"]

>> AppSetting.arr.first
=> "jim"

>> AppSetting.hsh = {"hash_1"=>1, "hash_2"=>"demo"}
=> {"hash_1"=>1, "hash_2"=>"demo"}

>> AppSetting.hsh["hash_2"]
=> "demo"

MongoDB - Ruby friendly document storage that doesn't rhyme with ouch

Tue, 02 Jun 2009 22:30:27 -0400

I had the opportunity to share MongoDB with the Dallas.rb group this evening. Thanks @mully and @jnunemaker for contributing slides!

MongoDB - Ruby document store that doesn’t rhyme with ouch

View more PDF documents from Wynn Netherland.

Same train, new station

Mon, 01 Jun 2009 10:18:22 -0400

At RailsConf recently we met a lot of folks who gave us kudos on this blog. Many were surprised to learn that Locomotivation is brought to you by the Squeejee team.

So, we’ve decided to integrate our code blog a bit more with our web site. Sorry if the move caused your RSS reader to panic. Hopefully, once the dust settles we’ll have more regular content from our crazy ride on Rails.

Installing CouchDB From Source On Mac OS X

Thu, 12 Mar 2009 00:00:00 -0400

Wynn and I started dabbling with CouchDB this week. As with any technology, step one when dabbling is installing the software.

For Mac OS X Leopard Intel users, there is a very nice CouchDB packaged zip file that can be downloaded from the Couch Projects Google Code site. The huge positive of going this route is that it is drop dead simple to install. Simply download the .zip file, extract the code, and double-click the “CouchDBX” file. You then simply click the “Play” icon and you are off and running with CouchDB.

The negative going this route is that, as the time of this blog post, this package runs CouchDB version 0.8.1 which is about 7 months old.

In order to run the new 0.9 version of CouchDB, you will need to pull the latest source code from the SVN trunk, install some prereqs, and compile from source. Here are the exact steps I took to get up and running.

NOTE: For details on each of these steps, you can look at the README file that is downloaded with the couchdb source code.

From the command line:

Browse to the following URL in a browser to view your CouchDB Futon admin tool:

http://127.0.0.1:5984/_utils/

Ruby on Rails Integration Testing with Integrity

Wed, 25 Feb 2009 00:00:00 -0500

I did some research on what was new in the Continuous Integration testing field in regards to Ruby this week. We have used CruiseControl.rb in the past which worked fine, but I remember the setup being a bigger pain than I thought it had to be. I was hoping for something simpler.

With this in mind, I decided to give Integrity a shot. Their title of “The Easy and Fun Automated Continuous Integration Server” was exactly what I was looking for. After installing, configuring, and playing around with it for the past day or so, I have not been disappointed.

With that being said, I did have to jump around to a few different sources to get everything running correctly on a server with Phusion Passenger, so I thought I would document the steps I had to take here.

Installation:

This will create an Integrity application directory with a couple files on your server, mainly config.yml, config.ru, and a “public” directory we will point our Apache virtual host to.

Edit config.yml to suit your needs. This file is very will commented and your edits should be self-explanatory. I kept the default Sqlite database configuration to keep things simple. However, due to this, I had to install Sqlite on our Ubuntu server via:

Setup a DNS entry to point to this server and create an Apache virtual host as you would any other Rails or Sinatra web application:

Once this is all complete, simply restart apache to access your new Integrity instance.

Project Creation:

Create a Test Database:

(You can skip this test if you can use Sqlite databases for testing. Our code inevitably has MySql specific SQL syntax in it which requires us to use MySql for our tests.)

Create a MySql user that will be used for Integrity tests:

Note that “IDENTIFIED BY PASSWORD” is optional. If it is not included, your user can log in without a password.

Project Creation:

Assuming the above was successful, you should now be able to browse to the Integrity URL you created. From here, click the “Add new project” link.

Name: Unique name for the project you will be tracking
Git Repo: .git URL of the project you will be tracking. For example, for an internal project on GitHub, the url would be something like: git@github.com:acme/my_cool_application.git
Branch: The git branch Integrity should be tracking
Build Script: The rake command to use for testing after the code is pulled into your CI (Continuous Integration) server.
- rake spec (for rspec)
- rake test (for test unit)
- rake features (for Cucumber)
- etc

Customizing Your Build Script

The build script can be a tad bit tricky. It is our practice not to keep database.yml in Git for password security purposes. Because of this, Integrity will not know the test database configuration settings needed to run the tests. To get around this issue, we created a database.yml.test file with the db credentials Integrity would need to run a test similar to this:

Using MySql:

Using Sqlite:

In my Rails project, I then created the below custom Rake task in “lib/tasks/my_cool_application.rake”:

Once this is done and the files are added/committed/pushed to git, go back to your edit project page in Integrity and change your “Build Script” to be “rake test:move_test_db_config features” for Cucumber tests, for example.

Manual Build

After all of the above is complete, you should be able to click the “manual build” link for your new project to pull your code and run your tests!

Automated Tests Via Git Service Hooks

Integrity tests are automated with Git Service Hooks. Adding a service hook is drop-dead simple with GitHub. To do so:

Browse to your project in GitHub
Click on the “Admin” tab
Click on “Service Hooks” in the header
In the “Post-Receive URLs” box, add a URL similar to this:
- http://username:password@integrity.example.com/my_cool_application/push
- where “username” and “password” are the HTTP authentication credentials setup in the integrity/config.yml file

Once this is setup, your CI Integrity test will be run everytime you code is pushed to the appropriate branch in GitHub.

Notifiers

Integrity also provides a notification system to inform you when tests are run.

Current notifiers include email, Jabber, Campfire, IRC, Twitter, and Basecamp. These are installed independently of Integrity via Gems. Once a gem is installed on your CI server, just log back into your project within your Integrity web site and setup your notifier credentials. You are also free to write your own custom notifiers for applications that are not supported yet.

Resources:

Thanks to the following sites for assisting me with my original setup and this blog post:

Workstreaming with Present.ly

Sat, 20 Dec 2008 00:00:00 -0500

Having fled the cubicle jungle of Corporate America, we have a strong distaste for status reports. That’s not to say we don’t believe in letting others on the team know what we’re doing. We just prefer to workstream our activities and let our work announce itself to the rest of the team.

Present.ly - our workstreaming tool

I hate to put Present.ly in a box by calling it “Twitter for Business” (because it is so much more than that), but that’s the easiest way to describe it.

Familiar Twitter feel

If you’re comfortable using Twitter, you’ll feel at home with Present.ly because it supports @replies, direct messages, and hashtags.

Questions and urgent messages

Present.ly adds some smarts to the display of messages or “tweets.” Messages that end in a question mark are displayed as questions to the group:

Need to mark a message as urgent? Simply add three exclamation marks:

Private and secure

Unlike Twitter, only your users see your messages.

Groups!

I really hope Twitter builds functionality for groups one day and when they do, I hope they rip off Present.ly’s implementation. Admins control who belongs to what groups. Perhaps the coolest feature of Present.ly groups is the broadcast. Broadcasts are like direct messages to a group except they are displayed as if sent from the group itself, which gets its own avatar. The following broadcast message

b system Elvis has left the building

would display as if it came from the System itself:

How we use it

Much of our Present.ly use is still manual. We tweet each other questions and comments just like we do on Twitter. In fact, Present.ly let’s you cross-post to twitter by using the t! command:

t! This message would go to Twitter, too!

But, increasingly, we’ve automated much of our workstreaming by sending broadcast messages at certain points in our development cycle.

Source control checkins

One of the most important things we do as a development team is add code to source control. We use “GitHub”:http://github.com for our code repositories. GitHub has a nifty feature called Service Hooks, which allows you to send a message to a number of third party services for code commits to your repo. Setting up Present.ly really easy:

Once set up, GitHub will send a new broadcast message from the group you specify with each new code push. The message includes the commit message and a link to the commit back on GitHub:

Deploy notices

Before Present.ly we had been sending our deployment notices to Campfire using Tinder. While this works well, we really preferred to have our workstream in one spot and leav eour Campfires for discussion and chat. Since Present.ly implements the Twitter API, wiring up the deploy notices from Capistrano was a piece of cake.

Now everybody knows when the build is done:

How are you using it?

We’d love to hear how you’re using Present.ly in your workflow? Any cool uses we’re missing?

People, sites, and software I am thankful for this year

Thu, 27 Nov 2008 00:00:00 -0500

I’ve been building web applications for a dozen years now and I’m convinced there has never been a better time to be a web developer! I can’t imagine doing my job if it weren’t for the impact of some great people, sites, and software. This post isn’t a Ten Biggest style article. I won’t be thanking Tim Berners-Lee, although we all should.

Rather, this list is simply the extraordinary people and things that have made my job a joy over the past year.

People I am thankful for

John Resig

Creator of jQuery, JavaScript ninja, John casts a large shadow on the web development landscape. Looking at John’s code has taught me more about JavaScript than every book I’ve ever bought on the subject.

Chris Wanstrath aka defunkt

Half of the err the blog duo, a third of the GitHub posse, Chris is the most energetic Rubyist I’ve seen. Just watching the amount of software his company puts out will either inspire you or send you into a shame spiral.

The Less Everything guys, Steve and Allan

Makers of cool, free, and downright funny software, these guys write more less code than anybody around. I’m thankful that guys like this are around to keep us from taking ourselves too seriously.

Jean-Philippe Lang

I’ve never seen anyone single-handedly will a project into existence like JP. While there is no doubt that Redmine has a large and growing community behind it, I’m sure everyone would agree that Redmine exists because of the sheer will of Jean-Phillipe.

Sites I am thankful for

GitHub

No other technology has impacted me more this year than Git, and I don’t think it would have happened without GitHub. We first signed up for GitHub because we wanted a turnkey central repository for our projects. But GitHub is so much more. GitHub brings social networking to source control. This means you can see what other folks are working on and discover new projects and tools all the time. It also means it’s never been as easy to contribute to open source. Want to improve an open source project? With GitHub, you just fork it and send a pull request. If the original owner goes off the grid, no worries, you’ve got your own repo and keep on chugging.

Twitter

I’ll have to admit at first I didn’t get Twitter. I mean, take a look at the public timeline. Who cares when random folk are bathing? I’ve since learned that there is no better tool for finding those other folks that are just as weird as you. You know, those long lost members of your tribe(s). Folks who are interested in those things you bug everyone else about.

Presently

Soon after I climbed on the Twitter train, I longed for something as easy as Twitter to workstream all those source control checkin notices, build statuses, and “what I am doing” updates. The folks at Intridea have hit a homerun with Present.ly. The groups and hashtag functionality is awesome. Try it for your team!

Software I am thankful for

Redmine

We’ve used a lot of issue trackers. Some great ones. Some workhorses. But we’ve never used anything as extensible as Redmine. Plugins, themes, and custom fields, oh my!

Webby

I want to thank Tim Pease for building such a cool little tool. I’m using Webby to write this post!

Blueprint CSS

Blueprint has been a part of my standard toolbox since it came out. Blueprint brings sanity to my layout and typography while make it so easy developers can do CSS!

This year I’ll be looking to Compass to semantify my Blueprint.

jQuery

I wasn’t in the market for a new JavaScript framework when I discovered jQuery, but I quickly fell in love with this one. jQuery lets me do more powerful things with less code than any other framework I’ve used. With comprehensive CSS3 (and bonus) selectors, chaining, and plugin architecture it’s a joy to use.

How about you?

What are the people, sites, and projects you’re thankful for this year?

Javascript and Ruby Frameworks at Lone Star Ruby Conf 2008

Sun, 07 Sep 2008 00:00:00 -0400

I had the opportunity to give a talk at Lone Star Ruby Conf 2008. There are a lot of hardcore Rubyists here writing great web applications in Rails, Merb, and Waves. It was fun to knock around ideas with so many folks about how Javascript fits into the mix.

View SlideShare presentation or Upload your own. (tags: ruby javascript)

Until the slides are posted on the official LSRC site you can grab them from SlideShare.

Simplify ActiveRecord Aggregates and Other Goodies via named_scope

Mon, 25 Aug 2008 00:00:00 -0400

We’ve been huge fans of named_scope since its introduction in Rails 2.1. If you are new to the named_scope concept, please read Ryan Daigle’s intro to named_scope and watch Ryan Bates’ named_scope RailsCast. Ryan Daigle also had a good follow up article explaining his utility scope gem.

With a recent project, we really started utilizing extensions with named_scope which is very powerful and cleaned up our code considerably.

For example, let’s say you have an issue tracking system that has an Entry model that belongs_to both a User model and a Project model. Using named_scope, we can easily access Entries for a date range filtered to both the Project and User level using named_scope lambdas:

To find the Entries for the last week for the first user, we would simply do:

u = User.first
u.entries.by_date(1.week.ago, Time.now)

To find the Entries for the last week for the first project, we would simply do:

p = Project.first
p.entries.by_date(1.week.ago, Time.now)

What happens if we want to see the sum of hours worked for a date range? This is where extensions come in very handy. Let’s add a “total_duration” extension to our named_scope:

Now to get the duration for the past week for the first user we simply do:

u = User.first
u.entries.by_date(1.week.ago, Time.now).total_duration

with similar logic at the project level.

To group all of the entries by project, we would do:

If you need the lambda arguments inside your named_scope extension, you will need to use “self.proxy_options”. For example, let’s say you need the from and to dates in order to calculate the average duration per weekday. That code would look something like this:

In the above example, “weekday_count” is a custom method we created to calculate the number of weekdays between 2 given dates.

The opportunities to simplify code using named_scope is great. As I was writing this post, one of our team members, Jason Derrett, created a named_scope to wrap a Xapian full text search into a Lesson model:

In this example, using Xapian to search for a keyword becomes as simple as:

Lesson.find_with_xapian "fishing"

How cool is that?!?

Are there any other cool named_scope tricks out there that any of you are using?

Simple Ruby on Rails Full Text Search Using Xapian

Wed, 23 Jul 2008 00:00:00 -0400

In my previous blog post, I wrote about the several full-text search engines that are available to us as Rails programmers and how we ultimately decided to go with Xapian.

Background

I am by no means a Xapian expert. I first heard about it less than a week ago. However, we have been very impressed with its combination of ease of installation and robust features.

Xapian is written in C++ and has been around in its current form since 2001. Its code base is derived from a previous search tool that went closed source.

Written in C++
Feature rich including relevance feedback, boolean search operators, stemming, wildcard searching, and similar results

Its user list includes del.icio.us and GMane both of which are indexing tens of millions of records, so scalability should not be an issue. Here is a quote about scalability from the site:

People often want to know how Xapian will scale. The short answer is “very well” - a previous version of the software powered BrightStation’s Webtop search engine, which offered a search over around 500 million web pages (around 1.5 terabytes of database files).

Searches took less than a second.

The largest recent installation we’re aware of is probably gmane, which currently indexes over 50 million mail messages.

On the other hand, indexing time could take a while if you have a huge set of documents. According to this article, it took about 5 hours to index the entire Wikipedia database using Xapian for offline Wikipedia access and searching.

Acts_As_Xapian is the Rails plugin for the Xapian. It was written in May 2008 by Francis Irving after he experienced some frustrations with Solr. Acts_As_Xapian has a google group at http://groups.google.com/group/acts_as_xapian.

Xapian Installation:

For Ubuntu Users:

sudo apt-get install libxapian15 libxapian-ruby1.8

For Mac / Other: (taken from the xapian.org install docs)

(version = 1.0.6 at the time of this writing)

wget http://oligarchy.co.uk/xapian/1.0.6/xapian-core-1.0.6.tar.gz
wget http://oligarchy.co.uk/xapian/1.0.6/xapian-bindings-1.0.6.tar.gz

Alternately if you haven’t installed wget on the mac:

curl -O http://oligarchy.co.uk/xapian/1.0.6/xapian-core-1.0.6.tar.gz
curl -O http://oligarchy.co.uk/xapian/1.0.6/xapian-bindings-1.0.6.tar.gz

tar zxvf xapian-core-versionnumber.tar.gz
tar zxvf xapian-bindings-versionnumber.tar.gz

cd xapian-core-versionnumber
./configure --prefix=/opt
make
sudo make install

If you don’t have root access to install Xapian, you can specify a prefix in your home directory, for example:

./configure—prefix=/home/jenny/xapian-install


cd xapian-bindings-
./configure XAPIAN_CONFIG=/opt/bin/xapian-config
make
sudo make install

Acts_As_Xapian Installation:

Install the plugin: script/plugin install git://github.com/frabcus/acts_as_xapian.git

Generate and execute the acts_as_xapian migration:

script/generate acts_as_xapian
rake db:migrate

Basic Model Code

To include fields to your index, you will need to add an “acts_as_xapian” call to the model you want to index.

class Lesson < ActiveRecord::Base
  acts_as_xapian :texts => [:name, :description]
end

In this case, only the data found in the “name” and “description” attributes of our Lesson model will be searched.

A Quick Test Of Your Index

At this point, you can quickly test that your Xapian install, plugin, and model code are all working together via the following rake commands:

To build the index:

rake xapian:rebuild_index models="Lesson" RAILS_ENV=development

(You can add more models with a space delimeter. “Lesson User Tag”) To update index:

rake xapian:update_index RAILS_ENV=development

To test index:

rake xapian:query models="Lesson" query="golf" RAILS_ENV=development

The Basic Search

Adding the this line of code @search = ActsAsXapian::Search.new([Lesson], @lesson_search_params)

will return a ActsAsXapian::Search object with the following (From the acts_as_xapian README):

description – a techy one, to check how the query has been parsed
matches_estimated – a guesstimate at the total number of hits
spelling_correction – the corrected query string if there is a correction, otherwise nil
words_to_highlight – list of words for you to highlight, perhaps with TextHelper::highlight
results – an array of hashes each containing:
- :model – your Rails model, this is what you most want!
- :weight – relevancy measure
- :percent – the weight as a %, 0 meaning the item did not match the query at all
- :collapse_count – number of results with the same prefix, if you specified collapse_by_prefix

So, if we want to return an ActiveRecord model object we could do something like this: @lessons = @search.results.collect {|r| r[:model]}

To get the recommended spelling for a misspelled word(s) in a search, we would do: @corrections = @search.spelling_correction

Similar Results

If you have a requirement to show similar or “You may also like” results based on a search result set or a single result, this can be done with a single line of code:

@similar_lessons = ActsAsXapian::Similar.new([Lesson], @lessons).results.collect {|r| r[:model]}

where @lessons in this case would be the list of lessons returned in the original search.

Updating The Index

Unfortunately, one drawback to Acts_As_Xapian is that the indexers are not updated automatically when data in an indexed model is added, modified, or deleted. When model data is changed, Xapian will put a record in the table ‘acts_as_xapian_jobs’ in your database to notify the update task to update the index. A cron job or something of the like will be needed to call the following rake task periodically to keep the index up-to-date:

rake xapian:update_index

Francis, the acts_as_xapian creator, commented on this on my original Mulling Over Our Ruby On Rails Full Text Search Options blog post.

with Xapian you can update and search simultaneously, and updates are immediate. However, only one thread can update a Xapian database at the same time. Since I wanted offline indexing anyway (as my index operation is risky, complex and slow, involving parsing Word documents, PDFs etc.), I didnâ€™t try to find a solution that causes a second thread in the web application to, say, wait for the database lock. So acts_as_xapian currently only supports offline indexing.

Conclusion

This is a very basic tutorial meant to highlight the steps needed to get up and running with simple Xapian searches quickly. There are quite a few topics that I did not touch on such as:

Sorting and grouping with the :values option in the model
Advanced searching with the :terms option in the model
Adding extended attributes from other models into a searched index. (Such as adding lesson comments to the Lesson index)
Filtering searched data (in our case, we had to filter our results so only “active” lessons were returned)
Highlighting keywords in the search results