Paul Dix Explains Nothing

Parse Domains from URLs easily with Domainatrix

2009-12-30T16:14:39-05:00

I've released another library just in time for the new year. The new library is called Domainatrix and she's used for parsing domain names, canonicalizing URLs, and a few other things. It uses the list of domain names from the Public Suffix List to know what constitutes a subdomain, domain, and public suffix.

Usage couldn't be simpler.

url = Domainatrix.parse("http://www.pauldix.net")
url.public_suffix  # => "net"
url.domain # => "pauldix"
url.canonical # => "net.pauldix"

url = Domainatrix.parse("http://foo.bar.pauldix.co.uk/asdf.html?q=arg")
url.public_suffix # => "co.uk"
url.domain # => "pauldix"
url.subdomain # => "foo.bar"
url.path  # => "/asdf.html?q=arg"
url.canonical # => "uk.co.pauldix.bar.foo/asdf.html?q=arg"

It's simple, but I think quite useful. Enjoy!

Resources for Synchronous Reads, Asynchronous Writes at RubyConf 2009

2009-11-21T16:56:19-05:00

Here are some links to articles and libraries that I mentioned in my talk at RubyConf 2009.

Here are links to images used during the talk.

Finally, here are the slides with the images removed.

Synchronous Reads Asynchronous Writes RubyConf 2009

View more presentations from pauldix.

Typhoeus, the best Ruby HTTP client just got better

2009-10-29T12:29:29-04:00

I've been quietly working on Typhoeus for the last few months. With the help of Wilson Bilkovich and David Balatero I've finished what I think is a significant improvement to the library. The new interface removes all the magic and opts instead for clarity.

It's really slick and includes improved stubing support, caching, memoization, and (of course) parallelism. The Typhoeus readme highlights all of the awesomeness. It should be noted that the old interface of including Typhoeus into classes and defining remote methods has been deprecated. I'll be removing that sometime in the next six months.

In addition to thanking everyone using the library and everyone contributing, I should also thank my employer kgbweb. If you're a solid Rubyist that likes parsing, crawling, and stuff, or a machine learning guy, or a Solr/Lucene/indexing bad ass, let me know. We need you and we're doing some crazy awesome stuff.

Bypassing wxWidgets error when building Erlang from Source on OS X

2009-10-26T11:23:13-04:00

I had the following error when trying to build Erlang from source on OS X:


make[3]: *** [i386-apple-darwin9.8.0/wxe_events.o] Error 1
make[2]: *** [opt] Error 2
make[1]: *** [opt] Error 2
make: *** [libs] Error 2

After a quick search I uncovered this post about building without wxWidgets support (which is fine by me for running RabbitMQ Server.) So putting it all together, building and installing Erlang looks like this:


# in the erlang source directory
./configure
touch lib/wx/SKIP
make
sudo make install

Detect and Parse Atom and RSS Feed URLs from HTML with TruffleHog

2009-10-23T11:55:31-04:00

TruffleHog is a small library I wrote for detecting and parsing Atom and RSS feeds in a web page. I'm sure there are other libraries out there, but I couldn't find one easily. I found Cory Forsyth's FeedDetector, but there were some cases that it didn't handle.

Anyway, here is everything you need to get running.

gem install truffle-hog --source http://gemcutter.org

# and then in teh rubys
require 'rubygems'
require 'truffle-hog'

# get atom and rss
feed_urls = TruffleHog.parse_feed_urls(some_html)

# get atom if available, otherwise rss
feed_urls = TruffleHog.parse_feed_urls(some_html, :atom)

# get rss if available, otherwise atom
feed_urls = TruffleHog.parse_feed_urls(some_html, :rss)

Behind the scenes it's grabbing the urls through a regex. Really just a few lines of code. I may update it with some fancy Nokogiri parsing and some other stuff, but for now it's just a simple way to parse out feed urls.