Technoblog

One of a kind laptop

2009-09-18T10:03:00.000-07:00

Please help! I am trying to sell my wife's pink powerbook and it is only going for $0.11 with no reserve!!! This is an awesome laptop and works great. If you know anyone, please show them the link.

http://cgi.ebay.com/ws/eBayISAPI.dll?ViewItem&item=250498868476

Apple 15" MacBook Pro 2.4Ghz Core2 Duo 4GB RAM 200GB HD

2009-09-02T12:29:00.000-07:00

Hello all, I am selling my 1 year old Macbook Pro to get a new one. The one I am selling is in great condition and has great software like photoshop cs3 and flash cs3. It has no reserve, so bid early and bid often.

http://cgi.ebay.com/ws/eBayISAPI.dll?ViewItem&item=250489787806

Compiling SWFUpload on Mac OS X

2008-11-10T15:25:00.000-08:00

SWFUpload is a great flash library for dynamic file upload progress meters, but making changes to the swf can be a pain in the ass if you don't have FlashDevelop, a windows-only app for compiling actionscript. Luckily I figured out how to do it on OS X.

First download flex sdk 3 from http://opensource.adobe.com/wiki/display/flexsdk/Download+Flex+3

Next download SWFUpload from http://code.google.com/p/swfupload/downloads/list

Now cd into the SWFUpload v2.x.x Core/Flash directory.

Modify the file obj/SWFUpload-v2Config.xml to change the windows paths. Here is what mine looks like:


<!--Flex compiler config for project SWFUpload-v2 generated by FDBuild-->
<!--============-->
<!--This file was generated by a tool.-->
<!--Any modifications you make may be lost.-->
<flex-config>
 <compiler>
   <source-path append="true">
     <path-element>/Users/lcarlson/SWFUpload v2.2.0 Core/Flash</path-element>
   </source-path>
 </compiler>
 <file-specs>
   <path-element>/Users/lcarlson/SWFUpload v2.2.0 Core/Flash/SWFUpload.as</path-element>
 </file-specs>
 <default-background-color>#FFFFFF</default-background-color>
 <default-frame-rate>15</default-frame-rate>
 <default-size>
   <width>300</width>
   <height>300</height>
 </default-size>
</flex-config>

Finally, all you have to do is this:

~/Downloads/flex_sdk_3/bin/amxmlc -load-config+=obj/SWFUpload-v2Config.xml -debug=true -incremental=true -benchmark=false -o swfupload.swf

Now you have compiled a new swfupload.swf. Woot.

Java Doesn't Scale, Official Proof!

2008-07-11T13:33:00.001-07:00

Oh my goodness, MobileMe has not been working for 3 days now, and it is built with Java. I guess it is official, Java does not scale.

This is the exact same logic as most people that use the terms Ruby and Twitter in the same sentence use. Interesting to look at it from the other foot.

It simply couldn't be that they are both trying to solve difficult problems, it must be the language or framework they are using.

DataMapper Review and Impressions

2008-04-21T12:39:00.000-07:00

On the first day, DHH made ActiveRecord, and ActiveRecord was good. Sure it wasn't originally thread safe and it created monster SQL queries updating every single column on a save even if only one value changed. Sure it may have used a ton of RAM and made you write a bit more SQL than you would have liked. But it worked and was good.

Today, there is a lot of buzz around Merb + DataMapper. I will review Merb later but I would like to let people know a few things about DataMapper.

DataMapper has a lot of promise. It is thread safe, smart about SQL updates, has a better interface for generating conditions for SQL queries. Unfortunately, it is in a very strange place right now.

DataMapper's gem is at 0.3.1. Unfortunately, the entire 0.3 release is all but completely unsupported by any community. Everyone in the DM community is working on the 0.9 release. That would be fine except for the fact that 0.9 is currently so far from working that you couldn't use it in production even if you wanted to.

So DM 0.3 is unsupported... that's ok as long as it works right? Well unfortunately, DM 0.3 has various very severe bugs that I can't get anyone in the DM community to care about... even though I have submitted patches to their bug tracking system.

When you declare an :order in a has_many association, it does not work at all... it re-orders the entire collection arbitrarily
The belongs_to association does not work if you have changed your primary key names
External database contexts do not work at all even though there are examples of them on the home page
DM 0.3 claims to be backwards compatible with ActiveRecord, yet you can not use your :from, :joins, or :group conditions when composing SomeClass.find(...) calls

And these are only the ones I found within 1 week of using DataMapper.

The thing that really scares me is that nobody seems to care. Any time I mention anything like this in the IRC channel, the standard response is: sorry, we are all working on 0.9, we have no idea about 0.3.

So in short, 0.3 is unstable with no supporting community behind it and 0.9 is completely unusable. Unfortunately, that is a bad combination in my book. I have branched 0.3 so that I can keep a stable version for myself, and if anyone wants to contribute you can find it at github. I am using it because I like it and know my way around the 0.3 source code so I can fix the large issues I encounter, however I would not recommend DM unless you are a seasoned Ruby developer who is not afraid to roam on your own inside of an ORM framework.

Book Review- The Ruby Programming Language

2008-03-07T16:18:00.000-08:00

David Flanagan, Matz, and _why teamed up to write The Ruby Programming Language. This book is in kind of an interesting purgatory right now. This book covers Ruby 1.8 and 1.9 which is both an asset and a problem. It is hard to recommend this book to a beginner because Ruby 1.9 will not be in mainstream use for quite some time now and learning about the new features can be more confusing than useful, but it is also hard to recommend this book for people who know Ruby well since there is no clear differentiation of 1.9 code, thus you can't use it as a reference for what's new.

The obvious comparison for this book is the seminal pickaxe v2 book. Although the Ruby Programming Language is well written and enjoyable, it does not cover Ruby as thoroughly as pickaxe. I could see the Ruby Programming Langauge v2 with a full reference section of 1.9 (or even 1.8+1.9 with some clever way of showing the differences) taking over my recommendation for #1 most important book for any Ruby programmer in a few years, but until then pickaxe is still required reading.

One thing that I found interesting was the decision not to cover many of the common standard libraries like CGI, logger, test/unit, or net/* (there is 1/2 page out of 400 dedicated to net/http). These are some of the oversights I hope will be fixed in v2, though the authors may choose to keep the discussion more pure and keep the book focused on blocks and operators. Personally, I think that would be a shame not to get into more detail.

Overall, I like this book, but feel like the only person that would benefit from it right now are those who know Ruby at a somewhat higher than beginner level already, but want to deepen their knowledge to a more advanced level and stay ahead of the curve with 1.9 at the same time.

Adding a comprehensive reference section with clear differentiation between 1.8 and 1.9, and more _why illustrations (one per chapter is way too meager) could make this book invaluable.

Book Review- Ruby on Rails: Enterprise Application Development

2008-03-07T15:23:00.000-08:00

I was just sent a copy of Ruby on Rails: Enterprise Application Development from Packt Publishing. I had never read a Packt book before, but I must say, this one impresses me. The tag line of this book states:

"Building a complete Ruby on Rails business application from start to finish"

This book delivers. One of the reasons I like this book so much is that it talks about using Rails in one of the best types of situations: internal-facing utility apps. These are the type of applications I first started building using Rails professionally and one of the places Rails shines brightest. Rails lets you quickly throw together utility applications that adds value to a business, yet maintain a clean codebase, and this book shows you how.

The book covers everything from source control, Rails basics, choosing a CGI implementation, choosing a database, AJAX development, and does a very good job of showing you both the OS X and Windows side of Rails development. It is centered around a story that is familiar to many small businesses: sharing a dynamic contact list within a sales team. It teaches you how to think about the problem in a structured way, including deciding on a database architecture and key format.

The thing I like best about this book is that it is terribly pragmatic. It steps you through a common situation and brings up the complexities just as they would show up in the real world and the best practices to handle those complexities.

I think the target audience for this book is anyone who is tired of spaghetti code but intimidated by seemingly scary terms like MVC. For years I have been begging some of my friends to try Rails instead of PHP, but for various reasons they don't get out of their comfort zone. If a friend has kept begging you to try Rails and it seems like too much effort, pick up this book.

As with most Rails books, this one does suffer from a few deprecations since publication and a few detail oversights. For example, pagination is now a plugin, so a newbie hitting that chapter could get confused quickly. In the book's coverage of various reverse proxies, it does not mention one of the key reverse proxies for Rails environments under high load: HAProxy, which can hide concurrent requests from Mongrel. A finer point is that in the example on uploading files, there is no note that when the file being uploaded is a certain small size, it is instantiated as a StringIO object, not a Tempfile object, which means that the File.basename will raise an error.

Overall, I would definitely recommend this book for people new to Rails.

Really Useful Ruby Mysql One Liner

2007-08-22T17:21:00.000-07:00

Mysql::Result.send(:include, Enumerable)

This one line makes interfacing with mysql results SO much easier.

require "mysql"
my = Mysql::new("host", "user", "passwd", "db")
res = my.query("select * from tbl")
res.each do |row|
 col1 = row[0]
 col2 = row[1]
end

This is from the standard documentation for the Ruby Mysql lib. What does it do? It grabs all items from tbl, and sets col1 and col2 to the first and second column of the LAST value of the result set. Not very useful unless you add a LIMIT 1 to your SQL statement. Much more useful would be this modified code.

require "mysql"
Mysql::Result.send(:include, Enumerable)
my = Mysql::new("host", "user", "passwd", "db")
res = my.query("select first_name, last_name from users")
full_names = res.map do |row|
 "#{row[0]} #{row[1]}"
end

Here we select all users from the table and end up with an array of full name strings. How did including Enumerable help us at all? It defined the map function for us, which natively does not exists for Mysql::Result objects. Why Enumerable is not included in the Mysql::Result class is a mystery to me.

Advanced Concepts in Ruby on Rails Hosting Part IV

2007-07-09T10:01:00.001-07:00

In these past weeks, we have discussed the transition from a standard reverse proxy (represented by a single manager handing documents to be translated to various translators, one at a time as soon as they came in) to a system I call drproxy with clients running on all application servers buffering requests and handing them to instances of the application as soon as they were ready (represented by office managers buffering documents for the translators).

It seems as though we have created a pretty streamlined system during these weeks, but there is still a bottleneck. Most people will never reach the bottleneck, but many people will feel pain around it. That bottleneck is the request server (represented by the manager who hands documents to office managers). The request server is very fast and can take many requests per second without flinching, however it is a single point of failure. If it goes down, nothing gets through. Not only that, but adding a second request server doubles the potential number of incoming requests. In the analogy, if the main manager stays home sick, no documents get translated that day. Drproxy is built to be able to run various request servers very easily, both for load distribution and redundancy.

Drproxy has been built in the crucible, we have been using it at MOG for over a month now, tweaking and making improvements. I am working on bundling the software, which will be open source, and when we release it should be very ready to use out of the box for most Rails websites. Thanks for following this series and I really hope you enjoy drproxy.

Advanced Concepts in Ruby on Rails Hosting Part III

2007-07-02T12:23:00.000-07:00

In our discussion of distributing web requests to different servers via the analogy of a translation company, we ended up last week with a question. To recap, the analogy compares application serving computers to translation offices and instances of the application as translators. Further, a manager (reverse proxy) sat in front of the offices distributing tasks to each office. Last week we realized that to increase efficiency of our translators, we could put a manager in each office in order to buffer requests. That way, any individual bottleneck could not hold up the queue from being processed. For distributing web requests, I created a reverse proxy called drproxy to do exactly this.

These types of systems can be found over and over again in the real world. Just this weekend I was in line to order a polish dog from Costco. There was a single line with two servers processing the line. I watched as a father couldn't get decisions from each of his three children, yet the other server's booth moved along smoothly and brought me closer and closer to the polish dog. You can find similar lines at Nordstrom Rack, Fry's Electronics, any restaurant, and many other locations.

One system that works like the less efficient "round-robin" method described a few weeks ago are lines found in grocery stores. You get in a line praying that the people in front of you don't like to write checks or count out change because if they take their time, they hold you up. How many times did you choose what looked like the fastest checkout line only to watch people go through other lines faster due to one price-check on isle five? Some grocery stores have started implementing self-checkout systems. I find that whenever given the choice, I tend to go directly to the self-checkout because it is always faster. One of the reasons it is faster is because there is a single line for four checkout machines. You could get one person counting change, one person doing a price-check and still have two machines checking people out smoothly. It is amazing that grocery stores have not realized this and implemented better line processing.

The question I posed at the end of last week was whether there was an even more efficient way to distribute requests. I propose that there is. Here is why: just as any individual translator might get a backed up queue of requests, translation offices could become overwhelmed. Based on pure probability, one office might build up a queue of 100 requests while another might sit there queue-less. There are a few ways to tackle this problem, but whichever way you choose to handle it, you must know the size of the queues in each office at any given time. Given this knowledge, you could choose to only hand requests to the least busy of the offices. I am not a big fan of this approach because it seems like you could imagine situations where large groups of documents go to the same office in a row and I like to distribute requests randomly to prevent buildups and attacks on the system. The way that I implemented drproxy to distribute requests was by randomly picking any off! ice, except for the busiest office. If one office builds up a backup of requests and the others have no requests, no new requests will be sent to the busy office until it frees up. This load balancing system works very effectively.

I hate to do this to you again, but can you think of any other major bottlenecks in our system? I can.

Advanced Concepts in Ruby on Rails Hosting Part II

2007-06-25T10:32:00.000-07:00

Last week, we were discussing the analogy of serving websites being similar to running a translation company. A request would come in as a document, be handed to an application server as a translator, and returned to the client. We left off with a scenario of three translation offices with 10 translators in each office. One of the simplest methods to distribute work among these translators is to hand out documents one at a time in a round-robin way. However, due to inherit traits of certain documents being longer than others and certain translators being faster than others, backups build up for some of the translators, leading to a random lag and customer complaints.

Rather than the brute force method of adding more offices and translators, can you think of a better way to distribute resources?

The bottleneck in the scenario sketched above is management. Our translation company still has only one manager, thus limiting his ability to distribute resources more effectively. If we hire office managers and let the manager hand documents to the office managers, this lets us think of more interesting distribution techniques. For example, instead of overwhelming our translators with a growing pile of documents, and thus a growing pile of responsibilities, the office manager can wait until each translator has finished their job before handing them a new document.

Let us think about the consequences of this change. First some assumptions. Assume John is faster at translating than Susie because he has less on his mind (in computer lingo this would mean that Susie is experiencing a memory leak, possibly due to a bad programming library). Further assume a pile of documents comes in with this order: a 10 page document, a 2 pager, a 20 pager, a 1 pager, a 3 pager. In our original setup we could easily find ourselves in the situation where Susie gets a pile with the 10 pager, the 20 pager, and finally the 3 pager; whereas John only got the 2 pager and 1 pager. You can see that Susie's 3 pager should have been easy and fast, but was stuck behind a few bigger documents and is in the hands of the slower translator.

With the new distribution algorithm, the worst case scenario would be that Susie would be chugging away at the 20 pager, but since John quickly made chump change of the other documents, he can turn over the the 3 pager before Susie even finishes the 20 pager. This is much more streamlined because the queue was processed as quickly as the resources freed themselves up as a group, not relying on the individual translator to handle the concurrency.

The typical Rails setup of a reverse proxy handing requests to mongrel is not the most efficient use of the resources, so I built a load balancer I call drproxy which sits between the reverse proxy and the Rails dispatchers and queues up requests, handing them out in a more efficient way as each resource is freed. Furthermore, I build drproxy in Erlang, a language built from the ground up to excel at concurrency. Ruby is a slug when it comes to handling concurrency and multi-threaded environments. Erlang is like a Porsche.

There are, however, even more ways to make the system more efficient in an algorithmic way. Think about it for a while and I will tell you what I did next week.

Advanced Concepts in Ruby on Rails Hosting Part I

2007-06-19T11:06:00.000-07:00

Let us imagine how a translation company starts out, lets call this company MOG Translation, Inc. At first, there might be one translator and one manager. The manager receives a document from a client and hands it to his translator. The translator might turn around the document in 1 hour, making its way back to the manager and then customer's hands. That is also the fundamentals behind hosting a website. In the most simple form, a web server acts like the manager: it takes in a request (http://mog.com/ for example) and hands it to the application server. The application server acts like a translator, it receives the request and turns it into HTML code that is passed back to the web server and shows up in your web browser.

Soon, MOG Translation, Inc. gets a good reputation and the translation documents come in faster than one per hour. Suddenly our poor lonely translator can't keep up and the papers keep piling up. We all know what to do: hire more translators. Now MOG Translation, Inc. has 10 translators on staff and the manager gets to pick how to hand out the work. One of the simplest ways to hand out documents is to pass them out one at a time to each translator. "One document for Nancy, One document for Drew ...", then when you get to the end of the line go back and starting handing out to Nancy again. Whenever the translators finish, they hand the translation back to the manager who makes sure it goes to the right client. In computer lingo, this system is called round-robin. You have one web server that distributes requests to 10 application servers in a round-robin way.

MOG Translation, Inc. gets such a good reputation that 10 translators is simply not enough. Unfortunately, the office is already a bit cramped so it is time for MOG to open a new branch, another office with another 10 translators. This is equivalent to adding a new server to handle more load. In order to make this change invisible to the customer, we do not want to change the face of our company, so the manager stays the same. But now when he gets to the end of the line in office 1, he faxes the next documents directly to the desks of the translators at office 2. This system is still round-robin, and works fine when adding even more offices.

This is the standard way that many web site's infrastructures grow. In order to handle more requests, they get more servers with more instances of the application running to handle more people visiting the site. The vast majority of web sites never need to grow beyond this point, however popular ones like MOG do. Imagine a talented manager who sends requests to three offices with 10 people in each office in a round-robin way. Problems arise because some documents are longer than others, and some translators are faster than others. This leads to congestion where even though the manager is handing the documents out evenly, some translators desks build up piles. When a pile builds up, even if a very easy document comes in, it might take a lot longer due to the other documents before it. Sure, you could just get more translators, but can you think of a better way to utilize the resources you already have?

Next week I will tell you what I did to better utilize our resources. For the more technical of my readers, the "manager" web server software we use is nginx, a very fast reverse proxy from Russia and the "translator" application software is mongrel which renders Ruby on Rails. Using a reverse proxy and mongrel is the canonical way to serve Rails web sites.

Joel and Norman Totally Miss the Point on Simplicity

2006-12-12T09:25:00.000-08:00

Norman wrote a overgeneralized post about the vices of simplicity to which Joel responded. Both of them have the premise that the most important part of simplicity is features. Joel even goes on about how everyone comments how simple the iPod design is, yet Joel tells us that is just aesthetics, as if aesthetics doesn't sell.

Simplicity is not how many features your product has, simplicity is an idea that can be applied to all parts of a product.

The user interface, the value proposition, the target audience, the infrastructure, the way its components interact, the default preferences, the way in which you change preferences, the way in which you communicate upgrades, the way the product interacts with other products, the api, the manner in which you pay for the product. This list can keep going, these are only a few examples.

Joel and Norman claim that complexity is better to strive for than simplicity, but I think that they are making too broad of a statement. Does anybody really think that the user interface should be more complex than needed? That the target audience be an amorphous void? Their main point seems to be that more features are always better than fewer, however, even that is an over-generalization.

In a startup environment for example, if you strive for more features instead of fewer, that could mean your startup fails to launch and without a product you have nothing. Beware of this over-general advice, these people like to write inflamatory un-predicated articles with no thought to the repercussions.

Context Switches

2006-11-16T06:32:00.000-08:00

Joel Spolsky responded to Dmitri Zimine's article about context swtiching. Dmitri talks about the developer pains caused by context switches and how abusing context switching can lead to lower productivity and less code for your dollar. Joel's response is that if you work against context switching, you are not being agile. He backs his reasoning with an example: a huge bug was discovered that crashed the browser for a loyal customer, so their 2.0 release was held back to fix this.

First of all, Joel declares that having two week release cycles is not agile, which I think is a faulty assumption to start with. Telling a client that they will have to wait two weeks for the feature of their dreams is not always the deal breaker situation Joel makes it out to be, in fact I would think most clients would be impressed if a company released a requested feature within two weeks.

More importantly, there is a huge difference between necessary and unnecessary context switches. The high priority bug fix that Joel mentions is indeed a good reason to switch contexts, however that doesn't mean that if some random person in Zimbabwe says that he wants them to add feature X before the next release, I would hardly think that would be a good reason to switch contexts and focus on X. There certainly should be some abstraction and safety nets in place to make sure that programmers switch context as little as possible but no less. To say, as Joel does, that a 30" monitor means that a great programmer should magically be able to switch context on demand for any reason is a gross over-generalization.

Abuse your car's clutch and it will eventually break, no matter how good of a car or how well you take care of the rest of it.

Syntactic Vinegar vs. Syntactic Sugar

2006-11-13T14:34:00.000-08:00

Syntactic vinegar is to punishing your dog for bad behavior as syntactic sugar is to rewarding them for good behavior.

I certainly don't want to be the dog being choked and scolded all day.

A Fatal Flaw in Opinionated Software

2006-11-13T11:17:00.000-08:00

David Heinemeier Hansson has written a lot about opinionated software and syntactic vinegar. At first glance, the term opinionated software sounds perfectly reasonable. It seems, at least to me, to mean that the software looks and acts in a certain way... it has opinions like DRY and convention over configuration. That is not what opinionated software means to David. To David, a piece of opinionated software is written in such a way that makes it easy to do things one way and ugly and difficult to do things another way.

What does this mean? Glad you asked. Let's look for example at the ActiveRecord find method.

User.find(:first, :conditions => "active = 1")
# => # "1", "active"=>"1"}>

User.find(:first, :with => {:active => 1})
# ArgumentError: Unknown key(s): with

Ok, so the find method does not accept the key :with. That is fine, but let's say you and I are clever hackers and we wanted to enhance ActiveRecord to accept the :with key in order to do some neat magic that reduces the amount of code we have to write. For a Ruby developer, this type of modification sounds like it should be easy, but this is what I found when I first looked at the Rails source.

def validate_find_options(options) #:nodoc:

options.assert_valid_keys([:conditions, :include, :joins, :limit, :offset, :order, :select, :readonly, :group, :from])

end

That is right, the keys were all hard coded, meaning that if I wanted to add my own :with key, I would have to completely overwrite the undocumented validate_find_options method, which inevitably would break things down the line as Rails added its own new keys.

When I first saw this I thought to myself: ah, must be an oversight. I wrote a patch in a few seconds that abstracted the array passed to options.assert_valid_keys into something mutable. I submitted the patch and had a lengthy discussion with DHH in which I was slapped by the opinionated software stick. Apparently, David thought that if I wanted to hack ActiveRecord, I was going to have to be punished by writing ugly code. It was the "opinion" of ActiveRecord that people shouldn't add a key to the find method, so David intentionally made it much harder than necessary to accomplish.

That is right, this was not a simple oversight, this type of code was completely intentional. It is David's opinion that Rails be harder than necessary to hack around and play with. That if I was going to stretch the limits of Rails, take it places nobody else had thought of, that I should be punished for doing so, and that punishment is ugly code.

This is the fatal flaw of opinionated software... in order to make design decisions that limit people's ability to do things you don't think they should do, you must punish them. The form of punishment that David is interested in is syntactic vinegar, like having to override a method instead of append to an array.

The problem with this approach is that punishment is only appropriate for children and criminals. By actively working to make my life as a hacker more difficult, David is treating me like a child or a criminal. Saying either that I am not old enough to know what I am doing and must be punished for doing it, or that I am wrong to think that way at all. Punishment is a very dangerous tool to use because it is fundamentally condescending.

Rails is David's baby, so he can do whatever he wants to it, and that is fine, but to treat a seasoned professional as a child only goes so far in my book. Some might argue that it is for the greater good... that it makes people who don't know what they are doing write better code, keeps people in their place. First of all, the people who end up being punished the most are the ones who want to push the envelope--a beginner is never going to try to add a keyword to the find method. Second of all, I don't want to give up my work productivity for the productivity of those who do not know what they are doing, I want a tool that works best for me. Every time I run into "opinionated software" in Rails, it makes the back of my ears burn reminding me of how much I can't wait to find, start, or help the next generation of highly productive web development software.

Opinionated software is fine until you want to do interesting, new, and innovative things with ease. I am not interested in being treated like a child or a criminal for doing so, which means those opinions which David holds most dear are the ones that will eventually drive out the people that Rails initially attracted... the hackers, the innovators, the bleeding edge developers. Give us freedom or give us another tool that provides freedom.

As a side note, apparently I may have been the first but certainly not the last to complain about the particular form of syntactic vinegar presented with validate_find_options because I see now that they ended up abstracting it and you can now add accepted keys by modifying the VALID_FIND_OPTIONS array. If you would like to see my failed attempt over a year ago to add this feature, check out this ticket.

Functional Programming in Ruby

2006-11-08T11:17:00.000-08:00

In Ruby, iteration is done mainly functionally with methods like each and map. We call this type of iteration functional because we are passing a function to be called on each item in the collection.


[1,2,3].each { |item| puts item + 1 }

In this example. the function being passed is { |item| puts item + 1 }. In fact!


print_function = lambda { |item| puts item + 1 }
[1,2,3].each &print_function

Accomplishes the exact same behavior.

This is very much from the functional programing paradigm. Now, one thing that I can not stand to see in Ruby code is this.


def convert(old)
  new = Array.new
  old.each do |item|
    new << transform(item)
  end
  new
end

The first hint that there is something wrong is the very odd call to return the new variable at the end of the convert method. It seems odd and awkward, and that is a hint that you are working against the grain. Here is how it should look.


def convert(old)
  old.map { |item| transform(item) }
end

Or alternatively:


def convert(old)
  old.map &method(:transform)
end

Notice how clean and beautiful this is. We are working with the grain of functional programming here. Not only that, but there is less to debug, less to maintain and it takes less time to grok what is going on.

If you want to learn more about the functional programming roots of Ruby, I recommend learning OCaml, I am really enjoying it.

How to Learn to Program: From PHP to Lisp in 4 easy steps

2006-09-14T01:33:00.000-07:00

I taught myself how to program. Having gone through a progression of programming languages, I have found patterns that help me piggy back into more and more interesting types of programming. I started with PHP, moved to object-oriented PHP, and moved to a much cleaner and powerful Ruby. Once you learn Ruby, you are introduced with ideas like iteration by defining a block that is called for every item in a collection. If you like those ideas, you are going to love OCaml. Once you grok OCaml, Lisp becomes nearly second nature. I suggest that if you are in the PHP world and want to flex your muscles, try this progression. One language a year and you should be a Lisp guru in 4 years.

How I processed a log file 20x SLOWER than before

2006-08-28T16:55:00.000-07:00

Sometimes you have to sit back and laugh at yourself. Having recently written Starfish to help speed up slow tasks, I tried to find as many uses as I could for it. At MOG we have to parse huge log files so I thought I would be clever and try to use Starfish for the task. After running it for a while, I looked at the stats only to find that it had been processing my file 20x slower than it would have without distributing it. At first I was puzzled, until I realized a very important thing about distributing processes. You have to make sure that the task you distribute takes longer than the distribution process.

if overhead_time > processing_time then puts "Don't use Starfish" end

It turns out that I could process 10,000 lines of the log in about a second... so to send each one of those lines over the network to have them processed was just silly. Even sending 10,000 lines at a time is relatively unnecessary.

I share this story so that you might not make the same mistake that I did. However I realized that Starfish can know when the overhead makes it not worth the trouble, I can actually warn people using Starfish when it is and is not a good use of resources. I will be adding this to the next release which shall come out shortly.

Title Match of the Century: Speed of Development vs. Speed of Computation

2006-08-23T21:09:00.000-07:00

It is entertaining to read the comments people have about Starfish over on Reddit:

If your task is intensive enough to warrant parallelization, it is intensive enough to warrant investigating faster languages. Ruby is good for a lot of things, but if my choice is between throwing more processors at the problem or finding a better solution I will go for the better solution every time.

Interesting point, not that he is right, but what he omits gives me pause. The vast majority of the comments where people talking about Ruby being 1000 times slower than their language, but they give no consideration at all to the most striking aspect of Starfish (in my opinion): I can do relatively advanced distributed programming in 6 lines of code.

I'll say it again because it is important. 6 lines of code.

With hardly more than a flick of my wrist, I can parallelize a task and get performance gains of 10, 20, 30 times, whatever I need. In less than a minute, I can write code that will go through a 10Gb log file grepping for a string and parsing that information, collecting that information and wait for new lines to process on demand, in a distributed system that can work over N machines.

I have written much simpler processes in faster languages like C and it takes me hours and hours, not only for writing the hundreds of lines of code but for debugging the darn thing. If I was tasked with creating a distributed log parser in C that did something non-trivial with each line of the log, it could take me a week and it still wouldn't be right.

I work at startups. I don't work for banks, I don't work for Microsoft, I don't work for enterprise. Can I, as a head programmer at a startup, afford over 80 hours of my time writing a log parser in C because it could be 1000 times faster? Not if my startup wants to succeed. Can Microsoft afford to have one of its tens of thousands of programmers spend the time to do that? Of course.

Starfish can and does, on a daily basis, parallelize and speed up what would have otherwise been a slower process. It does so with almost no code.

A few minutes and N times faster than a regular ruby script, or a few weeks and N times faster than a regular ruby script. I know which I choose, and it works extremely well for us. I am always a fan of the right tool for the right purpose. I know that Starfish is not always the right tool, but it is amazing how quickly people discount a tool without considering all of the issues involved. It is not always about processing power. Man hours saved can be much more valuable than a few extra orders of magnitude in processor power.

More Advanced Starfish Feature

2006-08-23T12:15:00.000-07:00

I promised you in Dynamically Add Methods to Classes Through their Objects in Ruby that there was a good use for that idea coming up. The time has come to show you how to use it.


server do |map_reduce|
  map_reduce.type = File
  map_reduce.input = "some_file_name.txt"

  map_reduce.process = lambda do |text|
    do_something_on_the_server(text)
  end

  map_reduce.finished = lambda do
    do_something_when_the_collection_is_totally_processed()
  end
end

client do |line|
  if line =~ /some_regular_expression/
    server.process($1)
  end
end

Notice how I am dynamically adding methods to map_reduce in the server declaration. I define the process and finished methods. The process method is called from the client via server.process and the finished method is called when the collection has been fully processed.

Astute readers will notice that being able to dynamically add server side helper methods does a non-distribtued version of reduce (from MapReduce), which is good enough for many real world situations. Enjoy!

Starfish is MapReduce for Ruby

2006-08-22T14:45:00.000-07:00

MapReduce and CORBA are huge honking power drills, Starfish is a nice little screw driver. I call Starfish the MapReduce of Ruby because they both do the same task: screw.

I am trying to build the simplest to use, easiest to setup, and fastest to enjoy screw driver on earth. People who ridicule Starfish for its simplicity and lack of features are inadvertently praising me for succeeding at the goal I set for myself. Therefore, ridiculers, I thank you.

How I sent emails 10x faster than before

2006-08-22T01:14:00.000-07:00

Like many startups, at MOG we send out regular updates to our users with news and information. As our user base expands, sending this email takes more and more time. Even though the call to deliver the mail only puts it in the sendmail queue, it can take a chunk of time to do so with so many users.

When I demoed Starfish to people, the common response was: that's great, I wish I had a use for it, I wish I had a DB source big enough to use. Well here is one, albeit not mission critical but still damn cool, of the ways we use Starfish.


require 'config/environment'
require 'user'
require 'notifier'

server do |map_reduce|
  map_reduce.type = User
  map_reduce.conditions = "opt_out = 0"
end

client do |user|
  Notifier.deliver_email(user)
end

This tiny amount of code with next to nothing that needs to be memorized and takes 30 seconds to write down can potentially save you hours in deliver time. Even running 10 clients at once on the SAME MACHINE gave us nearly 10x the speed it would have taken serially. This was not mission critical, but gives you a good sense of ways to apply Starfish to mission critical applications.

MapReduce for Ruby: Ridiculously Easy Distributed Programming

2006-08-16T17:28:00.000-07:00

I am very happy to announce that Google's MapReduce is now available for Ruby (via gem install starfish). MapReduce is the technique used by Google to do monstrous distributed programming over 30 terabyte files. I have been reading about MapReduce recently and thought that it was very exciting for Google to have laid out the ideas that ran Google. I also wondered how they could be applied to everyday applications.

Recently, I gave a talk on Ridiculously easy ways to distribute processor intensive tasks using Rinda and DRb. This talk came from my work with Rinda recently at MOG. We use distributed programming to handle real-time processor intensive needs for over 1 million requests a day. We also use it to make large changes or clean up our database. I realized that the plumbing I wrote in Rinda to accomplish these tasks could be abstracted and easily conform to the MapReduce technique.

Before I move on, I will provide a little more background of Google's MapReduce. MapReduce is a C++ library written by Google. There are about 12 MapReduce programs used to create the inverted index of the www that Google uses for searching. The term MapReduce itself refers to map and reduce functions. Joel recently wrote an article that explains what map a reduce do, so I will refrain from repeating him. One of the parts Joel unfortunately messed up on was this sentence though:

[...] you only have to get one supergenius to write the hard code to run map and reduce on a global massively parallel array of computers, and all the old code that used to work fine when you just ran a loop still works only it's a zillion times faster which means it can be used to tackle huge problems in an instant [...]

Google, nor anyone I know, has written a map function that will "replace" your existing calls to map, like a plugin. In fact, here is some real world MapReduce example code that is used to provide a word count on an arbitrarily sized document:


#include "mapreduce/mapreduce.h" 

// User's map function 
class WordCounter : public Mapper { 
  public: 
  virtual void Map(const MapInput& input) { 
    const string& text = input.value(); 
    const int n = text.size(); 
    for (int i = 0; i < n; ) { 
      // Skip past leading whitespace 
      while ((i < n) && isspace(text[i])) 
        i++; 
      // Find word end 
      int start = i; 
      while ((i < n) && !isspace(text[i])) 
        i++; 
      if (start < i) 
        Emit(text.substr(start,i-start),"1"); 
    } 
  } 
};

REGISTER_MAPPER(WordCounter); 

// User's reduce function 
class Adder : public Reducer { 
  virtual void Reduce(ReduceInput* input) { 
    // Iterate over all entries with the 
    // same key and add the values 
    int64 value = 0; 
    while (!input->done()) { 
      value += StringToInt(input->value()); 
      input->NextValue(); 
    } 
    // Emit sum for input->key() 
    Emit(IntToString(value)); 
  } 
}; 

REGISTER_REDUCER(Adder); 

int main(int argc, char** argv) { 
  ParseCommandLineFlags(argc, argv); 
  MapReduceSpecification spec; 
  
  // Store list of input files into "spec" 
  for (int i = 1; i < argc; i++) { 
    MapReduceInput* input = spec.add_input(); 
    input->set_format("text"); 
    input->set_filepattern(argv[i]); 
    input->set_mapper_class("WordCounter"); 
  } 
  
  // Specify the output files: 
  // /gfs/test/freq-00000-of-00100 
  // /gfs/test/freq-00001-of-00100 
  // ... 
  MapReduceOutput* out = spec.output(); 
  out->set_filebase("/gfs/test/freq"); 
  out->set_num_tasks(100); 
  out->set_format("text"); 
  out->set_reducer_class("Adder"); 
  
  // Optional: do partial sums within map 
  // tasks to save network bandwidth 
  out->set_combiner_class("Adder"); 
  
  // Tuning parameters: use at most 2000 
  // machines and 100 MB of memory per task 
  spec.set_machines(2000); 
  spec.set_map_megabytes(100); 
  spec.set_reduce_megabytes(100); 
  
  // Now run it 
  MapReduceResult result; 
  
  if (!MapReduce(spec, &result)) abort(); 
  // Done: 'result' structure contains info 
  // about counters, time taken, number of 
  // machines used, etc. 
  return 0; 
}

MapReduce takes a large data set (in this case a large file), divides the file into many different pieces, and lets 2000 machines each count words and aggregate statistics for a small part of that file, aggregating the result together in the end.

One of the parts that stood out to me is how there is a clear separation of how to do the call to map and how to do the call to reduce. The other part is all the set calls like spec.set_machines(2000);. I love the simplicity: you tell the system how to map, you tell it how to reduce, you set some options, and run it. Notice specifically that you are not writing network code... this is obviously a very network intensive task, but that is all hidden behind #include "mapreduce/mapreduce.h". This is much like Rinda for Ruby where you do not have to write any network code to distribute objects over the network. You do however have to learn an API to use either Rinda or DRb. MapReduce feels much less like an API and more like a layout, a template that you fill in.

I took the lessons from MapReduce, injected my background of Ruby and came up with what I call Starfish. The backend implementation of Starfish is vastly different than Google's MapReduce: MapReduce is highly optimized for speed and best use of 2000 computer resources at a time, Starfish is highly optimized for speed of development and ease of use. That said, the goal of Starfish is the same as MapReduce.

Starfish takes a large data set (in this case a database), divides the table into many different sections, and lets machines each do work on sections of the database in parallel, aggregating the result together in the end.

Here is some example code:


class Item < ActiveRecord::Base; end

server do |map_reduce|
  map_reduce.type = Item
end

client do |item|
  logger.info item.some_processor_intensive_task
end

You will notice a few major differences quite quickly. First, you do not need to require any libraries, if this file was called item.rb you would run

starfish item.rb

on the command line on as many servers as you want and it will do everything it needed to start working and distributing the work. Next, you do not specify map and reduce functions, rather you specify a client and a server. I loved the simplicity and clarity of defining the two most important parts to Google's MapReduce, but in Ruby it would have been silly to do so because it is not C++ and mapping and reducing is too easy. So I gave it some thought and came up with what I thought was the most important part of distributed programming: what does the server serve and how do the client process the served objects.

Aside from the differences, you will notice the similarity, in the server you are setting options, setting map_reduce.type = Item much like input->set_format("text"); in MapReduce. In the near future, you will be able to tell Starfish that the type is File and let Starfish process files the same way we saw MapReduce do it in the example. Also, logger.info sends some information back to the server that logs it to a file much the same way that out->set_filebase("/gfs/test/freq"); works.

However the biggest major difference is that Starfish is open-source and easy to use. Performing distributed tasks is now a ridiculously easy reality for programmers that may not have been steeped enough in CORBA or some other library to accomplish before.

I hope that you find this library helpful, please tell me how you use it and how I can make it work better for you. There any many options I didn't cover, so if you do use it, please read the documentation.

UPDATE: I wrote an example of how I sent emails 10x faster than before using Starfish.

Ruby Cookbook PDF

2006-08-11T09:11:00.000-07:00

I have some exciting news to share with you all, the Ruby Cookbook is now available for half price in PDF form from O'Reilly. That's right, you can now download and search the cookbook for all your recipes and needs. I usually don't prefer PDF books, but if I hadn't written it, the cookbook certainly would be an exception since each recipe is about a page or two in length. I think the PDF version of the cookbook makes more sense in PDF version than some other technical books. Do you guys prefer hard copies or PDF copies?