Astrails - Home

how to mock `backticks operator` in your test specs using RR

2010-07-05T15:43:00Z

I was working on tests for blender when I came upon a need to mock or stub a back-tick operator e.g. `shell command`.

Apparently this is just a shortcut for calling Kernel method :`.

So the mocking should probably be easier with Rspec mocks or mocha. something like that (didn’t test see below):

@obj.should_receive(:`, "shell command").and_return("....")

But the problem was that I’m using RR. And RR uses a much more succinct and easy to use syntax but in this particular case it was a problem. Consider this:

# This doesn't work. it stubs the :send, not the :`
stub(@obj).send(:`, "shell command") { "...." }

RR has an instance_eval syntax, but with it I don’t know how to pass a block to the back-ticks call:

stub(@obj) do
  `shell command` { "...." } # invalid syntax here ;). 
end

So after some digging inside RR’s sources I got the following working:

stub(@obj).
  __double_definition_create__.
    call(:`, "shell command") { "...." }

Open 2010 Conference

2010-06-16T19:06:00Z

The OPEN 2010 conference was very well organized and had many interesting talks.

The side of low level, infrastructure things was presented by people from Red Hat, VMWare and IBM. The buzz words “SaaS”, “PaaS” and the likes were all over the place together with recent hot topic of virtualization.

On the application level there were interesting presentations about Django, PostgreSQL. And of course our own presentations on Ruby, Rails, NoSQL along with a longer, 2 hours “Introduction to Ruby” workshop.

The food was great too. That’s important.

We uploaded our slides with english transcription for some of them on the “we speak” page.

open 2010 tomorrow and we are there big time

2010-06-09T08:41:00Z

We are giving 4 out of 20 sessions at Open 2010, an Israeli open source conference.

Michael is giving the last keynote and me and Boris will be opening 2 of the 3 tracks: open startup and open enterprise.

I’m going to talk about “NoSQL, when, why, and how” and Boris is giving an introduction to Rails, web development that doesn’t hurt.

After the tracks we will also have a long 2 hour Ruby and Rails workshop.

See you there ;)

P.S. btw, Astrails is sponsoring this event together with a couple of smaller companies like IBM, Red Hat and VMWare :)

P.P.S. Big thanks goes to Raphael Fogel and People&Computers for organizing the whole thing.

installing mysql gem on Snow Leopard

2010-05-21T02:46:00Z

I just recently reinstalled my MacBook Pro, this time with Snow Leopard. So I’m again going through various installation problems I already forgot about from few years back when I installed Leopard.

Anyway, just had to hunt down a problem with mysql gem installation on Snow Leopard.

Apparently if you just do ‘gem install mysql’ it seems to install fine, but doesn’t work.

You get “uninitialized constant MysqlCompat::MysqlRes” error when trying to run rails.

The solution is to pass it some arguments for installation:

ARCHFLAGS="-arch x86_64" gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config

Oh, and of course install 64bit mysql version before that. I usually just download the binaries from mysql.com.

Ruby is Better then Lisp (sometimes)

2010-05-20T13:37:00Z

A few days ago I gave a presentation about Ruby for Sayeret Lambda discussion group.

The title was “Ruby is an acceptable lisp” but the message is better served by “Ruby is Better then Lisp … sometimes” :)

The presentation slides are now online:

The html version is just an export from Keynote so nothing fancy, it only knows how to move forward :) Some slides are bigger, so don’t click again if it seems that its stuck (you will have to refresh to go back), just wait a little.

Markupslicer - Slice your HTML/CSS markup into Ruby on Rails layouts and partials.

2010-04-26T16:47:00Z

http://markupslicer.com

Supports ERB and HAML for now, vote on site for more formats.

Beautifully crafted, totally free and it’s kinda fun.

We originally came with the idea to skip this dull and dirty manual parsing and thought that others might find it useful as well.

Play with demo project and tell us how was it.

SmallRecord - Simple Object persistency library for Cassandra

2010-04-13T15:56:00Z

Simple Object persistency library for Cassandra

Home: http://astrails.com/smallrecord
Code: http://github.com/astrails/smallrecord
Blog: http://blog.astrails.com/smallrecord

Motivation & History (you can skip it :)

I was developing an multiplayer online game for a client (TBD: link when released :) and we decided to use Cassandra for performance and scaling benefits. Also the game’s internal data structures mapped very well to key-value semantics.

I did some research but couldn’t find anything that was Ready at the time to be used for development.

I did find the BigRecord project which somewhat kind of supported Cassandra, but I got an impression that Cassandra support was ‘bolted on’ after the fact, and their way of running a java Cassandra driver talking to some JRuby and interfacing with it through DRB was … can’t quite find the word for it, lets call it ‘awkward’ :).

So I started to work on my own library, “steaing” liberally from BigObject, which at least at the time was very close to ActiveRecord with just some parts of the code commented out or replaced.

Somewhere midway through the implementation I found the CassandraObject project by NZKoz. Now that was a much better one. So much better that for some time I just considered dumping all I did and using it as-is. After some thinking I decided against it though. The primary reason was that its data model was quite different from what I intended to use for the game. In CassandraObject all attributes are stored as columns in a simple ColumnFamily, and indexes and associations stored in separate ColumnFamilies. This has a benefit of not having restrictions on the number of associated object, but it does require additional DB queries to access associated objects.

I wanted to use Supercolumns instead, and store attributes and associations in different supercolumns for the given key. This has the benefit of being able to fetch all the data at once, but does restrict the number of associated objects. Since in my intended use-case the number of associations was rather small I decided to continue the development.

But, this doesn’t mean I didn’t use CassandraObject to “steal” some code from it too :). There were too many good ideas to pass by. I ended up copying a lot of code from it and throwing all of the BigRecord heritage. May be someday I’ll find a way to ‘combine’ SmallRecord back into CassandraObject. Meanwhile though I’m going to work on this one.

(Ugh, that ended up being rather long explanation :)

Status

This is work in progress. It is currently used in a client project and works well, but it still has rough edges and might have problems in your specific environment. The code base is quite small and modular though, so you should have no problems jumping in and fixing or extending it for your use case.

Data Model

This library is intended to be used with Supercolumns Families only (for now :).

Model’s attributes are stored inside “attributes” supercolumn, with attributes themselves being columns inside it. Associations are stored as separate supercolumns, with each association id being a column inside.

Example (json notation):

users: {
  "1": {
    "attributes": {
      "name": "Vitaly Kushner",
      "company": "Astrails"
    },
    "account_ids": {
      "123": "1",
      "456": "1"
    }
}

accounts: {
  "123": {
    "url": "http://astrails.com",
    "username": "vitaly",
    "password": "234987234509827345"
  },
  "456": {
    "url": "http://rubyonrails.org",
    "username": "vitaly",
    "password": "3084573945873945"
  }
}

As you can see we have a one-to-many association here. but contrary to how it is handled in ActiveRecord we don’t store the user_id in the account ‘record’, instead we store all the account_ids in the user record. This is because otherwise we would have no way of querying user.accounts except for the full accounts ‘table’ scan.

Implementation

Connecting to the DB

SmallRecord will look for the file config/small_record.yml :

production:
  adapter: cassandra
  host: 127.0.0.1
  port: 9160
  keyspace: astrails 

development:
  adapter: mock

test:
  adapter: mock

Notice the :mock adapter. This is just a simple Cassandra emulation using in-memory ruby hash. This is what I’m using for development and testing. It doesn’t require cassandra running. The emulation is not 100% off course but it does the job. And I didn’t yet have any bugs related to the difference b/w the mock and the real thing. Just remember that if you run develoment server with mock all the data will be gone once you restart. But that is probably not such a bad thing for development. Or it is. You decide.

Basics

You define your models the usual way:

class User < SmallRecord::Base
  ...
end

user = User.new :foo => "bar"
user.save
User.find(user.id)
User.first

Attributes

What is different from the ActiveRecord is that you have to tell SmallRecord about all your attributes since it can’t infer it from the database schema like ActiveModel does. There is no database schema duh! The attributes support mostly came form CassandraObject but changed quite a bit since then. One day I’ll document the differences :)

class User < SmallRecord::Base
  attribute :name
  attribute :age, :type => :integer
  attribute :create_at, :type => :time
end

ActiveModel

This library is built upon Rails’s ActiveModel pulling in many of the familiar features of the ActiveRecord. The following is supported:

Callbacks

before_save :do_something

The following callbacks are supported:

    :before_init,    :after_init,
    :before_find,    :after_find,
    :before_save,    :after_save,
    :before_create,  :after_create,
    :before_update,  :after_update,
    :before_destroy, :after_destroy,
    :before_validation,
    :before_validation_on_create,
    :before_validation_on_update

You can also define your own callbacks:

class User < SmallRecord::Base
  define_callbacks :after_activation

  after_activation :send_confirmation

  def activate!
    ...
    run_callbacks(:after_activation)
  end
end

Dirty attributes

>> user.changed
=> []
>> user.name = "foo"
=> "foo"
>> user.changed
=> ["name"]
>> user.name_changed?
=> true

When saving an object it will only save changed attributes:

>> user.save
=> true
>> user.name = "qwe"
=> "qwe"
>> user.save
  User Insert (0.000043)   insert(aa421ea0-c407-46fe-986f-09b2d749b1be, {"attributes"=>{"name"=>"\"qwe\"", "schema_version"=>"0"}}, {})
=> true

Validations

class User < SmallRecord::Base
  validates_presence_of :name
end

Associations

Association support in SmallRecord is rather basic. Only has_many is supported at the moment (feel free to add more :).

class User < SmallRecord::Base
  has_many :accounts
end

user.accounts
user.accounts.create
user.create_account
user.account_ids
user.accounts.first

SmallRecord tries hard to do the minimal required amount of work. the association is lazily loaded and only when really needed.

Migrations

Migrations are very different then what you are used to with ActiveRecord (this is too comes from CassandraObject).

You see, there might be a LOT of records in a Cassandra DB. To the point of it being quite impractical to run a full migration. Instead each ‘record’ contains its schema-version and we migrate it on read if its outdated. i.e. if we load a record into memory with schema_version that is less then the currently defined in the code we will migrate this record. If you save the record after that it will be saved with the updated version.

migrations are defined using blocks:

class User
  migrate 1 do |attrs|
    attrs[:foo] = attrs.delete(:bar)
  end
  ...
end

Read the code :)

TODO

There are a couple of things that I want to fix first:

The elephant in the room is the total lack of testing! Well, in the project I’m extracting this from the test coverage is quite high, so all the code was implicitly tested, but now that this is a separate project I need to add some specs.
There is this ugly read/write_data business. Apart from the bad naming (I still can’t think of a good one) all the supercolumns except for the ‘attributes’ are not managed. They are currently written directly into db on every change. Need to unify the ‘dirty’ handling in attributes with the rest of supercolumns. For that I think I’ll need to drop the Dirty mixin from the ActiveModel and just roll my own.
Documentation. This is also lacking at the moment and you will need to look at the code.
Need to research the possibility of merging with CassandraObject. Thought I’m not sure this is practical.

Copyright

Israel Ruby Group Meeting - Mar 2010

2010-03-22T15:19:00Z

About a week ago about 15 people were gathered in People and Computers offices thanks to Raphael Fogel.

Jerry Cheung, nice guy from Outspokes, told everyone how Outspokes was built from the inside and shared his view on building javascript intensive application with Rails as a backend.

Outspokes uses fancy javascript to allow in-browser collaboration of development/design/client teams to request changes, track progress and report problems on ongoing project. I definitely will try it out.

Boris Nadion from Astrails (that’s us in case you were wondering) told the story of our own MarkupSlicer - free to use project we wrote to simplify creating ERB/HAML layouts and partials out of HTML markup we get from our slicing team.

Vitaly Kushner, also from Astrails, made a nice intro presenation about Cassandra - our choice of NoSQL breed. We working with yet to be disclosed client on very technologically challenging project and cassandra is one of many interesting solutions we working with (You can expect a case study on this project in couple of months, as soon as it will go public).

Cassandra Intro presentation

astrails-safe 0.2.7

2010-01-21T22:20:00Z

There is a new (0.2.7) version of Astrails-Safe.

New features since 0.2.5:

default options for gpg now include '--no-use-agent'
support for 'command' option for gpg
quote values in mysql password file
add 'lib' to $:
[EXPERIMENTAL] Rackspace Cloud Files support

H. Wade Minter contributed support for Rackspace cloud files. I have no way of testing this, so please tell me if it works :)

To easy the development process, I added the 'lib' directory of the 'currently running' safe to the library load path"$:". This enables to work on a new version of safe even though you have some version already installed. The files in ./lib will come first in the load path.

IMPORTANT: I just stumbled upon a problem with the gpg encryption on a freshly installed Ubuntu EC2 instance. the gpg was asking for a password on the console even though it was passed the --passphrase-file option with the file containing the password. After some investigation I found out that gnupg's default template configuration file now contains 'use-agent' options. The result of it is that on the first invocation of astrails-safe it will work, but gpg will create a new ~/gnupg/gpg.conf file with 'use-agent' setting, so the next time you run astrails-safe it will fail and ask for the password on the console!

if you are running gpg 1.x series, just pass '--no-use-agent" to the "options" setting in the astrails-safe config in the 'gpg' section (since this 0.2.7 version this options is set in the default astrails-safe template config).

For gpg2 series, I suppose you'll need to run the gpg-agent (actually I'm not sure, I didn't research it too deeply, so I'll be happy to get feedback on the issue)

I also added 'command' option support for gpg which allows to override the executable astrails-safe is using for the enctyption. For example you can have "command "/usr/loca/bin/gpg" in the gpg section.

There is no magic, indeed.

2010-01-14T00:49:00Z

I’ve just read ”Do You Believe In Magic?” and the following quote resonated particularly well:

“It’s not magic. There is no magic involved. It’s just, if you never learnt Lisp, then you never learned to program, and Ruby’s power is exposing a deficiency in your skills.”

I find it to be very true, and I suppose one of the reasons I’m so happy about Ruby is that I learned Lisp and Scheme before.

Even though I didn’t use Lisp for any sizable project. I tried, but I found it to be quite unfriendly and ‘hairy’ in setup, environment and library support. Ruby solved all that for me w/o sacrificing almost anything power vise.

JobTheJob starts the startup

2009-12-16T17:45:00Z

The opportunity: 120 happy dreamers cooking for 54 hours of pure startup joy, well-fed and ready to work. Of course, I’m talking here about Startup Weekend Israel.

The challenge: Come up with an idea that can be implemented in a couple of days, yet is so cool, so innovative, and so useful that I could attract the best of the best.

Read the rest at blogtheblog.jobthejob.com

P.S Now we have some pictures too there, you should check it out

Job the Job

2009-12-16T12:20:00Z

If you follow us on twitter (@astrails if you wondering) you already know that we are at the Startup Weekend Israel right now.

Which is going amazing by the way, thanks for asking, lots and lots of nice people, very creative and energetic atmosphere, food and beer.

Turns out our idea attracted a great team of developers, designers and business developers.

The idea is very simple: aggregate job postings from various sources (starting with twitter) with automated translation to several languages (for now Hebrew, English and Russian) and automated tagging and display all on a web page with free text search and categories browsing.

It should make searching for a job a little bit less of a hassle as now you have to scavenge less sources to find your perfect job.

Why is it good for Startup Weekend:

Easy to explain
Is actually useful, solving real people pains.
Most importantly it really can be implemented within the constrains of the Startup Weekend (it seems like we’ll have only about 10-15 working hours on it)

The name of the service is JobTheJob.

So the team members are:

Me (Vitaly :) and Boris, obviously. Doing backend.
Michael can’t participate as he’s on the judges panel :)
Pavel Kostychev is doing the UI & identity.
Alan Rubin (backend)
Elyon DeKoven (front end, research, and all around)
Pini Sternshos (biz dev, marketing, etc)
Boris Reitman (front end)

We created a blog for the new company, it’s called BlogTheBlog of course. You can grab our progress and impression there and i also will be posting links here when there is a new content there, so you can check out JobTheJob category on this blog. Also we will be tweeting about it, and our personal twitter accounts(look to your right) so you welcome to follow the startup creation process in real time and if you have something to say about it we will be glad to hear.

Startup Weekend Comes to Israel

2009-11-22T18:07:00Z

Startup Weekend comes to Israel.

The event will take place at IBM offices in Petch Tikva from Dec-16 to Dec-18. We are going to participate and also sponsor the event.

Startup weekends are known to be a good place to network with all those startup people, like enterpreneurs, VCs, lawyers, developers and designers.

Israeli event seems to be kosher, so it’s not exactly a weekend: Wed - Fri :-) Grab your discounted ticket using discount code swearlybird, and as they say you should bring “laptop, good attitude, and creative energy” there.

Follow Startup Weekend Israel tweets to stay updated.

We will tweet our impressions and blog about them here.

Looking forward to have some geek fun.

Cassandra on OSX

2009-11-12T23:50:00Z

We just started a project for a client that involves Cassandra.

If you’ve been living under a rock and don’t know what Cassandra is let me tell you :)

Cassandra is a “second-generation distributed database” that was built for web scale. Its is one of the many distributed nosql databases that appear everywhere lately like mushrooms after a heavy rain :).

What sets Cassandra apart is that it comes from a recognizable entity - Facebook.

But I digress. This is not meant to be a Cassandra introduction, there are enough of those on the net. I Just created a new nosql section on this blog where I’m going to post various tidbits of information about cassandra (and probably others) as I learn them while working on this new project.

Here is the first one:

Cassandra gem is just an installer

If you are on Mac OSX and interested in Cassandra you probably know that its just a gem installation away (almost):

gem install cassandra

First thing to note though is that this will not install Cassandra. It will install cassandra installer! I got bitten by this when I took my laptop with me to my doughter’s dancing class. You see, parents are not allowed “in the room” to not interfere with the process :), so I have 45 minutes to find myself something to do each time. I installed cassandra gem at home and intended to play with Cassandra while there. Not so fast. When I run cassandra_helper cassandra which is supposed to start a test cassandra instance it went to connect to a github repository to download and install the actual database. Duh!

and the 2nd one:

Use java preferences

When I got back and finally built Cassandra I got the following message when starting it for the first time:

~ > cassandra_helper cassandra
Set the CASSANDRA_INCLUDE environment variable to use a non-default cassandra.in.sh and friends.
(in /Library/Ruby/Gems/1.8/gems/cassandra-0.5.6.2)
You need to configure your environment for Java 1.6.
If you're on OS X, just export the following environment variables:
  JAVA_HOME="/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home"
  PATH="/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/bin:$PATH"

First thing to note is that just typing JAVA_HOME="/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home" in the terminal wont help. You need to export it:

export JAVA_HOME="/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home"
PATH="/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/bin:$PATH"

No need to export PATH as its already exported.

But there is a better and simpler way!

Start “Java Preferences” (find it in /Applications/Utilities, or just use spotlight):

Then reorder the entries in the bottom “Java Applications” section so that Java 6 will be the 1st:

Now cassandra starts right away w/o any exports:

~ > cassandra_helper cassandra
Set the CASSANDRA_INCLUDE environment variable to use a non-default cassandra.in.sh and friends.
(in /Library/Ruby/Gems/1.8/gems/cassandra-0.5.6.2)
CASSANDRA_HOME: /Users/vitaly/cassandra/server
CASSANDRA_CONF: /Library/Ruby/Gems/1.8/gems/cassandra-0.5.6.2/conf
Listening for transport dt_socket at address: 8888
...

Cool, now go write your killer application!

Let my controller go or useful impurity in software design.

2009-10-27T11:06:00Z

Yeah, I know, MVC is the “Only True Way”. But sometimes, just sometimes, you need your link_to or html helpers working in the model.

For example, sometimes the cleanest way to implement something is to have to_html in the model (widgets anyone?). Doing this will most probably require generating some urls, and you need a controller for that. Usually I solved this by passing controller to the to_html function, but it always felt wrong.

Another example is when you are working on an API. You somtimes would like to have an option to add actual URIs of related services inside an xml. Like in

   <articles>
        <article>
            <title>Let my controller go</title>
            <href>http://example.com/articles/123</href>
            <created-at>2009-05-24T19:09:06Z</created-at>
        </article>
    </articles>

Now how will you implement that? It would be nice if @articles.to_xml just worked, but then how will you generate the href url? url_for is not available in models.

Solution? just make it global and let the purists tear their hairs out! :)

This plugin solves the issues above by making the current controller available from a global function current_controller. (actually it uses Thread local storage, so it will work even in multithreaded Rails environment)

The installation is simple

script/plugin install git://github.com/astrails/let_my_controller_go.git

braid -p git://github.com/astrails/let_my_controller_go.git

Then you can implement the above as

class Article < ActiveRecord::Base
  def href
    current_controller.url_for(self)
  end

  def to_xml(opts = {})
     super opts.merge(:methods => :href, :only => [:title, :created_at])
  end
end

Then Articles.all.to_xml will produce what you need (when running in a context of a web request, i.e. it won’t work from the console).

So, while it might not pass strict software design review, but it is useful. And as with any tool, it’s up to you to use it appropriatly.

Sources are at GitHub.

OK, flame away! /ducks and hides under the desk/