Richard Huang2017-03-05T12:34:35+00:00http://blog.huangzhimin.com/Richard Huangflyerhzm@gmail.comUpgrade to capistrano32013-11-02T00:00:00+00:00http://blog.huangzhimin.com/2013/11/02/upgrade-to-capistrano3<h2>New things</h2>
<p>I updated capistrano from 2.x to 3.0 for one project, it was a huge
change. The followings are the new things:</p>
<p><strong>1. New structure.</strong> If you use capify to generate base structure, you
will see some new syntax.</p>
<p>In Capfile, you need to require all dependencies/plugins you need for
deployment.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="nb">require</span> <span class="s1">'capistrano/setup'</span>
<span class="nb">require</span> <span class="s1">'capistrano/deploy'</span>
<span class="nb">require</span> <span class="s1">'capistrano/rvm'</span>
<span class="nb">require</span> <span class="s1">'capistrano/bundler'</span>
<span class="nb">require</span> <span class="s1">'capistrano/rails'</span>
<span class="no">Dir</span><span class="o">.</span><span class="n">glob</span><span class="p">(</span><span class="s1">'lib/capistrano/tasks/*.cap'</span><span class="p">)</span><span class="o">.</span><span class="n">each</span> <span class="p">{</span> <span class="o">|</span><span class="n">r</span><span class="o">|</span> <span class="n">import</span> <span class="n">r</span> <span class="p">}</span></code></pre></div>
<p>It also generates 2 stages <code>config/deploy/production.rb</code> and
<code>config/deploy/staging.rb</code>, which means you don't need capistrano-ext
anymore, capistrano 3 supports different stages itself.</p>
<p><strong>2. New plugins.</strong> All capistrano plugins for 2.x can't be used in
capistrano 3, like builtin capistrano plugin in bundler and rvm,
fortunately capistrano team already wrote the <a href="https://github.com/capistrano/bundler">bundler</a>, <a href="https://github.com/capistrano/rvm">rvm</a>
and <a href="https://github.com/capistrano/rails">rails</a> plugins. So you should remove old capistrano 2.x
plugins</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="c1"># capistrano 2.x</span>
<span class="nb">require</span> <span class="s1">'bundler/capistrano'</span>
<span class="nb">require</span> <span class="s1">'rvm/capistrano'</span></code></pre></div>
<p>and use new capistrano 3 plugins</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="c1"># capistrano 3</span>
<span class="nb">require</span> <span class="s1">'capistrano/bundler'</span>
<span class="nb">require</span> <span class="s1">'capistrano/rvm'</span></code></pre></div>
<p><strong>3. New flow.</strong></p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="c1"># capistrano 3</span>
<span class="ss">deploy</span><span class="p">:</span><span class="n">starting</span>
<span class="ss">deploy</span><span class="p">:</span><span class="n">started</span>
<span class="ss">deploy</span><span class="p">:</span><span class="n">reverting</span> <span class="o">-</span> <span class="n">revert</span> <span class="n">server</span><span class="p">(</span><span class="n">s</span><span class="p">)</span> <span class="n">to</span> <span class="n">previous</span> <span class="n">release</span>
<span class="ss">deploy</span><span class="p">:</span><span class="n">reverted</span> <span class="o">-</span> <span class="n">reverted</span> <span class="n">hook</span>
<span class="ss">deploy</span><span class="p">:</span><span class="n">publishing</span>
<span class="ss">deploy</span><span class="p">:</span><span class="n">published</span>
<span class="ss">deploy</span><span class="p">:</span><span class="n">finishing_rollback</span> <span class="o">-</span> <span class="n">finish</span> <span class="n">the</span> <span class="n">rollback</span><span class="p">,</span> <span class="n">clean</span> <span class="n">up</span> <span class="n">everything</span>
<span class="ss">deploy</span><span class="p">:</span><span class="n">finished</span></code></pre></div>
<p><strong>4. New syntax.</strong> Capistrano 3 introduces lots of new syntax and new
variables.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="c1"># capistrano 2.x</span>
<span class="n">set</span> <span class="ss">:repository</span><span class="p">,</span> <span class="s2">"git@github.com:railsbp/rails-bestpractices.com.git"</span>
<span class="c1"># capistrano 3</span>
<span class="n">set</span> <span class="ss">:repo_url</span><span class="p">,</span> <span class="s2">"git@github.com:railsbp/rails-bestpractices.com.git"</span></code></pre></div>
<p>capistrano 3 use repo_url instead of repository variable.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="c1"># capistrano 3</span>
<span class="n">set</span> <span class="ss">:linked_files</span><span class="p">,</span> <span class="sx">%w{config/database.yml config/memcache.yml}</span>
<span class="n">set</span> <span class="ss">:linked_dirs</span><span class="p">,</span> <span class="sx">%w{bin log tmp/pids tmp/cache tmp/sockets vendor/bundle public/system}</span></code></pre></div>
<p>linked_files and linked_dirs are very useful, it automatically creates
symbolic files and dirs, which you have to write your own task in
capistrano 2.x.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="c1"># capistrano 2.x</span>
<span class="n">namespace</span> <span class="ss">:css_sprite</span> <span class="k">do</span>
<span class="n">task</span> <span class="ss">:build</span><span class="p">,</span> <span class="ss">:roles</span> <span class="o">=></span> <span class="ss">:app</span> <span class="k">do</span>
<span class="n">run</span> <span class="s2">"cd </span><span class="si">#{</span><span class="n">release_path</span><span class="si">}</span><span class="s2">; </span><span class="si">#{</span><span class="n">rake</span><span class="si">}</span><span class="s2"> RAILS_ENV=</span><span class="si">#{</span><span class="n">rails_env</span><span class="si">}</span><span class="s2"> css_sprite:build"</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="c1"># capistrano 3</span>
<span class="n">namespace</span> <span class="ss">:css_sprite</span> <span class="k">do</span>
<span class="n">task</span> <span class="ss">:build</span> <span class="k">do</span>
<span class="n">on</span> <span class="n">roles</span><span class="p">(</span><span class="ss">:app</span><span class="p">)</span> <span class="k">do</span>
<span class="n">within</span> <span class="n">release_path</span> <span class="k">do</span>
<span class="n">with</span> <span class="ss">rails_env</span><span class="p">:</span> <span class="n">fetch</span><span class="p">(</span><span class="ss">:rails_env</span><span class="p">)</span> <span class="k">do</span>
<span class="n">execute</span> <span class="ss">:rake</span><span class="p">,</span> <span class="s2">"css_sprite:build"</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></div>
<p>capistrano 3 likes dsl (on, within, with, etc.) more.</p>
<p>Capistrano 3 also added parallel and sequence execution, and other
features.</p>
<h2>Problems</h2>
<p>But I also found some problems from capistrano 3</p>
<p>1. capistrano 3 doesn't allow invoke inside on() block, sometimes I
have to write duplicated code, see this <a href="https://github.com/ahmadsherif/capistrano-puma/pull/1">pull request</a>.</p>
<p>2. capistrano generate linked_files and linked_dirs only for app
servers, so when execute deploy:migrate on db server, it can't find
config/database.yml, here is my <a href="https://github.com/wecapslabs/capistrano/commit/2fe6bebe4a1536e2f4ccb0ef8402ff1555a8bf06">temp solution</a>.</p>
Use paperclip without activerecord2013-10-31T00:00:00+00:00http://blog.huangzhimin.com/2013/10/31/use-paperclip-without-activerecord<p>Recently I built an image upload api which didn't use activerecord, but
I don't want to handle resizing image thumbnails by myself, so I decided
to reuse <a href="https://github.com/thoughtbot/paperclip">paperclip</a>.</p>
<p>Paperclip is an easy file attachment management for ActiveRecord, but we
used activemodel without activerecord, I found a <a href="https://gist.github.com/basgys/5712426">gist</a> which gave me
a simple solution, but it was not enough. We continued the hacking work.</p>
<p>1. defined the attachment path and url. Paperclip used AR id partition
in default path, but activemodel don't have id attribute, so I have to
override the attachment path and url</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="c1"># config/initializers/paperclip.rb</span>
<span class="no">Paperclip</span><span class="o">.</span><span class="n">interpolates</span> <span class="ss">:uuid_partition</span> <span class="k">do</span> <span class="o">|</span><span class="n">attachment</span><span class="p">,</span> <span class="n">style</span><span class="o">|</span>
<span class="n">attachment</span><span class="o">.</span><span class="n">instance</span><span class="o">.</span><span class="n">uuid</span><span class="o">.</span><span class="n">scan</span><span class="p">(</span><span class="sr">/.{1,8}/m</span><span class="p">)</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="s2">"/"</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1"># app/models/image.rb</span>
<span class="n">has_attached_file</span> <span class="ss">:attachment</span><span class="p">,</span>
<span class="ss">styles</span><span class="p">:</span> <span class="p">{</span> <span class="ss">three_dot_five_inch</span><span class="p">:</span> <span class="s2">"640x960>"</span><span class="p">,</span> <span class="ss">four_inch</span><span class="p">:</span> <span class="s2">"640x1136>"</span> <span class="p">},</span>
<span class="ss">path</span><span class="p">:</span> <span class="s2">":rails_root/public/system/:attachment/:uuid_partition/:style/:filename"</span><span class="p">,</span>
<span class="ss">url</span><span class="p">:</span> <span class="s2">"/system/:attachment/:uuid_partition/:style/:filename"</span>
<span class="k">def</span> <span class="nf">initialize</span>
<span class="vi">@uuid</span> <span class="o">=</span> <span class="no">UUID</span><span class="o">.</span><span class="n">new</span><span class="o">.</span><span class="n">generate</span><span class="o">.</span><span class="n">gsub</span><span class="p">(</span><span class="s1">'-'</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span>
<span class="k">end</span></code></pre></div>
<p>Instead of auto incremented id, I used uuid partition for attachment
path and url, because it's more scalable.</p>
<p>2. run_callbacks during save, which will also trigger paperclip
callbacks</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">define_model_callbacks</span> <span class="ss">:save</span><span class="p">,</span> <span class="ss">only</span><span class="p">:</span> <span class="o">[</span><span class="ss">:after</span><span class="o">]</span>
<span class="k">def</span> <span class="nf">save</span>
<span class="n">run_callbacks</span> <span class="ss">:save</span> <span class="k">do</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></div>
<p>Then you can handle the attachment by activemodel and paperclip, I
pasted all code on gist <a href="https://gist.github.com/flyerhzm/7289979">here</a>.</p>
railsrumble 2013 - designapis2013-10-21T00:00:00+00:00http://blog.huangzhimin.com/2013/10/21/railsrumble-2013-designapis<p>I took part in this railsrumble last weekend, my entry is
<a href="http://designapis.com">designapis.com</a>, vote it up <a href="http://railsrumble.com/entries/182-designapis">here</a>.</p>
<p>Why did I built it? It comes from my working experience, I worked on
serveral projects that needs to design http apis for ios clients, as we
worked remotely, we have to write down the apis so that ios developers
and ruby developers can work independently.</p>
<p>Before we used google docs and github gists, they are both good to share
between team, but they don't provide any style/format, I have to set the
format by myself, it's not convenient, I have to remember all formats
for requests, responses and parameters, otherwise they will generate
wrong styles.</p>
<p>So I decided to build a service to simplify apis design/documentation,
as railsrumble last only 2 days, I just implemented a small set of
features I wanted, currently you can generate CRUD apis from a template
(inspired from rails generator), you can add any request, response and
parameter, it also gives you some hints that you should take care of
error responses like 404 and 422.</p>
<p>Feel free to try it without registration and any feedback is welcome.</p>
mongoid 3.0.x not set relation properly2013-09-08T00:00:00+00:00http://blog.huangzhimin.com/2013/09/08/mongoid-3.0.x-not-set-relation-properly<p>I was trying to fix <a href="https://github.com/flyerhzm/bullet">bullet</a> test failure with <a href="http://mongoid.org/en/mongoid/index.html">mongoid</a> 3.0.23,
the failed test is to test the 1-1 relationship as follows</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">it</span> <span class="s2">"should detect non preload association"</span> <span class="k">do</span>
<span class="no">Mongoid</span><span class="o">::</span><span class="no">Company</span><span class="o">.</span><span class="n">all</span><span class="o">.</span><span class="n">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">company</span><span class="o">|</span>
<span class="n">company</span><span class="o">.</span><span class="n">address</span><span class="o">.</span><span class="n">name</span>
<span class="k">end</span>
<span class="o">.</span><span class="n">.</span><span class="o">.</span><span class="n">.</span><span class="o">.</span><span class="n">.</span>
<span class="k">end</span></code></pre></div>
<p>After reading the logs, it generated 2 unexpected query</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="ss">MOPED</span><span class="p">:</span> <span class="mi">127</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="mi">1</span><span class="p">:</span><span class="mi">27017</span> <span class="no">QUERY</span> <span class="n">database</span><span class="o">=</span><span class="n">bullet</span> <span class="n">collection</span><span class="o">=</span><span class="n">mongoid_companies</span> <span class="n">selector</span><span class="o">=</span><span class="p">{}</span> <span class="n">flags</span><span class="o">=[</span><span class="ss">:slave_ok</span><span class="o">]</span> <span class="n">limit</span><span class="o">=</span><span class="mi">0</span> <span class="n">skip</span><span class="o">=</span><span class="mi">0</span> <span class="n">batch_size</span><span class="o">=</span><span class="kp">nil</span> <span class="n">fields</span><span class="o">=</span><span class="kp">nil</span> <span class="p">(</span><span class="mi">196</span><span class="o">.</span><span class="mi">5840</span><span class="n">ms</span><span class="p">)</span>
<span class="ss">MOPED</span><span class="p">:</span> <span class="mi">127</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="mi">1</span><span class="p">:</span><span class="mi">27017</span> <span class="no">QUERY</span> <span class="n">database</span><span class="o">=</span><span class="n">bullet</span> <span class="n">collection</span><span class="o">=</span><span class="n">mongoid_companies</span> <span class="n">selector</span><span class="o">=</span><span class="p">{}</span> <span class="n">flags</span><span class="o">=[</span><span class="ss">:slave_ok</span><span class="o">]</span> <span class="n">limit</span><span class="o">=</span><span class="mi">0</span> <span class="n">skip</span><span class="o">=</span><span class="mi">0</span> <span class="n">batch_size</span><span class="o">=</span><span class="kp">nil</span> <span class="n">fields</span><span class="o">=</span><span class="kp">nil</span> <span class="p">(</span><span class="mi">0</span><span class="o">.</span><span class="mi">8612</span><span class="n">ms</span><span class="p">)</span>
<span class="ss">MOPED</span><span class="p">:</span> <span class="mi">127</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="mi">1</span><span class="p">:</span><span class="mi">27017</span> <span class="no">QUERY</span> <span class="n">database</span><span class="o">=</span><span class="n">bullet</span> <span class="n">collection</span><span class="o">=</span><span class="n">mongoid_addresses</span> <span class="n">selector</span><span class="o">=</span><span class="p">{</span><span class="s2">"$query"</span><span class="o">=></span><span class="p">{</span><span class="s2">"company_id"</span><span class="o">=></span><span class="s2">"522c78a4c41a6b019b000014"</span><span class="p">},</span> <span class="s2">"$orderby"</span><span class="o">=></span><span class="p">{</span><span class="ss">:_id</span><span class="o">=></span><span class="mi">1</span><span class="p">}}</span> <span class="n">flags</span><span class="o">=[</span><span class="ss">:slave_ok</span><span class="o">]</span> <span class="n">limit</span><span class="o">=-</span><span class="mi">1</span> <span class="n">skip</span><span class="o">=</span><span class="mi">0</span> <span class="n">batch_size</span><span class="o">=</span><span class="kp">nil</span> <span class="n">fields</span><span class="o">=</span><span class="kp">nil</span> <span class="p">(</span><span class="mi">2</span><span class="o">.</span><span class="mi">9750</span><span class="n">ms</span><span class="p">)</span>
<span class="ss">MOPED</span><span class="p">:</span> <span class="mi">127</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="mi">1</span><span class="p">:</span><span class="mi">27017</span> <span class="no">QUERY</span> <span class="n">database</span><span class="o">=</span><span class="n">bullet</span> <span class="n">collection</span><span class="o">=</span><span class="n">mongoid_companies</span> <span class="n">selector</span><span class="o">=</span><span class="p">{</span><span class="s2">"$query"</span><span class="o">=></span><span class="p">{</span><span class="s2">"_id"</span><span class="o">=></span><span class="s2">"522c78a4c41a6b019b000014"</span><span class="p">},</span> <span class="s2">"$orderby"</span><span class="o">=></span><span class="p">{</span><span class="ss">:_id</span><span class="o">=></span><span class="mi">1</span><span class="p">}}</span> <span class="n">flags</span><span class="o">=[</span><span class="ss">:slave_ok</span><span class="o">]</span> <span class="n">limit</span><span class="o">=-</span><span class="mi">1</span> <span class="n">skip</span><span class="o">=</span><span class="mi">0</span> <span class="n">batch_size</span><span class="o">=</span><span class="kp">nil</span> <span class="n">fields</span><span class="o">=</span><span class="kp">nil</span> <span class="p">(</span><span class="mi">1</span><span class="o">.</span><span class="mi">2510</span><span class="n">ms</span><span class="p">)</span>
<span class="ss">MOPED</span><span class="p">:</span> <span class="mi">127</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="mi">1</span><span class="p">:</span><span class="mi">27017</span> <span class="no">QUERY</span> <span class="n">database</span><span class="o">=</span><span class="n">bullet</span> <span class="n">collection</span><span class="o">=</span><span class="n">mongoid_addresses</span> <span class="n">selector</span><span class="o">=</span><span class="p">{</span><span class="s2">"$query"</span><span class="o">=></span><span class="p">{</span><span class="s2">"company_id"</span><span class="o">=></span><span class="s2">"522c78a4c41a6b019b000015"</span><span class="p">},</span> <span class="s2">"$orderby"</span><span class="o">=></span><span class="p">{</span><span class="ss">:_id</span><span class="o">=></span><span class="mi">1</span><span class="p">}}</span> <span class="n">flags</span><span class="o">=[</span><span class="ss">:slave_ok</span><span class="o">]</span> <span class="n">limit</span><span class="o">=-</span><span class="mi">1</span> <span class="n">skip</span><span class="o">=</span><span class="mi">0</span> <span class="n">batch_size</span><span class="o">=</span><span class="kp">nil</span> <span class="n">fields</span><span class="o">=</span><span class="kp">nil</span> <span class="p">(</span><span class="mi">0</span><span class="o">.</span><span class="mi">9012</span><span class="n">ms</span><span class="p">)</span>
<span class="ss">MOPED</span><span class="p">:</span> <span class="mi">127</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="mi">1</span><span class="p">:</span><span class="mi">27017</span> <span class="no">QUERY</span> <span class="n">database</span><span class="o">=</span><span class="n">bullet</span> <span class="n">collection</span><span class="o">=</span><span class="n">mongoid_companies</span> <span class="n">selector</span><span class="o">=</span><span class="p">{</span><span class="s2">"$query"</span><span class="o">=></span><span class="p">{</span><span class="s2">"_id"</span><span class="o">=></span><span class="s2">"522c78a4c41a6b019b000015"</span><span class="p">},</span> <span class="s2">"$orderby"</span><span class="o">=></span><span class="p">{</span><span class="ss">:_id</span><span class="o">=></span><span class="mi">1</span><span class="p">}}</span> <span class="n">flags</span><span class="o">=[</span><span class="ss">:slave_ok</span><span class="o">]</span> <span class="n">limit</span><span class="o">=-</span><span class="mi">1</span> <span class="n">skip</span><span class="o">=</span><span class="mi">0</span> <span class="n">batch_size</span><span class="o">=</span><span class="kp">nil</span> <span class="n">fields</span><span class="o">=</span><span class="kp">nil</span> <span class="p">(</span><span class="mi">0</span><span class="o">.</span><span class="mi">2601</span><span class="n">ms</span><span class="p">)</span></code></pre></div>
<p>As you can see, every time it queries an address, it also queries a
company, but it already queries all companies, how stupid it is! It is
caused by Mongoid::Relations::Accessors#set_relation doesn't set
properly. I don't want to explain the details here, but solution is
simple, just update mongoid to 3.1.x. Here are the logs for the same
test running on mongoid 3.1.4</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="ss">MOPED</span><span class="p">:</span> <span class="mi">127</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="mi">1</span><span class="p">:</span><span class="mi">27017</span> <span class="no">QUERY</span> <span class="n">database</span><span class="o">=</span><span class="n">bullet</span> <span class="n">collection</span><span class="o">=</span><span class="n">mongoid_companies</span> <span class="n">selector</span><span class="o">=</span><span class="p">{}</span> <span class="n">flags</span><span class="o">=[</span><span class="ss">:slave_ok</span><span class="o">]</span> <span class="n">limit</span><span class="o">=</span><span class="mi">0</span> <span class="n">skip</span><span class="o">=</span><span class="mi">0</span> <span class="n">batch_size</span><span class="o">=</span><span class="kp">nil</span> <span class="n">fields</span><span class="o">=</span><span class="kp">nil</span> <span class="p">(</span><span class="mi">0</span><span class="o">.</span><span class="mi">4551</span><span class="n">ms</span><span class="p">)</span>
<span class="ss">MOPED</span><span class="p">:</span> <span class="mi">127</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="mi">1</span><span class="p">:</span><span class="mi">27017</span> <span class="no">QUERY</span> <span class="n">database</span><span class="o">=</span><span class="n">bullet</span> <span class="n">collection</span><span class="o">=</span><span class="n">mongoid_companies</span> <span class="n">selector</span><span class="o">=</span><span class="p">{}</span> <span class="n">flags</span><span class="o">=[</span><span class="ss">:slave_ok</span><span class="o">]</span> <span class="n">limit</span><span class="o">=</span><span class="mi">0</span> <span class="n">skip</span><span class="o">=</span><span class="mi">0</span> <span class="n">batch_size</span><span class="o">=</span><span class="kp">nil</span> <span class="n">fields</span><span class="o">=</span><span class="kp">nil</span> <span class="p">(</span><span class="mi">0</span><span class="o">.</span><span class="mi">3150</span><span class="n">ms</span><span class="p">)</span>
<span class="ss">MOPED</span><span class="p">:</span> <span class="mi">127</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="mi">1</span><span class="p">:</span><span class="mi">27017</span> <span class="no">QUERY</span> <span class="n">database</span><span class="o">=</span><span class="n">bullet</span> <span class="n">collection</span><span class="o">=</span><span class="n">mongoid_addresses</span> <span class="n">selector</span><span class="o">=</span><span class="p">{</span><span class="s2">"$query"</span><span class="o">=></span><span class="p">{</span><span class="s2">"company_id"</span><span class="o">=></span><span class="s2">"522c7b8ec41a6b3712000014"</span><span class="p">},</span> <span class="s2">"$orderby"</span><span class="o">=></span><span class="p">{</span><span class="ss">:_id</span><span class="o">=></span><span class="mi">1</span><span class="p">}}</span> <span class="n">flags</span><span class="o">=[</span><span class="ss">:slave_ok</span><span class="o">]</span> <span class="n">limit</span><span class="o">=-</span><span class="mi">1</span> <span class="n">skip</span><span class="o">=</span><span class="mi">0</span> <span class="n">batch_size</span><span class="o">=</span><span class="kp">nil</span> <span class="n">fields</span><span class="o">=</span><span class="kp">nil</span> <span class="p">(</span><span class="mi">0</span><span class="o">.</span><span class="mi">8950</span><span class="n">ms</span><span class="p">)</span>
<span class="ss">MOPED</span><span class="p">:</span> <span class="mi">127</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="mi">0</span><span class="o">.</span><span class="mi">1</span><span class="p">:</span><span class="mi">27017</span> <span class="no">QUERY</span> <span class="n">database</span><span class="o">=</span><span class="n">bullet</span> <span class="n">collection</span><span class="o">=</span><span class="n">mongoid_addresses</span> <span class="n">selector</span><span class="o">=</span><span class="p">{</span><span class="s2">"$query"</span><span class="o">=></span><span class="p">{</span><span class="s2">"company_id"</span><span class="o">=></span><span class="s2">"522c7b8ec41a6b3712000015"</span><span class="p">},</span> <span class="s2">"$orderby"</span><span class="o">=></span><span class="p">{</span><span class="ss">:_id</span><span class="o">=></span><span class="mi">1</span><span class="p">}}</span> <span class="n">flags</span><span class="o">=[</span><span class="ss">:slave_ok</span><span class="o">]</span> <span class="n">limit</span><span class="o">=-</span><span class="mi">1</span> <span class="n">skip</span><span class="o">=</span><span class="mi">0</span> <span class="n">batch_size</span><span class="o">=</span><span class="kp">nil</span> <span class="n">fields</span><span class="o">=</span><span class="kp">nil</span> <span class="p">(</span><span class="mi">0</span><span class="o">.</span><span class="mi">5548</span><span class="n">ms</span><span class="p">)</span></code></pre></div>
<p>Great, issue is solved.</p>
Safari video tag without referer header2013-08-07T00:00:00+00:00http://blog.huangzhimin.com/2013/08/07/safari-video-tag-without-referer-header<p>We're building a website which needs to play video online, we know it's
pretty easy for modern browsers who support html5, like chrome and
safari, they all support video tag, which can play video files online
directly.</p>
<div class="highlight"><pre><code class="language-html" data-lang="html"><span class="nt"><video</span> <span class="na">controls</span><span class="nt">></span>
<span class="nt"><source</span> <span class="na">src=</span><span class="s">"RESOURCE URL HERE"</span> <span class="nt">/></span>
<span class="nt"></video></span></code></pre></div>
<p>It's supposed to work for most cases, but our video resources are
uploaded to s3, and our s3 policy for video resources is only when HTTP
referer header is one of our websites, then the video resources can be
accessed, this is used to prevent our video resources to be played on
other website.</p>
<p>I noticed the videos are played welled on chrome, but not on safari,
after some time's digging, I found the chrome will send requests to
fetch video resources with expected http headers, including referer
header, but safari's video requests won't carry any http header, it
causes the video requests are refused by s3.</p>
<p>One solution is to use flash video player, which sends requests with
proper http headers, although flash is old, it's still installed on most
of computers, but it won't help on ios devices (iphone and ipad). So the
only way to make it work on ios is to change our s3 policy, allow the
special referer header "empty".</p>
JQuery AMD Plugin Template2013-08-06T00:00:00+00:00http://blog.huangzhimin.com/2013/08/06/jquery-amd-plugin-template<p>Several years ago I posted how to write a <a href="http://blog.huangzhimin.com/2010/12/20/jquery-plugin-template/">jquery plugin
template</a>,
but in the recent years, browser javascript is evolving, developers are
more likely using asynchronous module definition API (Require.js). So
the jquery plugin template should also be updated like</p>
<div class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="p">(</span><span class="kd">function</span> <span class="p">(</span><span class="nx">factory</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="k">typeof</span> <span class="nx">define</span> <span class="o">===</span> <span class="s1">'function'</span> <span class="o">&&</span> <span class="nx">define</span><span class="p">.</span><span class="nx">amd</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// AMD. Register as anonymous module.</span>
<span class="nx">define</span><span class="p">([</span><span class="s1">'jquery'</span><span class="p">],</span> <span class="nx">factory</span><span class="p">);</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="c1">// Browser globals.</span>
<span class="nx">factory</span><span class="p">(</span><span class="nx">jQuery</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}(</span><span class="kd">function</span> <span class="p">(</span><span class="nx">$</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">$</span><span class="p">.</span><span class="nx">fn</span><span class="p">.</span><span class="nx">pluginName</span> <span class="o">=</span> <span class="kd">function</span><span class="p">(</span><span class="nx">options</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">defaults</span> <span class="o">=</span> <span class="p">{</span>
<span class="c1">// define default options</span>
<span class="p">}</span>
<span class="kd">var</span> <span class="nx">o</span> <span class="o">=</span> <span class="nx">$</span><span class="p">.</span><span class="nx">extend</span><span class="p">({},</span> <span class="nx">defaults</span><span class="p">,</span> <span class="nx">options</span><span class="p">);</span>
<span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="nx">each</span><span class="p">(</span><span class="kd">function</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">var</span> <span class="nx">e</span> <span class="o">=</span> <span class="nx">$</span><span class="p">(</span><span class="k">this</span><span class="p">);</span>
<span class="c1">// write logic here</span>
<span class="p">});</span>
<span class="p">}</span>
<span class="p">});</span></code></pre></div>
<p>The difference is the jquery plugin uses asynchronous jquery module if
it exists, otherwise uses global jQuery as usual.</p>
Migrate rails-bestpractices.com to rails42013-07-14T00:00:00+00:00http://blog.huangzhimin.com/2013/07/14/migrate-rails-bestpractices-com-to-rails4<p>These 2 weeks I migrated <a href="http://rails-bestpractces.com">rails-bestpractices.com</a> to rails 4 from
rails 3.2.13. Here are some experience I'd like to share with you.</p>
<h3>Make sure you have good test code</h3>
<p>rails-bestpractices.com has many rspec and cucumber test code, they can
find out most of warnings and errors after migration.</p>
<h3>Update Gems</h3>
<p>First, update rails to 4.0.0 in Gemfile, but soon you will find you have
to update many gems, devise, compass-rails, cucumber-rails, etc., some
are rc version or raisl4 branch,</p>
<p>You also need to remove some gems, like strong_parameters and
turbo-sprockets-rails3.</p>
<h3>Update bin executables</h3>
<p>Rails 4 app finds executables in bin/ directory, run</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">rake</span> <span class="ss">rails</span><span class="p">:</span><span class="ss">update</span><span class="p">:</span><span class="n">bin</span></code></pre></div>
<p>to get bin/bundle, bin/rails and bin/rake</p>
<h3>Remove unused configs</h3>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">config</span><span class="o">.</span><span class="n">whiny_nils</span>
<span class="n">config</span><span class="o">.</span><span class="n">active_record</span><span class="o">.</span><span class="n">mass_assignment_sanitize</span>
<span class="n">config</span><span class="o">.</span><span class="n">active_record</span><span class="o">.</span><span class="n">auto_explain_threshold_in_seconds</span></code></pre></div>
<h3>Add new configs</h3>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">config</span><span class="o">.</span><span class="n">eager_load</span> <span class="o">=</span> <span class="kp">false</span></code></pre></div>
<p>to config/environments/development.rb and config/environments/test.rb</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">config</span><span class="o">.</span><span class="n">eager_load</span> <span class="o">=</span> <span class="kp">true</span></code></pre></div>
<p>to config/environments/production.rb</p>
<h3>New secret_token</h3>
<p>Rails 4 encrypts the contents of cookie-based sessions, need to use
secret_key_base instead of secret_token.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="no">Application</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">secret_token</span> <span class="o">=</span> <span class="s1">'xxx'</span>
<span class="c1"># =></span>
<span class="no">Application</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">secret_key_base</span> <span class="o">=</span> <span class="s1">'yyy'</span></code></pre></div>
<h3>Remove assets group</h3>
<p>Rails 4 has removed assets group, you should remove it from Gemfile and
config/application.rb</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="c1"># Gemfile</span>
<span class="n">group</span> <span class="ss">:assets</span> <span class="k">do</span>
<span class="n">gem</span> <span class="s1">'sass-rails'</span>
<span class="n">gem</span> <span class="s1">'coffee-rails'</span>
<span class="n">gem</span> <span class="s1">'uglifier'</span>
<span class="k">end</span>
<span class="c1"># =></span>
<span class="n">gem</span> <span class="s1">'sass-rails'</span>
<span class="n">gem</span> <span class="s1">'coffee-rails'</span>
<span class="n">gem</span> <span class="s1">'uglifier'</span>
<span class="c1"># config/application.rb</span>
<span class="k">if</span> <span class="n">defined?</span><span class="p">(</span><span class="no">Bundler</span><span class="p">)</span>
<span class="c1"># If you precompile assets before deploying to production, use this line</span>
<span class="no">Bundler</span><span class="o">.</span><span class="n">require</span><span class="p">(</span><span class="o">*</span><span class="no">Rails</span><span class="o">.</span><span class="n">groups</span><span class="p">(</span><span class="ss">:assets</span> <span class="o">=></span> <span class="sx">%w(development test)</span><span class="p">))</span>
<span class="c1"># If you want your assets lazily compiled in production, use this line</span>
<span class="c1"># Bundler.require(:default, :assets, Rails.env)</span>
<span class="k">end</span>
<span class="c1"># =></span>
<span class="no">Bundler</span><span class="o">.</span><span class="n">require</span><span class="p">(</span><span class="ss">:default</span><span class="p">,</span> <span class="no">Rails</span><span class="o">.</span><span class="n">env</span><span class="p">)</span></code></pre></div>
<h3>Filter parameters in initializer</h3>
<p>Rails 4 prefer setting filter_parameter in initializer.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="c1"># config/application.rb</span>
<span class="n">config</span><span class="o">.</span><span class="n">filter_parameters</span> <span class="o">+=</span> <span class="o">[</span><span class="ss">:password</span><span class="o">]</span>
<span class="c1"># =></span>
<span class="c1"># config/initializers/filter_parameter_logging.rb</span>
<span class="no">Rails</span><span class="o">.</span><span class="n">application</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">filter_parameters</span> <span class="o">+=</span> <span class="o">[</span><span class="ss">:password</span><span class="o">]</span></code></pre></div>
<h3>Fix routes</h3>
<p>Rails 4 doesn't allow using match (without setting via get or post), should
use get or post instead, like</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">match</span> <span class="s1">'/auth/failure'</span> <span class="o">=></span> <span class="n">redirect</span><span class="p">(</span><span class="s1">'/'</span><span class="p">)</span>
<span class="c1"># =></span>
<span class="n">get</span> <span class="s1">'/auth/failure'</span> <span class="o">=></span> <span class="n">redirect</span><span class="p">(</span><span class="s1">'/'</span><span class="p">)</span></code></pre></div>
<h3>New scope syntax</h3>
<p>Rails 4 only allows scopes as a proc</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">scope</span> <span class="ss">:published</span><span class="p">,</span> <span class="n">where</span><span class="p">(</span><span class="ss">:published</span> <span class="o">=></span> <span class="kp">true</span><span class="p">)</span>
<span class="c1"># =></span>
<span class="n">scope</span> <span class="ss">:published</span><span class="p">,</span> <span class="o">-></span> <span class="p">{</span> <span class="n">where</span><span class="p">(</span><span class="ss">:published</span> <span class="o">=></span> <span class="kp">true</span><span class="p">)</span> <span class="p">}</span></code></pre></div>
<h3>Enable turbolinks</h3>
<p>Assume you are not using any client side MVC framework, like Backbone or
Ember, turbolinks can speed up your web pages initialization.</p>
<p>1. add turbolinks gem in Gemfile</p>
<div class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="nx">gem</span> <span class="s2">"turbolinks"</span></code></pre></div>
<p>2. require turbolinks in application.js</p>
<div class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="c1">//= require turbolinks</span></code></pre></div>
<p>Please let me know if you have any problems to migrate to rails 4 :-)</p>
How to render, upload and download large files on heroku with s32013-06-18T00:00:00+00:00http://blog.huangzhimin.com/2013/06/18/how-to-upload-and-download-large-files-on-heroku-with-s3<p>I'm consulting on a rails project on heroku, it involves generating a
large pdf for customer, so you must already guess it led to 30s timeout
on heroku.</p>
<p>At first, I handled it with common sense, moving pdf render to a
background job, in the client side, it polls the status of bj, if job is
complete, then render the pdf.</p>
<p>Everything works fine on my laptop, but after pushing to heroku, it
succeed to running then job, polling the status, but finally it can't
find the generated pdf. Then I realized web dyno and worker dyno are
running on different servers, that means web dyno can't find the pdfs
which are generated on worker dyno. Okay, we need a cloud storage
service, of course, s3 is the first choice.</p>
<p>I used aws-sdk as the s3 client, it's pretty easy to upload pdf to s3,
as pdfs are private on s3, it has to download pdf and render to client
after polling successfully. The timeout problem still exists if the pdf
file is large or the network is not good. (take a long time to render
pdf content to client)</p>
<p>After googling some solutions, I decided to use <a href="http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html">S3 Temporary Security
Credentials</a>, it creates a resource url with a temporary credential,
you can set the expire date for the resource, it sacrifices some
privacy, the resources are still private, but they can be accessed by
the url with a temporary credential, we set the expire date to 1 hour
later, so it's not a big deal.</p>
<p>Resource url with temporary security credential doens't exist in aws-sdk
gem, so I have to implement it by myself.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="nb">require</span> <span class="s1">'openssl'</span>
<span class="nb">require</span> <span class="s1">'digest/sha1'</span>
<span class="nb">require</span> <span class="s1">'base64'</span>
<span class="k">def</span> <span class="nf">signed_url</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">expire_date</span><span class="p">)</span>
<span class="n">digest</span> <span class="o">=</span> <span class="no">OpenSSL</span><span class="o">::</span><span class="no">Digest</span><span class="o">::</span><span class="no">Digest</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="s1">'sha1'</span><span class="p">)</span>
<span class="n">string_to_sign</span> <span class="o">=</span> <span class="s2">"GET</span><span class="se">\n\n\n</span><span class="si">#{</span><span class="n">expire_date</span><span class="si">}</span><span class="se">\n</span><span class="s2">/</span><span class="si">#{</span><span class="no">S3_BUCKET</span><span class="si">}</span><span class="s2">/</span><span class="si">#{</span><span class="n">path</span><span class="si">}</span><span class="s2">"</span>
<span class="n">hmac</span> <span class="o">=</span> <span class="no">OpenSSL</span><span class="o">::</span><span class="no">HMAC</span><span class="o">.</span><span class="n">digest</span><span class="p">(</span><span class="n">digest</span><span class="p">,</span> <span class="no">S3_SECRET_ACCESS_KEY</span><span class="p">,</span> <span class="n">string_to_sign</span><span class="p">)</span>
<span class="n">signature</span> <span class="o">=</span> <span class="no">CGI</span><span class="o">.</span><span class="n">escape</span><span class="p">(</span><span class="no">Base64</span><span class="o">.</span><span class="n">encode64</span><span class="p">(</span><span class="n">hmac</span><span class="p">)</span><span class="o">.</span><span class="n">strip</span><span class="p">)</span>
<span class="s2">"https://</span><span class="si">#{</span><span class="no">S3_BUCKET</span><span class="si">}</span><span class="s2">.s3.amazonaws.com/</span><span class="si">#{</span><span class="n">path</span><span class="si">}</span><span class="s2">?AWSAccessKeyId=</span><span class="si">#{</span><span class="no">S3_ACCESS_KEY_ID</span><span class="si">}</span><span class="s2">&Expires=</span><span class="si">#{</span><span class="n">expire_date</span><span class="si">}</span><span class="s2">&Signature=</span><span class="si">#{</span><span class="n">signature</span><span class="si">}</span><span class="s2">"</span>
<span class="k">end</span></code></pre></div>
<p>It can generate signed url for different resources with different expire
date, now I just tell client the signed_url and the client just render
the pdf from s3 rather than heroku, so no timeout anymore, awesome!</p>
<p>So the whole process is as follows</p>
<ol>
<li>Client clicks "Print PDF" button, it sends a request to web dyno.</li>
<li>Web dyno asks worker dyno render pdf. i</li>
<li>Client keeps polling the job status.</li>
<li>Worker dyno render the pdf and uploads to s3, generates a signed url.</li>
<li>Client gets the jos complete message with s3 signed url.</li>
<li>Client print the pdf from s3.</li>
</ol>
My presentation at reddotrubyconf 20132013-06-07T00:00:00+00:00http://blog.huangzhimin.com/2013/06/07/my-presentation-at-reddotrubyconf-2013<p>This is my presentation on reddotrubyconf 2013 with notes, building
asynchronous apis.</p>
<script async="true" class="speakerdeck-embed" data-id="cd157a80b1700130a9093242a473c411" src="//speakerdeck.com/assets/embed.js"></script>
<p>3. Several years ago when I started learning rails, many people said rails was not fast, but it can significantly speed up development, the famous words are "Hardware is cheap, Programmers are expensive".</p>
<p>4. It is true when your business is still young, at first, you may have only one server, then you can buy more servers, distribute web, app and db to different servers, it's the easiest way to handle more traffic.</p>
<p>5. but...</p>
<p>6. one day your business gets successful, more and more users will bring more and more traffics, you have to buy more and more web servers, app servers and db servers.</p>
<p>7. At this stage, machines are not cheap any more, so programmers try to find ways to improve server performance, increase concurrency and reduce machines.</p>
<p>8. So you see linkedin moved from rails to node, 27 servers cut and up to 20x faster.</p>
<p>9. Iron.io mvoed from rails to go, 28 servers cut.</p>
<p>10. Does it mean ruby or rails is so slow? Does it mean we should drop ruby and use node.js or go instead?</p>
<p>11. From my experience, the answer is no, this is what I want to share with you today.</p>
<p>12. During my last job, I'm lucky that I have the opportunity to build the same leaderboard apis service running with multi processes, multi threads and asynchronous non blocking io ruby servers. At first we build the api service with rails and ree, db is mysql, the average response time is 50 ms, and it handled 60k rpm on production with 13 machines, 6 passenger instances per machine. After that we migrated to JRuby 1.7.0, it introduced 40% performance improved, average response time decreased to 30 ms, it was same to handle 60k rpm on production, but with only 10 machines, 1 torquebox instance with 5 threads per machine. Finally, we rewrote the api service, using ruby 1.9.3, and used goliath, which is a non-blocking ruby web server, switched db to redis, the average response time decrease to 4 ms, and handled 240k rpm on production with only 4 machines, 4 goliath instances each machine. If old rails api service also handled 240k rpm, it needs 52 machines.</p>
<p>13. So I can say moving from ruby to ruby, 48 servers cut and 10 times faster.</p>
<p>14. We run the api service with rails synchronously, all IO operations are blocked, but run with goliath is asynchronous, and IO operations are nonblocking. It's a bit unfair to compare the performance directly between Rails and Goliath, or Rails and Node.js, but it's good to know that building api service with asynchronous nonblocking io can significantly increase the concurrency.</p>
<p>15. How blocking IO works? e.g. when a request is coming, a process gets the cpu time, run the code, but when it calls database query, cpu has to wait for it to complete, we all know IO operations are slow, the blocking IO will block the whole process.</p>
<p>16. In multi processes model, when cpu is blocked in process A, it will schedule from process A to process B, keep running, when IO operation is completed in process A, cpu will schedule back to process A and continue working.</p>
<p>17. The advantages of multi processes are the multi processes can be executed in true parallel on multi cores cpu. Running with multi processes model is easy to manage, we can start or stop a processor by sending a signal. The disadvantages of multiple processes are process switching is expensive, it involves switching out all of the process resources. It also consumes many memory, multiple processes means multiple memory copy.</p>
<p>18. Multi threads model is similar to multi processes model, when cpu is blocked in thread A, it will schedule from thread A to thread B in one process.</p>
<p>19. The advantages of multi threads are threads switching is cheap, it involves switching out only the resources unique between threads. As many resources are shared between threads in one process, multi threads consume much less memory. The disadvantages are if you share mutable data across threads, you need to synchronize access to that data for thread safety, this will affect performance and concurrency. With ruby 1.9 or 2.0, GIL is still there, that means only one thread can handle request at a time. The exception is JRuby and Rubinius which already removed GIL and can make use of multi cores.</p>
<p>20. Evented model is running with a main loop, and never blocked, all io operations are asynchronous, when calling an io operation, instead of waiting, the main loop can process other requests, and come back when the response from io call is ready.</p>
<p>21. The advantages of the evented model are there is no blocking io, no context switching and it consumes least memory usage comparing to multi processes and multi threads models. The disadvantage is your code will be full of callbacks, make it difficult to understand.</p>
<p>22. In ruby world, we usually use eventmachine to implement non blocking io, but it's very common to write many nested callbacks, like this.</p>
<p>23. Good luck, ruby 1.9 introduces fiber, and a gem, named em-synchrony, fiber aware eventmachine can help solve too many callbacks issue.</p>
<p>24. The code works same as the last example, using em-synchrony, but no callbacks and more readable.</p>
<p>25. Let’s clarify the definitions of concurrency and parallelism, concurrency performs 2 operations in tandem, while parallelism performs 2 operations at the same time.</p>
<p>26. Evented model is used to increase concurrency in one processor, so in practice, we will use multi processes with evented in order to utilize all of the cores on your CPU.</p>
<p>27. We already talked why we should use async non-blocking IO, but how? I wrote a project on github named apis-bench, it implements a simple leaderboard apis service with multiple framework and run on multi processes, multi threads and asynchronous non blocking ruby server.</p>
<p>28. Assume we use rails to build the apis service, router dispatches the request to controller, controller creates, reads, updates or destroys models, then controller generate a json view and sends the response.</p>
<p>29. A good practice is skinny controller and fat model, so I write most logic in models.</p>
<p>30. Write the controller as simple as possible, and use respond_to / respond_with to render a json response.</p>
<p>31. Here is the router.</p>
<p>32. Instead of migrating asynchronous io directly, let's do a small step, migrating rails to grape framework. Grape is a micro framework to build REST apis, using grape instead of rails controller can decrease response time.</p>
<p>33. Grape is responsible for router, controller and view, gets request, ask model to do something and then render response.</p>
<p>34. Most developers prefer using activerecord, it provides many powerful ways to develop models rapidly. We can use activerecord without rails, so here, we don't need to do any change in model layer.</p>
<p>35. Grape provides its own DSL, we have to use Grape::API replace rails controller api, but we already followed the skinny controller practice, it should not be too much work to do.</p>
<p>36. Next step, let's migrate to asynchronous non-blocking io, goliath is a non-blocking ruby web server framework, adding goliath can significantly increase the throughput.</p>
<p>37. Here, each HTTP request is handled by goliath, request is executed within a ruby fiber, then goliath proxies request to grape, all IO operations are asynchronous.</p>
<p>38. In this migration, we also no need to change any code except adding goliath api, and telling it delegate request to grape api, very simple.</p>
<p>39. Besides we must replace our existing blocking io libraries to eventmachine's libraries, like mysql2 to em_mysql2, mongo to em_mongo, etc.</p>
<p>40. We have another option besides grape, it's sinatra, it can also decrease the response time.</p>
<p>41. Similar to grape, sinatra takes care of router, controller and view.</p>
<p>42. The only place we should change from rails to sinatra is the controller, here we define the route, action and render json</p>
<p>43. After using sinatra, it's also easy to migrate to asynchronous non blocking io by adding sinatra-synchrony, as its name implies, sinatra-synchrony adds em-synchrony to sinatra.</p>
<p>44. sinatra-synchrony is not a web server, so we have to use an event ruby server, like thin, then sinatra-synchrony executes each http request within a ruby fiber just like goliath.</p>
<p>45. Adding sinatra-synchrony is also easy, what you need to do is only register Sinatra::Synchrony as an extension, then it works.</p>
<p>46. Same, you need to replace blocking io library with eventmachine's libraries.</p>
<p>47. Finally, let's see the benchmark result.</p>
<p>48. The first benchmark test is a CPU bound action, db time takes about 10% total response time, it's tested with apache benchmark.</p>
<p>49. I tested with several groups, sending 1000 requests with 10 concurrency, 50 concurrency, 100 concurrency, 200 concurrency, 500 concurrency and last one is sending 2000 requests with 1000 concurrency, in each group, I tried rails, sinatra, grape, sinatra with threads, grape with threads, sinatra-synchrony with thin and grape with goliath, they all run in a single process, rails, sinatra and grape are running in unicorn, sinatra with threads and grape with thread are running in rainbows. The value here is the time taken for completing all requests. As you have seen, sinatra api is 40% faster than rails api and grape api is 30% faster than rails api, sinatra threads and grape threads in this case, about 2 times slower, it should perform better when running threads with jruby. Asynchrous non blocking io performs best, especially sinatra-synchrony. When sending requests with 200 concurrency, rails, sinatra, grape, sinatra threads and grape threads are all timed out, they are failed to handle so many requests, but grape with goliath and sinatra-synchrony with thin can handle 1000 concurrency without any errors.
With CPU bound actions, threads didn't perform well, but asynchronous non blocking io works can handle much more requests.</p>
<p>50. What about the IO bound action? the following test is an action, in which db time takes more than 90% total response time, I sent the requests with stable rates and measured the performance on newrelic.</p>
<p>51. I sent 1200 rpm to rails api, but it can only handle 984 rpm, others returns 502 timed out error.</p>
<p>52. Sinatra api can only handle 1080 rpm.</p>
<p>53. and grape api handle 1130 rpm, all of them failed to handle 1200 rpm.</p>
<p>54. I set rainbows work with 200 threads and 200 connections in db pool, sinatra threads successfully handle 3000 rpm.</p>
<p>55. Grape threads also succeed.</p>
<p>56. grape with goliath passed as well.</p>
<p>57. I failed to add newrelic with sinatra-synchrony, but it also succeed to handle 3000 rpm. Another known issue is I failed to add fiber_pool to goliath, I appreciate it if you can help to solve the issue and open a pull request to me.</p>
<p>58. Okay, the conclustion is Rails is good, it's still a good choice to build apis service rapidly, we can migrate rails to sinatra or grape to decrease response time, then migrate to sinatra-synchrony or goliath to increase the throughput, finally I have to say asynchronous non-blocking io is awesome, you should give it a try.</p>
another zero downtime deployment solution2013-01-30T00:00:00+00:00http://blog.huangzhimin.com/2013/01/30/another-zero-downtime-deployment-solution<p>I wrote <a href="http://huangzhimin.com/2012/11/14/jruby-at-openfeint-jruby-migration-success-story/">a post</a> for jruby migration 2 monthes ago, it mentioned a
solution to do zero downtime deployment: pull out server out of load
balancers, restart server, and then put in the server. It works but has
some cons</p>
<ol>
<li>you must have more than 1 app hosts.</li>
<li>deployment process gets much slower if you have lots of app hosts.</li>
<li>you lost one host's throughput during deployment.</li>
</ol>
<p>I'm using a different solution for zero downtime deployment now, instead
of processing app hosts one by one</p>
<ol>
<li>it starts replicated ruby instances on all app hosts.</li>
<li>reload load balancer (proxy) to send traffic to replicated ruby
instances.</li>
<li>stops original ruby instances.</li>
</ol>
<p>It won't slow down your deployment process, it also works well if
you only have 1 app host and you don't lost any throughput during
deployment.</p>
<p>The disadvantage is it needs more memory on your app host, it occupies
x2 ruby instances' memory during deployment. Our project is an api
service built on ruby not rails, memory usage is pretty low, only 50 mb
per ruby instance, so x2 memory usage is not a big deal.</p>
another redis automatic failover solution for ruby2013-01-13T00:00:00+00:00http://blog.huangzhimin.com/2013/01/13/another-redis-automatic-failover-solution-for-ruby<p>Redis gets more and more popular as a backend storage, so the redis
failover solution becomes important before you use redis as a critical
resource.</p>
<p>Currently the most popular automatic master/slave failover solution for
ruby is <a href="https://github.com/ryanlecompte/redis_failover">redis_failover</a>, it's based on ZooKeeper, if you already
have ZooKeeper in your infrastructure, it's great.</p>
<p>But I noticed that redis already has a built-in automatic failover
solution, called <a href="http://redis.io/topics/sentinel">Redis Sentinel</a>. In case you didn't heard of it,
please read the official document, it's simple and no other external
dependency. I searched on github, but none was working well. I have
to implement it by myself.</p>
<p>The key point is you never connect to redis master server directly.
Instead, you talk to redis sentinel servers, ask them where is the
master server, and then connect to the redis master server.</p>
<p>When your redis master server down, your redis sentinel servers will
tell you a new master server, so you just disconnect old server and
connect to new master server</p>
<p>My soluion is a monkey-patch to redis-rb gem, it's <a href="https://github.com/flyerhzm/redis-sentinel">redis-sentinel</a>,
before it tries to connect redis server, it firstly asks redis
sentinels where is master server, then connect as usual. Try it and give
me the feedback.</p>
newrelic-grape released2012-12-21T00:00:00+00:00http://blog.huangzhimin.com/2012/12/21/newrelic-grape-released<p><strong>No instrumentation, no performance tuning!</strong></p>
<p>This is my first time to use <a href="https://github.com/intridea/grape">grape</a> to build an api service, grape
repo has more than 2k watchers, but I'm surprised there is no existing
newrelic grape suppport, I just found some gists to do it, and this
<a href="http://artsy.github.com/blog/2012/11/29/measuring-performance-in-grape-apis-with-new-relic/">blog post</a> gave me the idea to add newrelic instrument as grape
middleware, but it's not the standard way newrelic recommends.</p>
<p>So I released <a href="https://github.com/flyerhzm/newrelic-grape">newrelic-grape</a> gem to help you integrate newrelic
into grape.</p>
<p>What you need to do is</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="nb">require</span> <span class="s2">"newrelic-grape"</span>
<span class="nb">require</span> <span class="s2">"rpm_contrib"</span></code></pre></div>
<p>and monitor the performance on newrelic.</p>
How I find out a memory leak in grape2012-12-16T00:00:00+00:00http://blog.huangzhimin.com/2012/12/16/how-i-find-out-a-memory-leak-in-rack-app<p>I'm helping my customer build a high performance api service these
weeks, we are close to release, but when I did load test this Wednesday,
I found the memory kept growing when I sent traffic and never went down,
it was obviously a memory leak.</p>
<p>Lucky is I can reproduce the memory leak on my local machine, so I can
detect it easily. Our api service is simple, only contains model layer
(AR and redis) and api layer (based on <a href="https://github.com/intridea/grape">grape</a>). At first, I disabled
model layer, but memory leak was still there, so I was pretty sure the
leak was in api layer.</p>
<p>Memory leak is always not easy to find, especially when I'm not sure
where it is, in my own code or some dependent libraries I used. I need
some tools' help.</p>
<p>First, I used <a href="https://github.com/Vasfed/heap_dump">heap_dump</a> to dump the ruby heap memory after sending
10 minutes' traffic, and searched the keywords used in our repository, I
noticed every request path string resided in memory, why? Was there an
array or hash used them? heap_dump can't answer me.</p>
<p>Then I tried ruby 1.9.3 ObjectSpace to find more info. I changed
Grape::API.call behavior, printing live objects for each request.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">def</span> <span class="nf">call_with_gc</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
<span class="no">GC</span><span class="o">.</span><span class="n">start</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">call_without_gc</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
<span class="nb">p</span> <span class="no">ObjectSpace</span><span class="o">.</span><span class="n">count_objects</span>
<span class="n">result</span>
<span class="k">end</span></code></pre></div>
<p>The followings are parts of the result</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">{</span>:TOTAL<span class="o">=</span>>331126, :FREE<span class="o">=</span>>218067, :T_OBJECT<span class="o">=</span>>3339, :T_CLASS<span class="o">=</span>>3394, :T_MODULE<span class="o">=</span>>474, :T_FLOAT<span class="o">=</span>>195, :T_STRING<span class="o">=</span>>55324, :T_REGEXP<span class="o">=</span>>1135, :T_ARRAY<span class="o">=</span>>20188, :T_HASH<span class="o">=</span>>926, :T_STRUCT<span class="o">=</span>>125, :T_BIGNUM<span class="o">=</span>>22, :T_FILE<span class="o">=</span>>4, :T_DATA<span class="o">=</span>>16011, :T_MATCH<span class="o">=</span>>13, :T_COMPLEX<span class="o">=</span>>1, :T_RATIONAL<span class="o">=</span>>33, :T_NODE<span class="o">=</span>>11273, :T_ICLASS<span class="o">=</span>>602<span class="o">}</span>
<span class="o">[</span>23153:INFO<span class="o">]</span> 2012-12-16 21:59:55 :: Status: 200, Content-Length: 19, Response Time: 42.43ms
<span class="o">{</span>:TOTAL<span class="o">=</span>>331126, :FREE<span class="o">=</span>>218066, :T_OBJECT<span class="o">=</span>>3339, :T_CLASS<span class="o">=</span>>3394, :T_MODULE<span class="o">=</span>>474, :T_FLOAT<span class="o">=</span>>195, :T_STRING<span class="o">=</span>>55325, :T_REGEXP<span class="o">=</span>>1135, :T_ARRAY<span class="o">=</span>>20188, :T_HASH<span class="o">=</span>>926, :T_STRUCT<span class="o">=</span>>125, :T_BIGNUM<span class="o">=</span>>22, :T_FILE<span class="o">=</span>>4, :T_DATA<span class="o">=</span>>16011, :T_MATCH<span class="o">=</span>>13, :T_COMPLEX<span class="o">=</span>>1, :T_RATIONAL<span class="o">=</span>>33, :T_NODE<span class="o">=</span>>11273, :T_ICLASS<span class="o">=</span>>602<span class="o">}</span>
<span class="o">[</span>23153:INFO<span class="o">]</span> 2012-12-16 21:59:56 :: Status: 200, Content-Length: 20, Response Time: 43.29ms
<span class="o">{</span>:TOTAL<span class="o">=</span>>331126, :FREE<span class="o">=</span>>218065, :T_OBJECT<span class="o">=</span>>3339, :T_CLASS<span class="o">=</span>>3394, :T_MODULE<span class="o">=</span>>474, :T_FLOAT<span class="o">=</span>>195, :T_STRING<span class="o">=</span>>55326, :T_REGEXP<span class="o">=</span>>1135, :T_ARRAY<span class="o">=</span>>20188, :T_HASH<span class="o">=</span>>926, :T_STRUCT<span class="o">=</span>>125, :T_BIGNUM<span class="o">=</span>>22, :T_FILE<span class="o">=</span>>4, :T_DATA<span class="o">=</span>>16011, :T_MATCH<span class="o">=</span>>13, :T_COMPLEX<span class="o">=</span>>1, :T_RATIONAL<span class="o">=</span>>33, :T_NODE<span class="o">=</span>>11273, :T_ICLASS<span class="o">=</span>>602<span class="o">}</span>
<span class="o">[</span>23153:INFO<span class="o">]</span> 2012-12-16 21:59:57 :: Status: 200, Content-Length: 20, Response Time: 45.74ms</code></pre></div>
<p>As you can see, every request, there was a string couldn't be garbage
collected, but I still didn't know where it was. I commented my logic
code in api layer, just returned an empty json, and string leak still
existed, then I went to grape source code, commented the code in
<a href="https://github.com/intridea/grape/blob/v0.2.2/lib/grape/api.rb#L49">Grape::API#call</a> method, updated as following code</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">def</span> <span class="nf">call</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
<span class="o">[</span><span class="mi">200</span><span class="p">,</span> <span class="p">{},</span> <span class="s2">""</span><span class="o">]</span>
<span class="k">end</span></code></pre></div>
<p>After that, the string memory leak disappeared, It was a strong
possibility that memory leak was in grape gem.</p>
<p>Next thing was pretty easy, tried to replace grape middleware one by one,
grape middleware has 3 methods, call!, before and after, will be called
in every request, I replaced all of them to figure out leak.</p>
<p>Finally, I found it's <a href="https://github.com/intridea/grape/blob/v0.2.2/lib/grape/middleware/formatter.rb#L43">method format_from_extension</a> in grape Formatter
middleware caused memory leak, it genereate a symbol no matter if there
is an extension in the request path, e.g.</p>
<p>if requesting /v1/blog/post/1, it will create symbol :"/v1/blog/posts/1"</p>
<p>if requesting /v1/blog/post/2, it will create symbol :"/v1/blog/posts/2"</p>
<p>......</p>
<p>In case you don't know, symbol won't be garbage collected in ruby, so
every time it got a request path different then before, it created a
symbol in memory which won't be garbage collected.</p>
<p>Problem detected, solution is <a href="https://github.com/intridea/grape/pull/291">here</a>.</p>
<p>In conclusion, be careful to ruby symbol, do not convert any non
controlled string to symbol.</p>
JRuby at OpenFeint - a JRuby migration success story2012-11-14T00:00:00+00:00http://blog.huangzhimin.com/2012/11/14/jruby-at-openfeint-jruby-migration-success-story<p><strong>TL;DR:</strong> OpenFeint gets 40% performance improvement after migrating
to JRuby from REE.</p>
<h2>About OpenFeint</h2>
<p>OpenFeint was the largest mobile social gaming platform in the world,
It was acquired by GREE for $104 million last year, and a new global
platform is building to replace OpenFeint. It is still one of the
biggest rails applications, with hundreds of thousands API calls per
minute.</p>
<p>OpenFeint platform is using rails 2.3.14 and was running on ree 1.8.7.</p>
<h2>Why try JRuby</h2>
<p>My main job is to improve the performance and scalability of OpenFeint
platform. This April, I attended Railsconf at Austin, there was
<a href="http://railsconf2012.com/sessions/66">a panel discussion</a> talking about real world rails apps, speakers
came from New Relic, Zendesk, Groupon, etc. They use the similar
achitecture like us, ree 1.8.7, rails 2.3, mysql, memcached, redis,
rabbitmq and so on. They all complained the slow gc of ruby 1.8.7, so
did we. After that, there are 2 jruby sessions interested me.</p>
<ul>
<li><a href="http://railsconf2012.com/sessions/62">Up and to the right - how Spiceworks is scaling 200 million requests
per month</a>, they shown how they migrate their rails app to jruby and
got 20% performance improvement.</li>
<li><a href="http://railsconf2012.com/sessions/16">Complex Made Simple: Sleep Better with Torquebox</a>, it introduced
torquebox, a ruby application server that is build on JRuby and JBoss
AS 7.</li>
</ul>
<p>When I went back to hotel, I googled something about jruby performance
and found <a href="http://torquebox.org/news/2011/10/06/torquebox-2x-performance/">torquebox performance benchmark</a>, it looked pretty
exciting. At that time I decided to try jruby on OpenFeint platform.</p>
<p><strong>Note:</strong> you probably know new relic and zendesk have already migrated
to ruby 1.9.</p>
<h2>Quick and dirty performance test with JRuby</h2>
<p>I always prefer doing the performance test by myself rather than blindly
believing the performance benchmark online. So the first thing I want to
do was to do a quick performance test with JRuby on OpenFeint platform.</p>
<p>It was expected that OpenFeint platform couldn't work with JRuby. To
quickly verfiy if JRuby could give us a great performance improvement, I
fixed incompatible ruby gems, like adding jruby-openssl gem, removing
SystemTimer gems and using activerecord-jdbcmysql-adapter instead of
mysql gem. I also did some dirty hacks, e.g. I disabled database
sharding, background job and other non working parts, just want to do
a quick performance test. Then I deployed app to one of our qa servers,
the result of quick performance test is as follows</p>
<ul>
<li>response time of ree + passenger is <strong>331ms</strong></li>
<li>response time of jruby + torquebox is <strong>51.5ms</strong></li>
</ul>
<p>I was shocked that JRuby is so fast, that made it easy to persuade
manager to migrate OpenFeint platform to JRuby.</p>
<p><strong>Note:</strong> our qa environment is quite different to production
environment, databases are shared between qa servers, but memcached,
redis, rabbitmq and app server are working together in one host, and
ree on qa server didn't do any gc tuning.</p>
<h2>JRuby migration strategy</h2>
<p>After the quick performance test, JRuby looked very promising, then I'm
allowed to focus my work on JRuby migration. Before I tell you how we
migrate to JRuby, please let me give you a short introduction about
what OpenFeint platform uses</p>
<ul>
<li>load balancer servers with nginxes.</li>
<li>app servers with nginx + passenger.</li>
<li>memcached servers for caches.</li>
<li>redis servers for feature flags, high score caches, device mapper, etc.</li>
<li>mysql servers for data storage.</li>
<li>uses rabbitmq server and workling servers to handle background jobs.</li>
</ul>
<p>Of course OpenFeint platform uses other servers for cron job, performance
test, continuous integration, full text search, log analytics, etc.</p>
<p>To handle the massive requests, OpenFeint platform splits app and
databse servers into different pools according to different
functionalities.</p>
<p>Each app pool is isolated, they don't know each other. Load balancer
servers decide sending requests to which pool according to the request
urls. Each pool will connect to all db servers, e.g. high score app
servers will fetch high score info from high score dbs and fetch
user/game info from core dbs.</p>
<p>Considering that we don't have experienced java ops and we only have
1 or 2 qas can involve in, it is a big risk to migrate the whole
OpenFeint platform to JRuby. So I decide to do JRuby migrate one app pool
by one app pool.</p>
<p>The advantage of migration one pool by one pool is it allows OpenFeint
gets the JRuby's speed earlier, 1 or 2 qas are enough to promise app is
working correctly for one pool, ops can setup jruby environment and tune
the jvm performance on one pool's hosts to accumulate jruby experience.</p>
<p>The disadvantage is we have to promise OpenFeint platform is working
well on both REE and JRuby, running app with REE on some pools and
running app with JRuby on other pools.</p>
<p><strong>Note:</strong> only load balancers and mysql servers are dedicated servers,
others are VPS.</p>
<h2>Fix incompatible gems</h2>
<p>The most problems for migrating a rails app to JRuby are incompatible
gems, like c extensions gems or some non thread-safe ruby gems. I
encountered 2 incompatible gems that wasted my time.</p>
<p>1. <a href="https://github.com/typhoeus/typhoeus">typhoeus</a>, it is one of the fastest http client ruby gems, it's
a c extenion gem, we used it to synchronize data between OpenFeint
platform and the new global platform. The official document says it is
built with FFI and is ready for use with any Ruby implementation. But
during performance test, I found it always crashed the JVM after running
about 1 hour. According to the crash log, I fixed a missing
attach_function <a href="https://github.com/aurorafeint/typhoeus/commit/3ed63661e4f8e6b12e340fe3e84ba3a80b5c6f26">here</a>, but it didn't help. I ended up using
net-http-persistent in JRuby while using typhoeus in REE. From
performance test, I surprisingly found JRuby + net-http-persistent isn't
slower than REE + JRbuy.</p>
<p>2. <a href="https://github.com/evan/memcached">memcached</a>, it is the fastest memcached client ruby gems, it's
also a c extension gem. At first I used <a href="https://github.com/ikai/jruby-memcache-client">jruby-memcache-client</a>, but
jruby-memcache-client uses Base64 to encode/decode value, which can't
work with memcached gem together. Then I chose <a href="https://github.com/mperham/dalli">dalli</a> which supports
both REE and JRuby, but it uses different hash and distribution
algorithms, which causes too much cache misses on production. I searched
some other jruby memcached clients, but none of them are compatible with
memcached gem, I ended up writing <a href="https://github.com/aurorafeint/jruby-memcached">jruby-memcached gem</a> by myself
based on spymemcached. I wrote a post about this gem before, check it
out <a href="http://huangzhimin.com/2012/07/24/jruby-memcached-0-1-0-released/">here</a>.</p>
<h2>Enable threadsafe</h2>
<p>By default, threadsafe is disabled in rails 2.3.14, which means every
requests are locked by Rack::Lock, it's not a big deal when running in
multi-processes servers, like unicorn or passenger, but it loses the
JRuby's natvie multi-threads power. So make sure you enable the
threadsafe when migrating to JRuby.</p>
<p>Enabling threadsafe means rails won't automatically load libraries under
lib/ directory, you have to load them by yourselves.</p>
<p>Enabling threadsafe also means you must consider thread safety seriously.
OpenFeint platform uses long-running threads to communicate with scribe,
there is a eager loaded global queue and a lazy loaded thread for each
process, when doing performance test with JRuby + Torquebox, sometimes it
will genereate several lazy loaded threads, and finally cause memory
leak. The solution is to eager load the long running thread.</p>
<h2>Pass all tests</h2>
<p>It's a common sense that you must have good coverage unit, functional
and integration tests before doing a big migration. When all tests
were passed, I was confident to go further.</p>
<p><strong>Note:</strong> JRuby always eat much more memory to run memory, for openfeint
platform, I have to allocate 2 GB memory</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">JRUBY_OPTS</span><span class="o">=</span>-J-Xmx2g jruby --client -S bundle <span class="nb">exec </span>rake <span class="nb">test</span></code></pre></div>
<h2>Pick up a JRuby server</h2>
<p>There are 4 JRuby servers that I can choose</p>
<ul>
<li><a href="https://github.com/trinidad/trinidad">Trinidad</a>, built on JRuby::Rack and Tomcat.</li>
<li><a href="https://github.com/torquebox/torquebox">Torquebox</a>, built on JBoss AS.</li>
<li><a href="https://github.com/matadon/mizuno">Mizuno</a>, built on Jetty.</li>
<li><a href="https://github.com/puma/puma">Puma</a>, a new ruby web server built for concurrency.</li>
</ul>
<p>Puma depends on rack ~> 1.2 but rails 2.3.14 depends on ~> 1.1.0, so I
can't try Puma for OpenFeint platform.</p>
<p>I chose Torquebox from the other 3 servers, the reasons are as follows.</p>
<p>1. Torquebox runs faster than Trinidad and Mizuno according to our own
performance test, I think this is bacause Torquebox is mostly written
by Java while other servers are written by Ruby.
2. Some Torquebox core team members are paid by Red Hat to work on
Torquebox project, that means we can get better supports.
3. Torquebox project is very actively developing, and always keeps up
with latest JBoss AS server and JRuby.</p>
<p><strong>Note:</strong> Recently I replaced torquebox with <a href="https://github.com/torquebox/torquebox-lite">torquebox-lite</a>, which
is a smaller, web-only version of Torquebox, you can easily add other
jboss submodules when necessary.</p>
<h2>Monitor JVM</h2>
<p>Running on JVM is quite different than running on REE, you probably face
some new issues, like memory leak and thread safety. We uses
<a href="http://newrelic.com/">New Relic</a> to monitor response time, throughput, etc., but it
doesn't help to monitor jvm heap / non heap memory and thread stacks.
Fortunately we also use <a href="http://scoutapp.com/">scout</a> to monitor our servers, scout
provides JMX Monitoring plugin which collects the memory usage of jvm.
It is okay for production so far, but we will use <a href="http://www.zabbix.com/">zabbix</a> for
better monitoring in the future.</p>
<p>In Java world, there are a lot of monitor tools. Command tools like
jstat, jstack and jmap, graphical tools like jconsole and visualvm, you
can easily get the heap / non heap memory usage, gc stats, each thead
stack trace, etc.</p>
<p>It's really important to monitor JVM when doing performance / stress test,
it can help you find out memory leak and thread safe issues before
running on production. Here are 2 examples.</p>
<p>1. memory leak, I noticed that heap memory (both edge and old) reached
100% during stress test. Although no OutOfMemoryError raised, it was
definitely a memory leak. I used jmap to dump all heap memory and read
them by <a href="http://www.eclipse.org/mat/">Eclipse MAT</a>, here is the result.</p>
<p><img src="http://farm9.staticflickr.com/8196/8134695856_e06ba13e7f.jpg" alt="memory leak in eclipse mat" /></p>
<p>It's a typical memory leak, objects in container can't be gabarge
collected.</p>
<p>2. thread safe, I also found the db connection pool in activerecord
2.3.14 is not thread safe. The throughput will decline after running
a long time, I used jstack to dump all threads stack trace and saw
most of threads are locked in connection_pool as follows.</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="s2">"http--127.0.0.1-8180-1"</span> daemon <span class="nv">prio</span><span class="o">=</span><span class="m">10</span> <span class="nv">tid</span><span class="o">=</span>0x00007f4a17609800 <span class="nv">nid</span><span class="o">=</span>0x725a in Object.wait<span class="o">()</span> <span class="o">[</span>0x0000000049dfc000<span class="o">]</span>
java.lang.Thread.State: TIMED_WAITING <span class="o">(</span>on object monitor<span class="o">)</span>
at java.lang.Object.wait<span class="o">(</span>Native Method<span class="o">)</span>
- waiting on <0x0000000704f40e18> <span class="o">(</span>a org.jruby.libraries.ThreadLibrary<span class="nv">$ConditionVariable</span><span class="o">)</span>
......
at rubyjit.ActiveRecord::ConnectionAdapters::ConnectionPool#checkout_0978F3C1EFB2CBFA2CD717B12DA76E3113CD78B7.block_1<span class="nv">$RUBY$__file__</span><span class="o">(</span>/home/deploy/rails_apps/ openfeint_platform/shared/bundle/jruby/1.8/gems/activerecord-2.3.14/lib/active_record/connection_adapters/abstract/connection_pool.rb:192<span class="o">)</span>
at rubyjit<span class="nv">$ActiveRecord</span>::ConnectionAdapters::ConnectionPool#checkout_0978F3C1EFB2CBFA2CD717B12DA76E3113CD78B7<span class="nv">$block_1$RUBY$__file__</span>. call<span class="o">(</span>rubyjit<span class="nv">$ActiveRecord</span>::ConnectionAdapters::ConnectionPool#checkout_0978F3C1EFB2CBFA2CD717B12DA76E3113CD78B7<span class="nv">$block_1$RUBY$__file__</span>:65535<span class="o">)</span>
......</code></pre></div>
<p>But the count of http threads is equal to the count of db connections,
no thread should be locked. Considering our situation, I added a monkey
patch to connection_pool with one db connection per thread. It's not
perfect but works well.</p>
<h2>Tune JVM performance</h2>
<p>There are several jvm settings you should set for JRuby performance.</p>
<p>1. Xms and Xmx, when we hot deployed app to Torquebox by touching
-knob.yml.dodeploy, it took more than 20 minutes to complete, which was
unacceptable, after discussing with Torqeubox support team, I knew
default value for Xms is 64m and Xmx is 256m, they are too small, then I
increased them to 2g, it took only 100 seconds to hot deploy. The root
cause is hot deployment will increase memory a lot, which causing lots
of full GCs.</p>
<p>2. CodeCache, when we do the performance test, I found response time
suddenly jumped after running a few minutes, the torquebox log told me
"CodeCache is full. Compiler has been disabled." CodeCache is a part of
non heap memory in Hopspot JVM, it's 64m by default, so I increased it
to 256m by setting -XX:ReservedCodeCacheSize=256m, then I don't see
the response time jump anymore.</p>
<p>There are a lot of JVM parameters you can tune for your application,
talk and learn from some Java experts.</p>
<h2>Performance / Stress test</h2>
<p>I mentioned I already did a quick performance test, but it didn't make a
big sense, because qa and production have different environments. So
this time I did performance / stress tests on a reserved host, which has
the exactly same environments with production servers, connecting to
production database, memcache and redis servers.</p>
<p>Here are test results for actions in one pool.</p>
<table style="margin-bottom: 18px">
<tr>
<td></td>
<td> read action </td>
<td> write action </td>
</tr>
<tr>
<td>REE 1.8.7 + passenger</td>
<td> 448 ms</td>
<td> 44 ms</td>
</tr>
<tr>
<td>Ruby 1.9.3 + passenger</td>
<td> 374 ms</td>
<td> 42 ms</td>
</tr>
<tr>
<td>JRuby 1.7.0.RC2 + torquebox-lite </td>
<td> 187 ms</td>
<td> 38 ms</td>
</tr>
</table>
<p>JRuby is much faster than REE 1.8.7 and Ruby 1.9.3 in both read and
write actions. It's promising we can get a big performance improvement
on production.</p>
<p>Make sure you run your stress tests multiple times and run long time,
some memory leak and thread safety issues are not reproduced every time
or not occurred in a short time.</p>
<p><strong>Note:</strong> REE in reserved host is already optimized with <a href="https://gist.github.com/841168">twitter's
settings</a>.</p>
<h2>Deployment strategy</h2>
<p>Everything was ready, it was time to think about deployment strategy.</p>
<p>In Java world, you can deploy an app by packaging your source code into
a war file and copying the war package to app server. We can do the same
thing with JRuby, but it will break our existing capistrano deployment
script.</p>
<p>We kept existing capistrano deployment script except deploy:restart
task, replacing</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash">touch tmp/restart.txt</code></pre></div>
<p>with</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash">touch /opt/torquebox/current/jboss/standalone/deployments/openfeint_platform-knob.yml.dodeploy</code></pre></div>
<p>Torquebox will detect openfeint_platform-knob.yml.dodeploy, undeploy
old openfeint_platform and deploy new openfeint_platform, works very
similar to passenger. But I found everytime we redeploy app, the non
heap memory will jump a lot and the app will be super slow (multiple
times slower than usual) during redeployment process.</p>
<p>So I decided to deploy app by restarting jboss instead of hot
deployment.</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash">sudo /etc/init.d/jboss-as-standalone restart</code></pre></div>
<p>It solved memory issue, mitigated the slow requests, but introduced a
new issue, it will lost the requests during restarting jboss. The
solution we used is rolling restart to provide zero downtime
deployment, e.g. we have 3 app servers A, B, C</p>
<ol>
<li>tell load balancers stop sending http requests to server A.</li>
<li>restart jboss on server A.</li>
<li>tell load balancers resend http resquests to server A when jboss
on server A is ready.</li>
</ol>
<p>And restart server B and C one by one following the above steps.</p>
<p>So far, it works perfect, no memory jump and no request lost.</p>
<h2>JRuby on production</h2>
<p>Finally we successfully migrated to JRuby on production and the response
time dropped a lot.</p>
<p><img src="http://farm9.staticflickr.com/8328/8130182602_8106be24de.jpg" alt="performance improvement with JRuby" /></p>
<p>It was about 40% performance improvement, although it was expected, I
was still very excited. Actually after fully warming up, it run even
faster than you see on the figure.</p>
<p>The following is the response time comparing to ree's 1 week ago.</p>
<p><img src="http://farm9.staticflickr.com/8046/8134823364_f065545213.jpg" alt="jruby performance comparison" /></p>
<p>This is the successful migration for one pool on OpenFeint platform, we
have already migrated 5 pools to JRuby, all got ~ 40% performance
improvement. I'm still working on the rest pools' migration and looking
forward to replacing all OpenFeint servers to JRuby.</p>
<p>Some JRuby servers have been running on OpenFeint platform for more than
2 months, they are running stably and much faster than before according
to New Relic's weekly report.</p>
<h2>Further</h2>
<p>Java 7 introduced invokedynamic feature, a lot of people said
enabling invokedynamic made JRuby 1.7 run much faster, closer to Java
speed. But I'm failed tn enable invokedynamic feature with Torquebox,
saw the following error</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash">18:29:03,515 ERROR <span class="o">[</span>org.torquebox.core.runtime<span class="o">]</span> <span class="o">(</span>Thread-71<span class="o">)</span> Error during execution: ENV<span class="o">[</span><span class="s1">'RAILS_ROOT'</span><span class="o">]=</span>RACK_ROOT
ENV<span class="o">[</span><span class="s1">'RAILS_ENV'</span><span class="o">]=</span>RACK_ENV
require %q<span class="o">(</span>org/torquebox/web/rails/boot<span class="o">)</span>
: org.jruby.exceptions.RaiseException: <span class="o">(</span>LoadError<span class="o">)</span> load error: haml/buffer -- java.lang.NoClassDefFoundError: org/jruby/runtime/ThreadContext
at org.jruby.RubyKernel.require<span class="o">(</span>org/jruby/RubyKernel.java:1010<span class="o">)</span> <span class="o">[</span>jruby.jar:<span class="o">]</span>
at ActiveSupport::Dependencies::Loadable.require<span class="o">(</span>/home/deploy/rails_apps/openfeint_platform/shared/bundle/jruby/1.8/gems/activesupport-2.3.14/lib/active_support/dependencies.rb:182<span class="o">)</span></code></pre></div>
<p>Torquebox team is trying to fix this issue, I will definitely enable
invokedynamic with new Torquebox release, and am looking forward to
another big performance improvement.</p>
<h2>Some Resources</h2>
<p>If you join the JRuby world, the first thing you need to do is to follow
<a href="https://twitter.com/headius">Charles Nutter</a> on twitter, he is one of the JRuby core team
members and always shares a lot of JRuby knowledge. Also check out his
<a href="https://speakerdeck.com/headius">presentations</a> to get latest JRuby features and benchmarks.
<a href="https://github.com/jruby/jruby/wiki/_pages">JRuby wiki pages</a> are helpful to learn everything about JRuby.</p>
<p>At the end, please allow me to thank JRuby and Torquebox team for
providing such great things and thank <a href="http://product.gree.net/us/en">Gree</a> for allowing me to share
the knowledge.</p>
switch_user 0.9 released2012-10-31T00:00:00+00:00http://blog.huangzhimin.com/2012/10/31/switch_user-0.9-released<p><a href="https://github.com/flyerhzm/switch_user">switch_user</a> provides a convenient way to switch current user that
speeds up your development and reproduce user specified error on
production.</p>
<p>Today switch_user gem 0.9.0 is released, all thanks to <a href="https://github.com/lcowell">Luke Cowell</a>.</p>
<p>He is a collaborator of switch_user gem, and did a great job for 0.9.0
gem.</p>
<ol>
<li>did lots of refactors.</li>
<li>added unit tests.</li>
<li>made switch_user a rails engine.</li>
</ol>
<p>check out the changelog <a href="https://github.com/flyerhzm/switch_user/compare/v0.8.0...v0.9.0">here</a>.</p>
<p>I also updated <a href="https://github.com/flyerhzm/switch_user_example">switch_user example</a> to use switch_user 0.9.0.</p>
zero downtime deployment2012-10-23T00:00:00+00:00http://blog.huangzhimin.com/2012/10/23/zero-downtime-deployment<p>This is my new post on jrubytips. It teaches you how to achieve zero
downtime deployment for jruby servers.</p>
<p><a href="http://jrubytips.com/posts/5-zero-downtime-deployment">http://jrubytips.com/posts/5-zero-downtime-deployment</a></p>
set proper value for CodeCache2012-10-19T00:00:00+00:00http://blog.huangzhimin.com/2012/10/19/set-proper-value-for-codecache<p>This is my new post on jrubytips. It tells you the jvm CodeCache which
may affect your server performance.</p>
<p><a href="http://jrubytips.com/posts/4-set-proper-value-for-codecache">http://jrubytips.com/posts/4-set-proper-value-for-codecache</a></p>
newrelic-rake released2012-10-12T00:00:00+00:00http://blog.huangzhimin.com/2012/10/12/newrelic-rake-released<p>4 months ago, I released <a href="https://github.com/aurorafeint/newrelic-workling">newrelic-workling</a> gem, which helps us
montior the performance of background jobs. We used it to find out a
GC performance issue. But we still have some cron jobs, who call rake
tasks, running in the black box.</p>
<p>So I created a new project <a href="https://github.com/flyerhzm/newrelic-rake">newrelic-rake</a> that adds newrelic
instrument for rake tasks. Now when I go to the newrelic, I can see the
rake tasks listed in Background tasks section, it shows me the average
execution time and call count for all rake tasks.</p>
<p><img src="http://farm9.staticflickr.com/8467/8078580542_a85b59f8bd.jpg" alt="newrelic rake tasks" /></p>
<p>I can also see the performance breakdown for each rake task.</p>
<p><img src="http://farm9.staticflickr.com/8475/8078589421_7d3aa63972.jpg" alt="newrelic rake instrument" /></p>
<p>This rake task probably needs to use persistence net http or some c
extension http client, and reduce the GC calls.</p>
<p><strong>It's really important to do monitor first, then do optimize.</strong></p>
avoid using rubyzip2012-10-02T00:00:00+00:00http://blog.huangzhimin.com/2012/10/02/avoid-using-rubyzip<p><strong>More precisely I want to say allocating as less objects as you can,
rubyzip is just an example.</strong></p>
<p>We have a background job compressing webui assets, uploading to S3, so
mobile sdk can download assets to update webui dynamically.</p>
<p>After iphone5 and ios6 came to the market, we received much more webui
requests than before, it was expected, but our background job couldn't
consume so much asynchronous messages. We could easily scale out by adding
more background job servers, but I decided diving deeply into webui job to
see if I could speed it up and increase throughput.</p>
<p>Thank newrelic for providing great monitoring service, I saw the webui
job took averagely 725ms to complete a webui job, and 80% time was taken
by GC calls, WTF. Instead of blaming ruby gc, I blamed our bad code.</p>
<p>I noticed that we used <a href="https://rubygems.org/gems/rubyzip">rubyzip</a> to compress webui assets, it was the
root reason to cause so much GC.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">def</span> <span class="nf">create</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">files</span><span class="p">)</span>
<span class="no">Zip</span><span class="o">::</span><span class="no">ZipFile</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="no">Zip</span><span class="o">::</span><span class="no">ZipFile</span><span class="o">::</span><span class="no">CREATE</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">z</span><span class="o">|</span>
<span class="n">files</span><span class="o">.</span><span class="n">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">file</span><span class="o">|</span>
<span class="n">source_path</span> <span class="o">=</span> <span class="s2">"</span><span class="si">#{</span><span class="no">Rails</span><span class="o">.</span><span class="n">root</span><span class="si">}</span><span class="s2">/public/webui/</span><span class="si">#{</span><span class="n">file</span><span class="si">}</span><span class="s2">"</span>
<span class="n">expand_dirs</span><span class="p">(</span><span class="n">file</span><span class="p">)</span><span class="o">.</span><span class="n">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">dir</span><span class="o">|</span>
<span class="k">begin</span>
<span class="n">z</span><span class="o">.</span><span class="n">mkdir</span> <span class="n">dir</span>
<span class="k">rescue</span> <span class="no">Errno</span><span class="o">::</span><span class="no">EEXIST</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">z</span><span class="o">.</span><span class="n">add</span> <span class="n">file</span><span class="p">,</span> <span class="n">source_path</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></div>
<p>It sucks, all files are reading and compressing in ruby VM, too many
objects are allocated, then cause several GC calls. So I tried to use
shell zip command instead of rubyzip.</p>
<p>I did an experiment between rubyzip and shell zip. The followings are
code examples.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="nb">require</span> <span class="s1">'zip/zip'</span>
<span class="no">GC</span><span class="o">::</span><span class="no">Profiler</span><span class="o">.</span><span class="n">enable</span>
<span class="n">before_stats</span> <span class="o">=</span> <span class="no">ObjectSpace</span><span class="o">.</span><span class="n">count_objects</span>
<span class="n">start</span> <span class="o">=</span> <span class="no">Time</span><span class="o">.</span><span class="n">now</span>
<span class="no">Zip</span><span class="o">::</span><span class="no">ZipFile</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="s2">"test.zip"</span><span class="p">,</span> <span class="no">Zip</span><span class="o">::</span><span class="no">ZipFile</span><span class="o">::</span><span class="no">CREATE</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">z</span><span class="o">|</span>
<span class="no">Dir</span><span class="o">[</span><span class="s2">"**/*"</span><span class="o">].</span><span class="n">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">file</span><span class="o">|</span>
<span class="n">z</span><span class="o">.</span><span class="n">add</span> <span class="n">file</span><span class="p">,</span> <span class="n">file</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="nb">puts</span> <span class="s2">"Total time: </span><span class="si">#{</span><span class="no">Time</span><span class="o">.</span><span class="n">now</span> <span class="o">-</span> <span class="n">start</span><span class="si">}</span><span class="s2">"</span>
<span class="n">after_stats</span> <span class="o">=</span> <span class="no">ObjectSpace</span><span class="o">.</span><span class="n">count_objects</span>
<span class="nb">puts</span> <span class="s2">"[GC Stats] </span><span class="si">#{</span><span class="n">before_stats</span><span class="o">[</span><span class="ss">:FREE</span><span class="o">]</span> <span class="o">-</span> <span class="n">after_stats</span><span class="o">[</span><span class="ss">:FREE</span><span class="o">]</span><span class="si">}</span><span class="s2"> new allocated objects."</span>
<span class="c1"># Total time: 0.75344</span>
<span class="c1"># [GC Stats] 718691 new allocated objects.</span></code></pre></div>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="no">GC</span><span class="o">::</span><span class="no">Profiler</span><span class="o">.</span><span class="n">enable</span>
<span class="n">before_stats</span> <span class="o">=</span> <span class="no">ObjectSpace</span><span class="o">.</span><span class="n">count_objects</span>
<span class="n">start</span> <span class="o">=</span> <span class="no">Time</span><span class="o">.</span><span class="n">now</span>
<span class="n">files</span> <span class="o">=</span> <span class="no">Dir</span><span class="o">[</span><span class="s2">"**/*"</span><span class="o">].</span><span class="n">map</span> <span class="p">{</span> <span class="o">|</span><span class="n">file</span><span class="o">|</span> <span class="n">file</span> <span class="k">unless</span> <span class="no">File</span><span class="o">.</span><span class="n">directory?</span><span class="p">(</span><span class="n">file</span><span class="p">)</span> <span class="p">}</span>
<span class="sb">`zip test.zip </span><span class="si">#{</span><span class="n">files</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="s2">" "</span><span class="p">)</span><span class="si">}</span><span class="sb">`</span>
<span class="nb">puts</span> <span class="s2">"Total time: </span><span class="si">#{</span><span class="no">Time</span><span class="o">.</span><span class="n">now</span> <span class="o">-</span> <span class="n">start</span><span class="si">}</span><span class="s2">"</span>
<span class="n">after_stats</span> <span class="o">=</span> <span class="no">ObjectSpace</span><span class="o">.</span><span class="n">count_objects</span>
<span class="nb">puts</span> <span class="s2">"[GC Stats] </span><span class="si">#{</span><span class="n">before_stats</span><span class="o">[</span><span class="ss">:FREE</span><span class="o">]</span> <span class="o">-</span> <span class="n">after_stats</span><span class="o">[</span><span class="ss">:FREE</span><span class="o">]</span><span class="si">}</span><span class="s2"> new</span>
<span class="s2">allocated objects."</span>
<span class="c1"># Total time: 0.349816</span>
<span class="c1"># [GC Stats] 2269 new allocated objects.</span></code></pre></div>
<p>As you can see, rubyzip allocates > 700k objects for reading and
compressing, and it also takes more than double time to finish the
script, shell zip command is a much better solution. So I replaced
rubyzip with shell zip in our product.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">def</span> <span class="nf">create</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">files</span><span class="p">)</span>
<span class="sb">`cd </span><span class="si">#{</span><span class="no">Rails</span><span class="o">.</span><span class="n">root</span><span class="si">}</span><span class="sb">/public/webui && zip </span><span class="si">#{</span><span class="n">path</span><span class="si">}</span><span class="sb"> </span><span class="si">#{</span><span class="n">files</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="s1">' '</span><span class="p">)</span><span class="si">}</span><span class="sb">`</span>
<span class="k">end</span></code></pre></div>
<p>After deploying to background job server, I see a big performance
improved, it takes only 218ms for webui job to finish, and only 28%
time is taken by GC calls. The throughput is also increased from 44cpm
to 64cpm, and it can keep up with the webui asyncrhonous messages, we
don't need to add more servers, money saved. :-)</p>
<p>So keep in mind, allocating less objects means less GC calls, also means
better performance.</p>
<p><strong>Updated</strong>: <a href="https://rubygems.org/gems/zipruby">zip_ruby</a> gem gives a similar speed of shell zip command.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="nb">require</span> <span class="s1">'zipruby'</span>
<span class="no">GC</span><span class="o">::</span><span class="no">Profiler</span><span class="o">.</span><span class="n">enable</span>
<span class="n">before_stats</span> <span class="o">=</span> <span class="no">ObjectSpace</span><span class="o">.</span><span class="n">count_objects</span>
<span class="n">start</span> <span class="o">=</span> <span class="no">Time</span><span class="o">.</span><span class="n">now</span>
<span class="no">Zip</span><span class="o">::</span><span class="no">Archive</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="s2">"test.zip"</span><span class="p">,</span> <span class="no">Zip</span><span class="o">::</span><span class="no">CREATE</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">z</span><span class="o">|</span>
<span class="no">Dir</span><span class="o">[</span><span class="s2">"**/*"</span><span class="o">].</span><span class="n">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">file</span><span class="o">|</span>
<span class="n">z</span><span class="o">.</span><span class="n">add_file</span> <span class="n">file</span><span class="p">,</span> <span class="n">file</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="nb">puts</span> <span class="s2">"Total time: </span><span class="si">#{</span><span class="no">Time</span><span class="o">.</span><span class="n">now</span> <span class="o">-</span> <span class="n">start</span><span class="si">}</span><span class="s2">"</span>
<span class="n">after_stats</span> <span class="o">=</span> <span class="no">ObjectSpace</span><span class="o">.</span><span class="n">count_objects</span>
<span class="nb">puts</span> <span class="s2">"[GC Stats] </span><span class="si">#{</span><span class="n">before_stats</span><span class="o">[</span><span class="ss">:FREE</span><span class="o">]</span> <span class="o">-</span> <span class="n">after_stats</span><span class="o">[</span><span class="ss">:FREE</span><span class="o">]</span><span class="si">}</span><span class="s2"> new allocated objects."</span>
<span class="c1"># Total time: 0.367729</span>
<span class="c1"># [GC Stats] 1116 new allocated objects.</span></code></pre></div>
speed up git deployment with depth 12012-09-14T00:00:00+00:00http://blog.huangzhimin.com/2012/09/14/speed-up-git-deployment-with-git_shallow_clone<p>By default, when you deploy your application by capistrano git, it will
clone the repository with entire history on production server, but it's
meaningless. You should never go to production host and check git log,
instead you just need latest code on production host.</p>
<p>With your application grows, git clone with entire history may take a
bit longer time than you expected. The following is the time spent with
fully cloning.</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">time </span>git clone git@github.com:railsbp/rails-bestpractices.com.git
Cloning into <span class="s1">'rails-bestpractices.com'</span>...
remote: Counting objects: 11438, <span class="k">done</span>.
remote: Compressing objects: 100% <span class="o">(</span>3915/3915<span class="o">)</span>, <span class="k">done</span>.
remote: Total <span class="m">11438</span> <span class="o">(</span>delta 7012<span class="o">)</span>, reused <span class="m">11277</span> <span class="o">(</span>delta 6886<span class="o">)</span>
Receiving objects: 100% <span class="o">(</span>11438/11438<span class="o">)</span>, 5.52 MiB <span class="p">|</span> <span class="m">127</span> KiB/s, <span class="k">done</span>.
Resolving deltas: 100% <span class="o">(</span>7012/7012<span class="o">)</span>, <span class="k">done</span>.
git clone git@github.com:railsbp/rails-bestpractices.com.git 0.55s user 0.26s system 1% cpu 55.275 total</code></pre></div>
<p>But if clone with depth 1, it's finished much faster since there is only
1 revision fetched.</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">time </span>git clone --depth <span class="m">1</span> git@github.com:railsbp/rails-bestpractices.com.git
Cloning into <span class="s1">'rails-bestpractices.com'</span>...
remote: Counting objects: 1635, <span class="k">done</span>.
remote: Compressing objects: 100% <span class="o">(</span>1243/1243<span class="o">)</span>, <span class="k">done</span>.
remote: Total <span class="m">1635</span> <span class="o">(</span>delta 265<span class="o">)</span>, reused <span class="m">1367</span> <span class="o">(</span>delta 189<span class="o">)</span>
Receiving objects: 100% <span class="o">(</span>1635/1635<span class="o">)</span>, 3.02 MiB <span class="p">|</span> <span class="m">134</span> KiB/s, <span class="k">done</span>.
Resolving deltas: 100% <span class="o">(</span>265/265<span class="o">)</span>, <span class="k">done</span>.
git clone --depth <span class="m">1</span> git@github.com:railsbp/rails-bestpractices.com.git 0.24s user 0.17s system 1% cpu 34.236 total</code></pre></div>
<p>It's time to apply this on your capistrano file to speed up your
deployment.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">set</span> <span class="ss">:scm</span><span class="p">,</span> <span class="ss">:git</span>
<span class="n">set</span> <span class="ss">:git_shallow_clone</span><span class="p">,</span> <span class="mi">1</span></code></pre></div>
<p><strong>Warning</strong>: git_shallow_clone can't work with branch.</p>
enable threadsafe for rails2012-09-09T00:00:00+00:00http://blog.huangzhimin.com/2012/09/09/enable-threadsafe-for-rails<p>This is my new post on jrubytips. It tells what threadsafe is in rails,
and what're the benefits with threadsafe in jruby server.</p>
<p><a href="http://jrubytips.com/posts/3-enable-threadsafe-for-rails">http://jrubytips.com/posts/3-enable-threadsafe-for-rails</a></p>
rolling out with feature flags2012-09-02T00:00:00+00:00http://blog.huangzhimin.com/2012/09/02/rolling-out-with-feature-flags<p>This is my new post on rails-bestpractices. It tells how we use feature
flags to rolling out our features.</p>
<p><a href="http://rails-bestpractices.com/posts/697-rolling-out-with-feature-flags">http://rails-bestpractices.com/posts/697-rolling-out-with-feature-flags</a></p>
how to write a jruby gem - part 22012-08-23T00:00:00+00:00http://blog.huangzhimin.com/2012/08/23/how-to-write-a-jruby-gem-part-2<p>In my <a href="http://huangzhimin.com/2012/08/06/how-to-write-a-jruby-gem-part-1/">previous post</a>, I introduced how to write a jruby gem with
ruby code, today I will show you how to write a jruby extension with
java code, which can give you better performance.</p>
<h3>Standard Steps</h3>
<p>1. create java classes to wrap any java library you need, and the java
classes must extend RubyObject, then it can be called from jruby. e.g.</p>
<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">class</span> <span class="nc">Memcached</span> <span class="kd">extends</span> <span class="n">RubyObject</span> <span class="o">{</span>
<span class="c1">// MemcachedClient is what we want to wrap</span>
<span class="kd">private</span> <span class="n">MemcachedClient</span> <span class="n">client</span><span class="o">;</span>
<span class="c1">// java constructor</span>
<span class="kd">public</span> <span class="nf">Memcached</span><span class="o">(</span><span class="kd">final</span> <span class="n">Ruby</span> <span class="n">ruby</span><span class="o">,</span> <span class="n">RubyClass</span> <span class="n">rubyClass</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">super</span><span class="o">(</span><span class="n">ruby</span><span class="o">,</span> <span class="n">rubyClass</span><span class="o">);</span>
<span class="o">}</span>
<span class="c1">// ruby initialize</span>
<span class="kd">public</span> <span class="n">IRubyObject</span> <span class="nf">initialize</span><span class="o">(</span><span class="n">ThreadContext</span> <span class="n">context</span><span class="o">)</span> <span class="o">{</span>
<span class="n">client</span> <span class="o">=</span> <span class="n">MemcachedClient</span><span class="o">.</span><span class="na">new</span><span class="o">();</span>
<span class="o">}</span>
<span class="c1">// wrapper method, the first argument for jruby methods must be ThreadContext</span>
<span class="kd">public</span> <span class="n">IRubyObject</span> <span class="nf">get</span><span class="o">(</span><span class="n">ThreadContext</span> <span class="n">context</span><span class="o">,</span> <span class="n">IRubyObject</span> <span class="n">key</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="o">(</span><span class="n">IRubyObject</span><span class="o">)</span> <span class="n">client</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">key</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span></code></pre></div>
<p>Keep in mind, every objects you read from ruby or return to ruby must be
a RubyObject. So you have to convert between RubyObject and java Object in
your wrapper methods.</p>
<p>2. add JRubyModule, JRubyClass, JRubyMethod and JRubyConstant annotations.</p>
<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="nd">@JRubyClass</span>
<span class="kd">class</span> <span class="nc">Memcached</span> <span class="kd">extends</span> <span class="n">RubyObject</span> <span class="o">{</span>
<span class="nd">@JRubyMethod</span>
<span class="kd">public</span> <span class="n">IRubyObject</span> <span class="nf">initialize</span><span class="o">(</span><span class="n">ThreadContext</span> <span class="n">context</span><span class="o">)</span> <span class="o">{</span>
<span class="n">client</span> <span class="o">=</span> <span class="n">MemcachedClient</span><span class="o">.</span><span class="na">new</span><span class="o">();</span>
<span class="o">}</span>
<span class="nd">@JRubyMethod</span>
<span class="kd">public</span> <span class="n">IRubyObject</span> <span class="nf">get</span><span class="o">(</span><span class="n">ThreadContext</span> <span class="n">context</span><span class="o">,</span> <span class="n">IRubyObject</span> <span class="n">key</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="o">(</span><span class="n">IRubyObject</span><span class="o">)</span> <span class="n">client</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">key</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span></code></pre></div>
<p>JRuby annotations tells jvm which classes and methods should be open to
ruby world. It can tell the details of classes and methods, like
what's the parent class, how many arguments of a methods, and so on.</p>
<p>3. load all jruby modules, classes and methods with BasicLibraryService.</p>
<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MemcachedService</span> <span class="kd">implements</span> <span class="n">BasicLibraryService</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">basicLoad</span><span class="o">(</span><span class="kd">final</span> <span class="n">Ruby</span> <span class="n">ruby</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span> <span class="o">{</span>
<span class="c1">// define Memcached class</span>
<span class="n">RubyClass</span> <span class="n">memcached</span> <span class="o">=</span> <span class="n">ruby</span><span class="o">.</span><span class="na">defineClass</span><span class="o">(</span><span class="s">"Memcached"</span><span class="o">,</span> <span class="n">ruby</span><span class="o">.</span><span class="na">getObject</span><span class="o">(),</span> <span class="k">new</span> <span class="nf">ObjectAllocator</span><span class="o">()</span> <span class="o">{</span>
<span class="kd">public</span> <span class="n">IRubyObject</span> <span class="nf">allocate</span><span class="o">(</span><span class="n">Ruby</span> <span class="n">ruby</span><span class="o">,</span> <span class="n">RubyClass</span> <span class="n">klazz</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">Memcached</span><span class="o">(</span><span class="n">ruby</span><span class="o">,</span> <span class="n">klazz</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">});</span>
<span class="c1">// define all methods with @JRubyMethods in Memcached class</span>
<span class="n">memcached</span><span class="o">.</span><span class="na">defineAnnotatedMethods</span><span class="o">(</span><span class="n">Memcached</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
<span class="k">return</span> <span class="kc">true</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span></code></pre></div>
<p>BasicLibraryService is the standard load mechanism for easy extensions,
you should implement basicLoad method to define ruby modules, classes
and methods.</p>
<p>4. finally, load MemcachedService in your ruby file</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="c1"># MemcachedService is in com.openfeint.memcached package</span>
<span class="nb">require</span> <span class="s1">'com/openfeint/memcached/memcached'</span></code></pre></div>
<p>Then you can load your jruby gem, and use any Memcached classes and
methods you defined.</p>
<h3>Some Advanced Tips:</h3>
<p>1. JRuby method names.</p>
<p><strong>different name</strong></p>
<p>in ruby</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">def</span> <span class="nf">active?</span>
<span class="k">end</span></code></pre></div>
<p>in java</p>
<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="nd">@JRubyMethod</span><span class="o">(</span><span class="n">name</span> <span class="o">=</span> <span class="s">"active?"</span><span class="o">)</span>
<span class="kd">public</span> <span class="n">IRubyObject</span> <span class="nf">active_p</span><span class="o">(</span><span class="n">ThreadContext</span> <span class="n">context</span><span class="o">)</span> <span class="o">{</span>
<span class="o">}</span></code></pre></div>
<p><strong>alias methods</strong></p>
<p>in ruby</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">alias</span> <span class="ss">:"[]"</span> <span class="ss">:get</span></code></pre></div>
<p>in java</p>
<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="nd">@JRubyMethod</span><span class="o">(</span><span class="n">name</span> <span class="o">=</span> <span class="o">{</span> <span class="s">"get"</span><span class="o">,</span> <span class="s">"[]"</span> <span class="o">})</span>
<span class="kd">public</span> <span class="n">IRubyObject</span> <span class="nf">get</span><span class="o">(</span><span class="n">ThreadContext</span> <span class="n">context</span><span class="o">,</span> <span class="n">IRubyObject</span> <span class="n">key</span><span class="o">)</span> <span class="o">{</span>
<span class="o">}</span></code></pre></div>
<p>2. JRuby method arguments.</p>
<p><strong>rest arguments</strong></p>
<p>in ruby</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">)</span>
<span class="k">end</span></code></pre></div>
<p>in java</p>
<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="nd">@JRubyMethod</span><span class="o">(</span><span class="n">name</span> <span class="o">=</span> <span class="s">"initialize"</span><span class="o">,</span> <span class="n">rest</span> <span class="o">=</span> <span class="kc">true</span><span class="o">)</span>
<span class="kd">public</span> <span class="n">IRubyObject</span> <span class="nf">initialize</span><span class="o">(</span><span class="n">ThreadContext</span> <span class="n">context</span><span class="o">,</span> <span class="n">IRubyObject</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="o">}</span></code></pre></div>
<p><strong>arguments with default value</strong></p>
<p>in ruby</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">marshal</span><span class="o">=</span><span class="kp">true</span><span class="p">)</span>
<span class="k">end</span></code></pre></div>
<p>in java</p>
<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="nd">@JRubyMethod</span><span class="o">(</span><span class="n">name</span> <span class="o">=</span> <span class="s">"get"</span><span class="o">,</span> <span class="n">required</span> <span class="o">=</span> <span class="mi">1</span><span class="o">,</span> <span class="n">optional</span> <span class="o">=</span> <span class="mi">1</span><span class="o">)</span>
<span class="kd">public</span> <span class="n">IRubyObject</span> <span class="nf">get</span><span class="o">(</span><span class="n">ThreadContext</span> <span class="n">context</span><span class="o">,</span> <span class="n">IRubyObject</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="n">Ruby</span> <span class="n">ruby</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="na">getRuntime</span><span class="o">();</span>
<span class="n">RubyString</span> <span class="n">key</span> <span class="o">=</span> <span class="o">(</span><span class="n">RubyString</span><span class="o">)</span> <span class="n">args</span><span class="o">[</span><span class="mi">0</span><span class="o">];</span>
<span class="n">RubyBoolean</span> <span class="n">marshal</span> <span class="o">=</span> <span class="n">ruby</span><span class="o">.</span><span class="na">getTrue</span><span class="o">();</span>
<span class="k">if</span> <span class="o">(</span><span class="n">args</span><span class="o">.</span><span class="na">length</span> <span class="o">></span> <span class="mi">1</span><span class="o">)</span> <span class="o">{</span>
<span class="n">marshal</span> <span class="o">=</span> <span class="n">args</span><span class="o">[</span><span class="mi">1</span><span class="o">];</span>
<span class="o">}</span>
<span class="o">}</span></code></pre></div>
<p>3. custom exceptions</p>
<p>Exception is also a class, so, you could define an Exception in jruby
just like defining a class.</p>
<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="nd">@JRubyClass</span><span class="o">(</span><span class="n">name</span> <span class="o">=</span> <span class="s">"Memcached::Error"</span><span class="o">,</span> <span class="n">parent</span> <span class="o">=</span> <span class="s">"RuntimeError"</span><span class="o">)</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">Error</span> <span class="o">{</span>
<span class="c1">// you should wrap your custom exception with RaiseException for java land throwing purpose.</span>
<span class="kd">public</span> <span class="kd">static</span> <span class="n">RaiseException</span> <span class="nf">newNotFound</span><span class="o">(</span><span class="n">Ruby</span> <span class="n">ruby</span><span class="o">,</span> <span class="n">String</span> <span class="n">message</span><span class="o">)</span> <span class="o">{</span>
<span class="n">RubyClass</span> <span class="n">errorClass</span> <span class="o">=</span> <span class="n">ruby</span><span class="o">.</span><span class="na">getModule</span><span class="o">(</span><span class="s">"Memcached"</span><span class="o">).</span><span class="na">getClass</span><span class="o">(</span><span class="s">"NotFound"</span><span class="o">);</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">RaiseException</span><span class="o">(</span><span class="n">RubyException</span><span class="o">.</span><span class="na">newException</span><span class="o">(</span><span class="n">ruby</span><span class="o">,</span> <span class="n">errorClass</span><span class="o">,</span> <span class="n">message</span><span class="o">),</span> <span class="kc">true</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="c1">// Yes, it is a subclass.</span>
<span class="nd">@JRubyClass</span><span class="o">(</span><span class="n">name</span><span class="o">=</span><span class="s">"Memcached::NotFound"</span><span class="o">,</span> <span class="n">parent</span><span class="o">=</span><span class="s">"Memcached::Error"</span><span class="o">)</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">NotFound</span> <span class="kd">extends</span> <span class="n">Error</span> <span class="o">{</span>
<span class="o">}</span>
<span class="c1">// Finally, load the Error in MemcachedService.</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">MemcachedService</span> <span class="kd">implements</span> <span class="n">BasicLibraryService</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">basicLoad</span><span class="o">(</span><span class="kd">final</span> <span class="n">Ruby</span> <span class="n">ruby</span><span class="o">)</span> <span class="kd">throws</span> <span class="n">IOException</span> <span class="o">{</span>
<span class="n">RubyClass</span> <span class="n">runtimeError</span> <span class="o">=</span> <span class="n">ruby</span><span class="o">.</span><span class="na">getRuntimeError</span><span class="o">();</span>
<span class="n">RubyClass</span> <span class="n">memcachedError</span> <span class="o">=</span> <span class="n">memcached</span><span class="o">.</span><span class="na">defineClassUnder</span><span class="o">(</span><span class="s">"Error"</span><span class="o">,</span> <span class="n">runtimeError</span><span class="o">,</span> <span class="n">runtimeError</span><span class="o">.</span><span class="na">getAllocator</span><span class="o">());</span>
<span class="n">memcached</span><span class="o">.</span><span class="na">defineClassUnder</span><span class="o">(</span><span class="s">"NotFound"</span><span class="o">,</span> <span class="n">memcachedError</span><span class="o">,</span> <span class="n">memcachedError</span><span class="o">.</span><span class="na">getAllocator</span><span class="o">());</span>
<span class="k">return</span> <span class="kc">true</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span></code></pre></div>
<p>so when your call Error.newNotFound(ruby, "Not Found") in your java
code, it can be catched with Memcached::NotFound in ruby.</p>
<p>4. object convertion</p>
<p><strong>RubyObject to java Object</strong></p>
<p>you can use RubyObject convertToXXX methods</p>
<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">convertToArray</span>
<span class="n">convertToFloat</span>
<span class="n">convertToHash</span>
<span class="n">convertToInteger</span>
<span class="n">convertToString</span></code></pre></div>
<p>e.g.</p>
<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">List</span><span class="o"><</span><span class="n">String</span><span class="o">></span> <span class="n">keys</span> <span class="o">=</span> <span class="o">(</span><span class="n">List</span><span class="o"><</span><span class="n">String</span><span class="o">>)</span> <span class="n">args</span><span class="o">.</span><span class="na">convertToArray</span><span class="o">();</span></code></pre></div>
<p><strong>java Object to RubyObject</strong></p>
<p>you can use Ruby newXXX methods</p>
<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">newArray</span>
<span class="n">newBoolean</span>
<span class="n">newFixnum</span>
<span class="n">newFloat</span>
<span class="n">newString</span></code></pre></div>
<p>e.g.</p>
<div class="highlight"><pre><code class="language-java" data-lang="java"><span class="n">ruby</span><span class="o">.</span><span class="na">newString</span><span class="o">(</span><span class="s">"hello world"</span><span class="o">);</span></code></pre></div>
<p>You can read the source code of <a href="https://github.com/aurorafeint/jruby-memcached">jruby-memcached</a> to get more
information. Feel free to leave a comment if you have any question or
suggestion.</p>
jruby-memcached 0.5.0 released2012-08-22T00:00:00+00:00http://blog.huangzhimin.com/2012/08/22/jruby-memcached-0-5-0-released<p>I just released <a href="https://rubygems.org/gems/jruby-memcached/versions/0.5.0">jruby-memcached 0.5.0</a>, it contains the following
changes:</p>
<ol>
<li>add travis-ci support, testing jruby-18mode, jruby-19mode and
jruby-head environment.</li>
<li>update spymemcached to 2.8.3, which set shouldOptimize to false by
default, there are some bugs with true shouldOptimize so far.</li>
<li>fix increment/decrement issue, in < 0.5.0, incr/decr with unmarshal
encode while get with marshal decode.</li>
<li>accept exception_retry_limit option.</li>
<li>add Memcached::ATimeoutOccurred error to handle timeout case,
otherwise you will probably see following error.</li>
</ol>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="no">ActionView</span><span class="o">::</span><span class="ss">TemplateError</span><span class="p">:</span> <span class="n">undefined</span> <span class="nb">method</span> <span class="sb">`clean_message' for #<Java::NetSpyMemcached::OperationTimeoutException:0x26e02e71></span></code></pre></div>
<p>check out the full code changes <a href="https://github.com/aurorafeint/jruby-memcached/compare/v0.4.0...v0.5.0">here</a>.</p>
jruby-memcached 0.4.0 released2012-08-17T00:00:00+00:00http://blog.huangzhimin.com/2012/08/17/jruby-memcached-0-4-0-released<p>I just released <a href="https://rubygems.org/gems/jruby-memcached/versions/0.4.0">jruby-memcached 0.4.0</a>, it contains the following
changes:</p>
<ol>
<li>run spymemcached as a daemon thread. I found when running rake task
with jruby-memcached < 0.4.0, it won't stop unless you press Ctrl+C.</li>
<li>get method can accept multiple keys.</li>
<li>add Memcached::Rails as a rails cache_store. Of course, it is
compatible with Memcached::Rails in memcached.gem.</li>
<li>make full use of jruby annotation to reduce method definitions with
optional and rest arguments.</li>
</ol>
<p>check out the full code changes <a href="https://github.com/aurorafeint/jruby-memcached/compare/v0.3.0...v0.4.0">here</a>.</p>
jruby-memcached 0.3.0 released2012-08-07T00:00:00+00:00http://blog.huangzhimin.com/2012/08/07/jruby-memcached-0-3-0-released<p>I just released <a href="https://rubygems.org/gems/jruby-memcached/versions/0.3.0">jruby-memcached 0.3.0</a>, it runs about 10%-20%
faster than 0.2.0, I removed ruby code and totally wrote it by java
code, check out the <a href="https://github.com/aurorafeint/jruby-memcached/compare/v0.2.0...v0.3.0">file changes</a>.</p>
<p>2 weeks ago, I released jruby-memcached 0.1.0, in <a href="http://huangzhimin.com/2012/07/24/jruby-memcached-0-1-0-released/">that post</a> I
mentioned jruby-memcached response time in a request is 40+ms while
memcached.gem response time is 30+ms, it looked fine, but I was
still investigating the way to improve jruby-memcached performance.</p>
<p>After reading <a href="https://github.com/headius/jruby-spymemcached">jruby-spymemcached</a> gem and jruby source code, I
rewrote jruby-memcached by pure java code instead of ruby code,
because calling java from java is much faster than from ruby.</p>
<p>I did the performance compare with new jruby-memcached, the result is
as follows:</p>
<div class="highlight"><pre><code class="language-text" data-lang="text">MBP 2.8G i7 jruby-memcached 0.3.0
ruby-1.9.3-p194
user system total real
memcached set 1.110000 1.020000 2.130000 ( 4.592509)
memcached get 0.970000 1.000000 1.970000 ( 4.172170)
user system total real
dalli set 8.360000 1.650000 10.010000 ( 10.193101)
dalli get 8.040000 1.670000 9.710000 ( 9.828392)
jruby-1.6.7.2
user system total real
jruby-memcached set 5.842000 0.000000 5.842000 ( 5.842000)
jruby-memcached get 5.561000 0.000000 5.561000 ( 5.561000)
user system total real
jruby-spymemcached set 5.919000 0.000000 5.919000 ( 5.919000)
jruby-spymemcached get 5.615000 0.000000 5.615000 ( 5.615000)
user system total real
dalli set 10.132000 0.000000 10.132000 ( 10.132000)
dalli get 10.600000 0.000000 10.600000 ( 10.600000)</code></pre></div>
<p>As you can see, jruby-memcached runs as fast as jruby-spymemcached, and
it provides memcached.gem compatible apis and hashing algorithm.
jruby-memcached is still slower than memcached.gem, and on production,
the response time for memcached has reduced to 40-ms, which is very
close to the memcached.gem performance.</p>
how to write a jruby gem - part 12012-08-06T00:00:00+00:00http://blog.huangzhimin.com/2012/08/06/how-to-write-a-jruby-gem-part-1<p>In my <a href="http://huangzhimin.com/2012/07/24/jruby-memcached-0-1-0-released/">previous post</a>, I mentioned I have written a jruby memcached
gem. I'm glad to share my experience how to extend jruby here.</p>
<p>JRuby is a 100% java implementation of ruby programming language, it
allows you calling java code from ruby code. Java world has much more
libraries than ruby gems, to make use of those java jar, it makes your
code easier and faster.</p>
<p>I assume you already had the experience to create a pure ruby gem, the
first step to create a jruby gem is just the same as ruby gem, the gem
structure is as follows:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="p">|</span>- lib/ // ruby implementation code
<span class="p">|</span> <span class="p">|</span>- memcached/
<span class="p">|</span> <span class="p">|</span>- memcached.rb
<span class="p">|</span>- <span class="nb">test</span>/ // ruby <span class="nb">test </span>code <span class="o">(</span>rspec or minitest<span class="o">)</span>
<span class="p">|</span> <span class="p">|</span>- memcached/
<span class="p">|</span> <span class="p">|</span>- memcached_test.rb
<span class="p">|</span> <span class="p">|</span>- test_helper.rb
<span class="p">|</span>- Gemfile
<span class="p">|</span>- jruby_memcached.gemspec // your gem manifest
<span class="p">|</span>- Rakefile
<span class="p">|</span>- README.md</code></pre></div>
<p>Then let's introduce the java jar into our jruby gem.</p>
<p>It's well-known to use maven2 to manage your java source code and
dependencies, maven uses pom.xml as a config file to define compile,
test and package processes, it also defines the dependencies, looks
like the combination of rake and bundler. All the java implementation
and test code are put in src directory, while compiled classes and jar
files are put in target directory. Now the structure looks like:</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="p">|</span>- lib/
<span class="p">|</span> <span class="p">|</span>- memcached/
<span class="p">|</span> <span class="p">|</span>- memcached.rb
<span class="p">|</span>- <span class="nb">test</span>/
<span class="p">|</span> <span class="p">|</span>- memcached/
<span class="p">|</span> <span class="p">|</span>- memcached_test.rb
<span class="p">|</span> <span class="p">|</span>- test_helper.rb
<span class="p">|</span>- src/ // java <span class="nb">source </span>code
<span class="p">|</span> <span class="p">|</span>- main/ // java implementation code
<span class="p">|</span> <span class="p">|</span> <span class="p">|</span>- java/
<span class="p">|</span> <span class="p">|</span>- <span class="nb">test</span> // java <span class="nb">test </span>code
<span class="p">|</span> <span class="p">|</span>- java/
<span class="p">|</span>- target/
<span class="p">|</span> <span class="p">|</span>- classes/ // compiled classes files
<span class="p">|</span> <span class="p">|</span>- spymemcached-ext-0.0.1.jar // package java <span class="nb">source </span>code to a jar
<span class="p">|</span>- Gemfile
<span class="p">|</span>- jruby_memcached.gemspec
<span class="p">|</span>- pom.xml // maven config file, compile, <span class="nb">test</span>, package
<span class="p">|</span>- Rakefile
<span class="p">|</span>- README.md</code></pre></div>
<p>In pom.xml, I said it depends on spymemcached 2.8.1 jar, so I can import
spymemcached in my hack code under src/main/java. I also defined package
shade plugin which package spymemcached 2.8.1 jar and my hack code
together into target/spymemcached-ext-0.0.1.jar.</p>
<p>The last step is to combine the ruby and java code. JRuby provides the
power to easily use spymemcached-ext-0.0.1.jar in ruby code.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="nb">require</span> <span class="s1">'target/spymemcached-ext-0.0.1.jar'</span>
<span class="n">java_import</span> <span class="s1">'net.spy.memcached.MemcachedClient'</span>
<span class="n">java_import</span> <span class="s1">'net.spy.memcached.ConnectionFactoryBuilder'</span>
<span class="n">java_import</span> <span class="s1">'net.spy.memcached.AddrUtil'</span>
<span class="n">builder</span> <span class="o">=</span> <span class="no">ConnectionFactoryBuilder</span><span class="o">.</span><span class="n">new</span>
<span class="vi">@client</span> <span class="o">=</span> <span class="no">MemcachedClient</span><span class="o">.</span><span class="n">new</span> <span class="n">builder</span><span class="o">.</span><span class="n">build</span><span class="p">,</span> <span class="no">AddrUtil</span><span class="o">.</span><span class="n">getAddresses</span><span class="p">(</span><span class="nb">Array</span><span class="p">(</span><span class="n">addresses</span><span class="p">)</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="s1">' '</span><span class="p">))</span></code></pre></div>
<p>As you seen, after require the java jar file, you can import the java
classes and call the java methods with ruby syntax, jruby is smart enough
to convert ruby code into java code. Check out more about how to calling
java from jruby <a href="https://github.com/jruby/jruby/wiki/CallingJavaFromJRuby">here</a>.</p>
<p>You can check out jruby-memcached 0.2.0 source code <a href="https://github.com/aurorafeint/jruby-memcached/tree/2adc85e8121229527a57a71f221fdade40de61df">here</a> to get
more details. This is the simplest solution to create jruby gem, in
next post I will introduce you how to write a real jruby ext, which can
improve the performance of your jruby gem.</p>
<p><strong>Update</strong>: I have written <a href="http://huangzhimin.com/2012/08/23/how-to-write-a-jruby-gem-part-2/">part 2</a> for a real jruby ext, don't miss it.</p>
jruby-memcached 0.1.0 released2012-07-24T00:00:00+00:00http://blog.huangzhimin.com/2012/07/24/jruby-memcached-0-1-0-released<p>I just released <a href="https://github.com/aurorafeint/jruby-memcached">jruby-memcached</a> 0.1.0 gem, which is the fastest
jruby memcached client so far and it is also compatible with
memcached.gem. The following is the story why I created
jruby-memcached gem.</p>
<p>We are trying to migrate our service from ree to jruby. It's a big
project for us, as our repository is written from early 2009, it becomes
bigger and bigger, and nobody can promise migrating it to jruby without
any errors. Fortunately we are separating our service into different
pools, like one pool to handle high scores requests, one pool to handle
achievements requests, so our strategy is to migrate to jruby one pool
by one pool, it makes migrating processes easier, everytime we only focus
on one pool.</p>
<p>This raises one problem, we are using <a href="https://github.com/evan/memcached">evan's memcached gem</a> which is
the fastest memcached client for MRI, but it isn't working in jruby, yes,
it's a ruby gem with c extention. But we have to solve the situation that
memcached client must work on both ree pool and jruby pool.</p>
<p>The first idea in my mind is to use a jruby memcached client, after
googling I found <a href="https://github.com/ikai/jruby-memcache-client">jruby-memcached-client</a>, but soon I get to know
they can't be used together. jruby-memcached-client marshal dump the
value, encode the value to base64 then save to memcached, and memcached
gem only marshal dump, do not encode to base 64. Althrough I can fork
and change jruby-memcached-client, but there is still a string issue
passing from ruby to jave, I will mention it later.</p>
<p>So I have to give up jruby-memcached-client, then I try to use pure ruby
memcached client like memcache-client or dalli. I pick up <a href="https://github.com/mperham/dalli">dalli</a>
which is faster than memcached-client, but after we deployed it on
production, we found memcached misses jump too high which we can't
afford, see</p>
<p><img src="http://farm8.staticflickr.com/7128/7635380018_69f4cc5247.jpg" alt="Memcached misses" /></p>
<p>we have to revert the release.</p>
<p>I did some research about memcached, then I realized memcached and dalli
are incompatible, why? You may think memcached is simple, just set
key/value pair and get value based on key. It is right in common, but if
you have more than 1 memcached servers, memcached client should know
what key is on which memcached server, client must not fetch the key from
server1 now but fetch it from server2 10 minutes later.</p>
<p>Let me introduce you 2 important client configurations</p>
<ol>
<li>Hashing, it is the algorithm to convert you string key to long hash.</li>
<li>Consistent Hashing, aka distribution in memcached gem, it is the
algorithm to map you generated hash to one of your memcached servers,
it promises low ratio cache reassigns when you add or remove a
memcached server. See more about consistent hashing on <a href="http://en.wikipedia.org/wiki/Consistent_hashing">wikipedia</a>.</li>
</ol>
<p>memcached uses fnv1_32 as hash algorithm by default and dalli uses
crc32, and their distribution algorithm are not compatible as well.</p>
<p>Finally I decided to write a jruby memcached based on a java memcached
library by myself. After googling, I find 2 options: <a href="http://code.google.com/p/xmemcached/">xmemcached</a> and
<a href="http://code.google.com/p/spymemcached/">spymemcached</a>. The author of xmemcached shows the benchmark which
said xmemcached is a bit faster than spymemcached and it provides
libmemcached compatible hasing algorithm (although I was cheated in the
end :-) ), so I gave it a try first.</p>
<p>xmemcached's LibmemcachedMemcachedSessionLocator is not compatbile with
libmemcached, at least not compatible with libmemcached 0.32 which is
used by memcached gem. I have to dive into libmemcached 0.32 source code
and override xmemcached MemcachedSessionLocator, and write a jruby gem
to wrap the xmemcached. (writing a jruby gem is not difficult, I
probably write a new post to introduce in the future) Then I released it
on our reverse pool, sending high traffic to see the performance, I'm
disappointed, memcached get time increased from 30+ ms to 60 ms, and it
generated about 200 threads for xmemcached (we have 30 memcached servers
and 2 memcached client instances).</p>
<p>Quickly I replaced xmemcached to spymemcached, and memcached get time
decreased to 40+ ms and it only generates 2 threads, awesome. And its
hash and distribution algorithms are 100% compatible to libmemcached
0.32. You can read the source code in <a href="https://github.com/aurorafeint/jruby-memcached/tree/master/src/main/java">src/main/java</a> to see all
hacks I did for spymemcached.</p>
<p>I mention a string issue above, it is when we passing a zlib deflated
value, like "x\234c?P?<em>?/?I\001\000\b8\002a", it changes to
"x?c?P?</em>?/?8a" in java, so we can't pass deflated string directly,
instead we pass bytes.</p>
<p>I also did some benchmark between memcached, jruby-memcached and dalli.</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash">in ruby-1.9.3
user system total real
memcached <span class="nb">set </span>1.110000 1.020000 2.130000 <span class="o">(</span> 4.592509<span class="o">)</span>
memcached get 0.970000 1.000000 1.970000 <span class="o">(</span> 4.172170<span class="o">)</span>
user system total real
dalli <span class="nb">set </span>8.330000 1.560000 9.890000 <span class="o">(</span> 10.094499<span class="o">)</span>
dalli get 8.530000 1.680000 10.210000 <span class="o">(</span> 10.331083<span class="o">)</span>
in jruby-1.6.7.2
user system total real
jruby-memcached <span class="nb">set </span>6.902000 0.000000 6.902000 <span class="o">(</span> 6.902000<span class="o">)</span>
jruby-memcached get 6.845000 0.000000 6.845000 <span class="o">(</span> 6.845000<span class="o">)</span>
user system total real
dalli <span class="nb">set </span>13.251000 0.000000 13.251000 <span class="o">(</span> 13.251000<span class="o">)</span>
dalli get 13.536000 0.000000 13.536000 <span class="o">(</span> 13.536000<span class="o">)</span></code></pre></div>
<p>see more <a href="https://github.com/aurorafeint/jruby-memcached/blob/master/benchmark.rb">here</a>, as you seen, both memcached and jruby-memcached are
2x faster than dalli.</p>
my railsconf 2012 video2012-06-14T00:00:00+00:00http://blog.huangzhimin.com/2012/06/14/my-railsconf-2012-video<p>This is my railsconf 2012 video on youtube.</p>
<iframe width="560" height="315"
src="http://www.youtube.com/embed/900BvuBzINI" frameborder="0"
allowfullscreen></iframe>
newrelic-workling released2012-06-07T00:00:00+00:00http://blog.huangzhimin.com/2012/06/07/newrelic-workling-released<p>We are using <a href="https://github.com/purzelrakete/workling">workling</a> with <a href="http://www.rabbitmq.com/">RabbitMQ</a> as our background
service and monitoring RabbitMQ on <a href="https://scoutapp.com/">scout</a>. Last month, we released
a new background job which generates tons of messages in RabbitMQ, then
messages in RabbitMQ queue kept growing, that means our workling
processes are not many enough to handle that messages. We fixed it by
reverting that job, using cron job to handle instead.</p>
<p>We thought about this accident, and we decided to add <a href="http://newrelic.com/">newrelic</a>
support to measure workling instrument, so that we can have an idea
about how many messages generates for each job and how much does it cost
to consume one message.</p>
<p>We finally released the <a href="https://github.com/aurorafeint/newrelic-workling">newrelic-workling</a> 1.0 gem today, thank
newrelic's help, we are the official support for newrelic workling, feel
free to ping us if you have any question. The following is the
screenshot for the workling instrument on newrelic.</p>
<p><img src="http://farm8.staticflickr.com/7094/7185962145_55cf2790f0.jpg" alt="workling
instrument" /></p>
bullet 4.0.0 released2012-05-09T00:00:00+00:00http://blog.huangzhimin.com/2012/05/09/bullet-4-released<p><a href="https://github.com/flyerhzm/bullet">bullet</a> is designed to help you reduce the number of db queries, such as
adding eager loading to kill n+1 queries and removing unused eager
loadings.</p>
<p>bullet works well in activerecord from 2.1 to 3.2 before, today I
released bullet 4.0.0, it starts to support mongoid (>= 2.4.1) now.</p>
<p>Why does bullet need to support mongoid?
Does mongo also have n+1 queries issue?</p>
<p>The answer is yes, check out the <a href="http://mongoid.org/performance.html">performance metric of mongoid eager
loading</a>, about 40% performance improved. 1 year ago I already
created a gem <a href="https://github.com/flyerhzm/mongoid-eager-loading">mongoid-eager-loading</a> to add eager loading feature
in mongoid, it is deprecated as mongoid has already supported eager
loading natively.</p>
<p>Be aware that bullet for mongoid doesn't support 2 level deep eager
loading and counter cache because they are not supported in mongoid so
far.</p>
<p>What about mongomapper, I'd like to support it in future, but I have no
experience in it, does anybody have interests to implement it? Feel free
to contact me.</p>
<p>Another big improvement in 4.0.0 is much better integration tests. If
you check out the source code, you will see I separate different
integration tests for activerecord 2, activerecord 3 and mongoid, I also
add these integration tests to different Gemfiles, and ask travis to
test all of them for bullet, see the <a href="http://travis-ci.org/#!/flyerhzm/bullet/builds/1283580">build result</a>.</p>
<p>If you have any problems to use bullet gem, feel free to mail me, tweet
me or open an issue on github.</p>
my presentation on railsconf 20122012-05-03T00:00:00+00:00http://blog.huangzhimin.com/2012/05/03/my-presentation-on-railsconf-2012<p>I attended and spoke at railsconf 2012 last week, the following is my
presentation</p>
<div style="width:425px" id="__ss_12677703"> <strong
style="display:block;margin:12px 0 4px"><a
href="http://www.slideshare.net/flyerhzm/semi-automatic-code-review"
title="Semi Automatic Code Review" target="_blank">Semi Automatic Code
Review</a></strong> <iframe
src="http://www.slideshare.net/slideshow/embed_code/12677703"
width="425" height="355" frameborder="0" marginwidth="0"
marginheight="0" scrolling="no"></iframe> <div style="padding:5px 0
12px"> View more <a href="http://www.slideshare.net/"
target="_blank">presentations</a> from <a
href="http://www.slideshare.net/flyerhzm" target="_blank">Richard
Huang</a> </div> </div>
<p>If you have any questions or suggestions, feel free to email me, tweet
me or open issues on github.</p>
redis mget/mset vs get/set2012-04-05T00:00:00+00:00http://blog.huangzhimin.com/2012/04/05/redis-mget-mset-vs-get-set<p>Our application uses redis a lot to perform large numbers of data
reads/writes. But we didn't use it well enough, e.g. we call redis get
and set in loop, just like touching mysql and memcache many times, it
takes a long time to send multiple redis commands, if we can reduce the
commands, it saves on round trip time.</p>
<p>The following script is used to bencharmark different commands count.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="nb">require</span> <span class="s1">'redis'</span>
<span class="nb">require</span> <span class="s1">'benchmark'</span>
<span class="n">redis</span> <span class="o">=</span> <span class="no">Redis</span><span class="o">.</span><span class="n">new</span>
<span class="no">Benchmark</span><span class="o">.</span><span class="n">bm</span><span class="p">(</span><span class="mi">50</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">bm</span><span class="o">|</span>
<span class="n">bm</span><span class="o">.</span><span class="n">report</span> <span class="s2">"redis set"</span> <span class="k">do</span>
<span class="mi">10000</span><span class="o">.</span><span class="n">times</span> <span class="k">do</span> <span class="o">|</span><span class="n">i</span><span class="o">|</span>
<span class="n">redis</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="s2">"key</span><span class="si">#{</span><span class="n">i</span><span class="si">}</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"value</span><span class="si">#{</span><span class="n">i</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">bm</span><span class="o">.</span><span class="n">report</span> <span class="s2">"redis get"</span> <span class="k">do</span>
<span class="mi">10000</span><span class="o">.</span><span class="n">times</span> <span class="k">do</span> <span class="o">|</span><span class="n">i</span><span class="o">|</span>
<span class="n">redis</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"key</span><span class="si">#{</span><span class="n">i</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">bm</span><span class="o">.</span><span class="n">report</span> <span class="s2">"redis mset with 1000"</span> <span class="k">do</span>
<span class="mi">1000</span><span class="o">.</span><span class="n">times</span> <span class="k">do</span> <span class="o">|</span><span class="n">i</span><span class="o">|</span>
<span class="n">keys</span> <span class="o">=</span> <span class="p">(</span><span class="mi">10</span><span class="o">*</span><span class="n">i</span><span class="o">.</span><span class="n">.</span><span class="o">.</span><span class="mi">10</span><span class="o">*</span><span class="n">i</span><span class="o">+</span><span class="mi">10</span><span class="p">)</span><span class="o">.</span><span class="n">map</span> <span class="p">{</span> <span class="o">|</span><span class="n">j</span><span class="o">|</span> <span class="o">[</span><span class="s2">"yek</span><span class="si">#{</span><span class="n">j</span><span class="si">}</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"value</span><span class="si">#{</span><span class="n">j</span><span class="si">}</span><span class="s2">"</span><span class="o">]</span> <span class="p">}</span><span class="o">.</span><span class="n">flatten</span>
<span class="n">redis</span><span class="o">.</span><span class="n">mset</span><span class="p">(</span><span class="o">*</span><span class="n">keys</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">bm</span><span class="o">.</span><span class="n">report</span> <span class="s2">"redis mget with 1000"</span> <span class="k">do</span>
<span class="mi">1000</span><span class="o">.</span><span class="n">times</span> <span class="k">do</span> <span class="o">|</span><span class="n">i</span><span class="o">|</span>
<span class="n">keys</span> <span class="o">=</span> <span class="p">(</span><span class="mi">10</span><span class="o">*</span><span class="n">i</span><span class="o">.</span><span class="n">.</span><span class="o">.</span><span class="mi">10</span><span class="o">*</span><span class="n">i</span><span class="o">+</span><span class="mi">10</span><span class="p">)</span><span class="o">.</span><span class="n">map</span> <span class="p">{</span> <span class="o">|</span><span class="n">j</span><span class="o">|</span> <span class="s2">"yek</span><span class="si">#{</span><span class="n">j</span><span class="si">}</span><span class="s2">"</span> <span class="p">}</span>
<span class="n">redis</span><span class="o">.</span><span class="n">mget</span><span class="p">(</span><span class="o">*</span><span class="n">keys</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">bm</span><span class="o">.</span><span class="n">report</span> <span class="s2">"redis mset with 100"</span> <span class="k">do</span>
<span class="mi">100</span><span class="o">.</span><span class="n">times</span> <span class="k">do</span> <span class="o">|</span><span class="n">i</span><span class="o">|</span>
<span class="n">keys</span> <span class="o">=</span> <span class="p">(</span><span class="mi">100</span><span class="o">*</span><span class="n">i</span><span class="o">.</span><span class="n">.</span><span class="o">.</span><span class="mi">100</span><span class="o">*</span><span class="n">i</span><span class="o">+</span><span class="mi">100</span><span class="p">)</span><span class="o">.</span><span class="n">map</span> <span class="p">{</span> <span class="o">|</span><span class="n">j</span><span class="o">|</span> <span class="o">[</span><span class="s2">"eky</span><span class="si">#{</span><span class="n">j</span><span class="si">}</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"value</span><span class="si">#{</span><span class="n">j</span><span class="si">}</span><span class="s2">"</span><span class="o">]</span> <span class="p">}</span><span class="o">.</span><span class="n">flatten</span>
<span class="n">redis</span><span class="o">.</span><span class="n">mset</span><span class="p">(</span><span class="o">*</span><span class="n">keys</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">bm</span><span class="o">.</span><span class="n">report</span> <span class="s2">"redis mget with 100"</span> <span class="k">do</span>
<span class="mi">100</span><span class="o">.</span><span class="n">times</span> <span class="k">do</span> <span class="o">|</span><span class="n">i</span><span class="o">|</span>
<span class="n">keys</span> <span class="o">=</span> <span class="p">(</span><span class="mi">100</span><span class="o">*</span><span class="n">i</span><span class="o">.</span><span class="n">.</span><span class="o">.</span><span class="mi">100</span><span class="o">*</span><span class="n">i</span><span class="o">+</span><span class="mi">100</span><span class="p">)</span><span class="o">.</span><span class="n">map</span> <span class="p">{</span> <span class="o">|</span><span class="n">j</span><span class="o">|</span> <span class="s2">"eky</span><span class="si">#{</span><span class="n">j</span><span class="si">}</span><span class="s2">"</span> <span class="p">}</span>
<span class="n">redis</span><span class="o">.</span><span class="n">mget</span><span class="p">(</span><span class="o">*</span><span class="n">keys</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">bm</span><span class="o">.</span><span class="n">report</span> <span class="s2">"redis mset with 10"</span> <span class="k">do</span>
<span class="mi">10</span><span class="o">.</span><span class="n">times</span> <span class="k">do</span> <span class="o">|</span><span class="n">i</span><span class="o">|</span>
<span class="n">keys</span> <span class="o">=</span> <span class="p">(</span><span class="mi">1000</span><span class="o">*</span><span class="n">i</span><span class="o">.</span><span class="n">.</span><span class="o">.</span><span class="mi">1000</span><span class="o">*</span><span class="n">i</span><span class="o">+</span><span class="mi">1000</span><span class="p">)</span><span class="o">.</span><span class="n">map</span> <span class="p">{</span> <span class="o">|</span><span class="n">j</span><span class="o">|</span> <span class="o">[</span><span class="s2">"eyk</span><span class="si">#{</span><span class="n">j</span><span class="si">}</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"value</span><span class="si">#{</span><span class="n">j</span><span class="si">}</span><span class="s2">"</span><span class="o">]</span> <span class="p">}</span><span class="o">.</span><span class="n">flatten</span>
<span class="n">redis</span><span class="o">.</span><span class="n">mset</span><span class="p">(</span><span class="o">*</span><span class="n">keys</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">bm</span><span class="o">.</span><span class="n">report</span> <span class="s2">"redis mget with 10"</span> <span class="k">do</span>
<span class="mi">10</span><span class="o">.</span><span class="n">times</span> <span class="k">do</span> <span class="o">|</span><span class="n">i</span><span class="o">|</span>
<span class="n">keys</span> <span class="o">=</span> <span class="p">(</span><span class="mi">1000</span><span class="o">*</span><span class="n">i</span><span class="o">.</span><span class="n">.</span><span class="o">.</span><span class="mi">1000</span><span class="o">*</span><span class="n">i</span><span class="o">+</span><span class="mi">1000</span><span class="p">)</span><span class="o">.</span><span class="n">map</span> <span class="p">{</span> <span class="o">|</span><span class="n">j</span><span class="o">|</span> <span class="s2">"eyk</span><span class="si">#{</span><span class="n">j</span><span class="si">}</span><span class="s2">"</span> <span class="p">}</span>
<span class="n">redis</span><span class="o">.</span><span class="n">mget</span><span class="p">(</span><span class="o">*</span><span class="n">keys</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">bm</span><span class="o">.</span><span class="n">report</span> <span class="s2">"redis mset with 1"</span> <span class="k">do</span>
<span class="n">keys</span> <span class="o">=</span> <span class="p">(</span><span class="mi">0</span><span class="o">.</span><span class="n">.</span><span class="o">.</span><span class="mi">10000</span><span class="p">)</span><span class="o">.</span><span class="n">map</span> <span class="p">{</span> <span class="o">|</span><span class="n">j</span><span class="o">|</span> <span class="o">[</span><span class="s2">"kye</span><span class="si">#{</span><span class="n">j</span><span class="si">}</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"value</span><span class="si">#{</span><span class="n">j</span><span class="si">}</span><span class="s2">"</span><span class="o">]</span> <span class="p">}</span><span class="o">.</span><span class="n">flatten</span>
<span class="n">redis</span><span class="o">.</span><span class="n">mset</span><span class="p">(</span><span class="o">*</span><span class="n">keys</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">bm</span><span class="o">.</span><span class="n">report</span> <span class="s2">"redis mget with 1"</span> <span class="k">do</span>
<span class="n">keys</span> <span class="o">=</span> <span class="p">(</span><span class="mi">0</span><span class="o">.</span><span class="n">.</span><span class="o">.</span><span class="mi">10000</span><span class="p">)</span><span class="o">.</span><span class="n">map</span> <span class="p">{</span> <span class="o">|</span><span class="n">j</span><span class="o">|</span> <span class="s2">"kye</span><span class="si">#{</span><span class="n">j</span><span class="si">}</span><span class="s2">"</span> <span class="p">}</span>
<span class="n">redis</span><span class="o">.</span><span class="n">mget</span><span class="p">(</span><span class="o">*</span><span class="n">keys</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></div>
<p>This is the benchmark result.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="c1"># user system total real</span>
<span class="c1"># redis set 0.280000 0.170000 0.450000 ( 0.809112)</span>
<span class="c1"># redis get 0.290000 0.160000 0.450000 ( 0.806711)</span>
<span class="c1"># redis mset with 1000 0.070000 0.020000 0.090000 ( 0.148474)</span>
<span class="c1"># redis mget with 1000 0.080000 0.020000 0.100000 ( 0.142837)</span>
<span class="c1"># redis mset with 100 0.050000 0.000000 0.050000 ( 0.067859)</span>
<span class="c1"># redis mget with 100 0.050000 0.010000 0.060000 ( 0.063040)</span>
<span class="c1"># redis mset with 10 0.040000 0.000000 0.040000 ( 0.060200)</span>
<span class="c1"># redis mget with 10 0.050000 0.000000 0.050000 ( 0.057818)</span>
<span class="c1"># redis mset with 1 0.040000 0.000000 0.040000 ( 0.062318)</span>
<span class="c1"># redis mget with 1 0.050000 0.000000 0.050000 ( 0.057483)</span></code></pre></div>
<p>It's obvious that less redis commands means fast running time.</p>
master slave replication in rails2012-03-28T00:00:00+00:00http://blog.huangzhimin.com/2012/03/28/master-slave-replication-in-rails<h3>Introduction</h3>
<p>By default activerecord works well with single db, it's applicable for
most of websites with small/medium traffic, but if you website grows
fast and gets much more reads than writes, you should definitly set up
master slave replication for your databse. All inserts/updates are sent
to master db, and reads are sent to slave db, it will reduce read load
on your master db.</p>
<p>Master slave replication allows to set up as many slave dbs as you need,
it's scalable, that means you can easily increase you db read throughput
by adding more slave dbs. It also allows you to move some tasks like
analytics on slave db without affecting your master db.</p>
<h3>Replication in rails</h3>
<p>How do we config master slave replication in rails app? There are a lot
of <a href="https://www.ruby-toolbox.com/categories/Active_Record_Sharding">choices</a>, pick up one and setup according to its document. I
don't want to discuss about these tools here, I will tell you how to use
master slave replication in rails above these tools.</p>
<h3>Problems</h3>
<p>Master slave replication looks well, but it has a big problem in
practice - replication lag. There is a lag between data inserted in
master db and sync to slave db, let's see a case.</p>
<ol>
<li><p>a user create a post on your application.</p></li>
<li><p>the post is inserted to master db.</p></li>
<li><p>your application redirects user to post show page.</p></li>
<li><p>your application read from slave db, but the post is not sync yet.</p></li>
<li><p>a 404 page is shown. :-(</p></li>
<li><p>the post is sync to slave db. (too late)</p></li>
</ol>
<p>Lots of similar issues will raise after you applying master slave
replication, how to solve them?</p>
<h3>Solution</h3>
<p>The solution is send some reads to master db to promise get fresh data.</p>
<p>By default all reads will be sent to master db in one db transaction,
like</p>
<div class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">BEGIN</span>
<span class="k">SELECT</span> <span class="o">*</span> <span class="k">from</span> <span class="n">users</span> <span class="k">where</span> <span class="n">id</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">posts</span><span class="p">(</span><span class="n">title</span><span class="p">,</span> <span class="n">user_id</span><span class="p">)</span> <span class="k">VALUES</span><span class="p">(</span><span class="s1">'test'</span><span class="p">,</span> <span class="mi">1</span><span class="p">);</span>
<span class="k">COMMIT</span></code></pre></div>
<p>In the following cases I will send reads to master db as well</p>
<ul>
<li>queries in background job, like delayed_job, resque, workling, etc.</li>
</ul>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">clas</span> <span class="no">Post</span> <span class="o"><</span> <span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Base</span>
<span class="n">after_create</span> <span class="ss">:notify</span>
<span class="kp">protected</span>
<span class="k">def</span> <span class="nf">notify</span>
<span class="no">Delayed</span><span class="o">::</span><span class="no">Job</span><span class="o">.</span><span class="n">enqueue</span><span class="p">(</span><span class="no">DelayedJob</span><span class="o">::</span><span class="no">NotifyAdmin</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="nb">self</span><span class="o">.</span><span class="n">id</span><span class="p">))</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">class</span> <span class="nc">DelayedJob</span><span class="o">::</span><span class="no">NotifyAdmin</span> <span class="o"><</span> <span class="no">Struct</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="ss">:post_id</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">perform</span>
<span class="n">post</span> <span class="o">=</span> <span class="no">Post</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="n">post_id</span><span class="p">)</span>
<span class="o">.</span><span class="n">.</span><span class="o">.</span><span class="n">.</span><span class="o">.</span><span class="n">.</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></div>
<p>It's probably the post does not exist when reading it from slave db in
background job.</p>
<ul>
<li>queries in the request which follows a redirect reponse</li>
</ul>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">class</span> <span class="nc">PostsController</span> <span class="o"><</span> <span class="no">ApplicationController</span>
<span class="k">def</span> <span class="nf">show</span>
<span class="vi">@post</span> <span class="o">=</span> <span class="no">Post</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="n">params</span><span class="o">[</span><span class="ss">:id</span><span class="o">]</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">create</span>
<span class="vi">@post</span> <span class="o">=</span> <span class="no">Post</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="n">params</span><span class="o">[</span><span class="ss">:post</span><span class="o">]</span><span class="p">)</span>
<span class="k">if</span> <span class="vi">@post</span><span class="o">.</span><span class="n">save</span>
<span class="n">redirect_to</span> <span class="n">post_path</span><span class="p">(</span><span class="vi">@post</span><span class="p">)</span>
<span class="k">else</span>
<span class="n">render</span> <span class="ss">:new</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></div>
<p>This case is too common, creating/updating then redirecting, if the
resource is not sync to slave db before next request, user will get a
404 page or get some fake data.</p>
<p>We know when we should explictly send reads to master db, but how can we
do that. It's</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Base</span><span class="o">.</span><span class="n">with_master</span> <span class="p">{</span>
<span class="no">User</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="n">post</span><span class="o">.</span><span class="n">user_id</span><span class="p">)</span>
<span class="p">}</span></code></pre></div>
<p>Almost all of replication gem provide with_master method, any queries in
the block will be sent to master db. I added a monkey patch to background
job, wrapping it with with_master.</p>
<p>I added add a monkey patch to action controller as well, adding a parameter
if the response is a redirect, then add a around_filter to controller to
check if the reads in such request should be sent to master or slave db.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">class</span> <span class="nc">ApplicationController</span> <span class="o"><</span> <span class="no">ActionController</span><span class="o">::</span><span class="no">Base</span>
<span class="n">around_filter</span> <span class="ss">:manage_slaving</span>
<span class="k">def</span> <span class="nf">manage_slaving</span>
<span class="k">if</span> <span class="n">force_master?</span>
<span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Base</span><span class="o">.</span><span class="n">with_master</span> <span class="p">{</span> <span class="k">yield</span> <span class="p">}</span>
<span class="k">else</span>
<span class="k">yield</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></div>
<p>force_master? is a convenient way to manage your master/slave db on
controller levels, you can also enable/disalbe master/slave for some
specfied requests.</p>
<p>Finally test your application and add ActiveRecord::Base.with_mater {}
if necessary.</p>
bullet 2.3.0 released2012-03-25T00:00:00+00:00http://blog.huangzhimin.com/2012/03/25/bullet-2.3.0-released<p><a href="https://github.com/flyerhzm/bullet">bullet</a> is a gem to help you increase your application's performance
by reducing the number of sql requests it makes. Today I released bullet
2.3.0 to better support rails 3.1 and 3.2 and performance improved. It's
a long time I didn't do any changes to bullet, let me tell you the story
I work for bullet 2.3.0.</p>
<p>At the beginning of this month, bullet got its 1000th watcher on github,
I realized it's time to improve it e.g. speed up and compatible with
edge rails.</p>
<p>The first thing I did is to refactor tests. Before I created several
rspec tests, but they are more like integration tests instead of unit
tests, so I move them to spec/integration/ directory. Then I added a
bunch of test units to cover all codes, which can promise the
correctness of further code refactors. I also use <a href="https://github.com/guard/guard">guard</a> instead of
watchr to do auto tests, why I preferred guard? It's much easier and has
more extensions, like guard-rspec.</p>
<p>Then I moved AR models, which are used for integration tests, from
integration tests to spec/models, and I also moved db connection, db
schema and db seed to spec/support/, moved test helpers to spec/support/
as well. Now my tests looks much cleaner and run much faster (only
connect db once).</p>
<p>After refactoring tests, I tried to improve the bullet performance, I
already created a <a href="https://github.com/flyerhzm/bullet/blob/master/perf/benchmark.rb">benchmark script</a> before, bullet 2.2.1 with rails
3.0.12 spent 30s to complete</p>
<div class="highlight"><pre><code class="language-text" data-lang="text">bullet 2.2.1 with rails 3.0.12
user system total real
Querying & Iterating 1000 Posts with 10000 Comments and 100 Users 29.970000 0.270000 30.240000 ( 30.452083)</code></pre></div>
<p>Then I used perftools.rb to measure cpu time for methods, the result is
garbage_collector, String#=~ and Kernel#caller</p>
<ol>
<li>garbage_collector, it depends on how many objects allocated</li>
<li>String#=~, bullet use regexp to check if caller contains load_target</li>
<li>Kernel#caller, bullet uses caller to tell what codes caused n+1 query</li>
</ol>
<p>I found the easiest is to mitigate String#=~, as bullet only check
regexp with constant string load_target, so I simply used
.include?("load_target") instead.</p>
<div class="highlight"><pre><code class="language-text" data-lang="text">bullet 2.3.0 with rails 3.0.12
user system total real
Querying & Iterating 1000 Posts with 10000 Comments and 100 Users 26.120000 0.430000 26.550000 ( 27.179304)</code></pre></div>
<p>another change is to store object's ar_key instead of object itself.</p>
<div class="highlight"><pre><code class="language-text" data-lang="text">{<#Post id:1, title:"post1", body:"post body", created_at:..., updated_at:...> => [:comments]}</code></pre></div>
<p>to</p>
<div class="highlight"><pre><code class="language-text" data-lang="text">{"Post:1" => [:comments]}</code></pre></div>
<p>it speeds up hash comparison time and save the hash size.</p>
<p>I also hacked ActiveRecord::Associations::SingularAssociation#reader
instead of ActiveRecord::Associations::Association#load_target for rails
3.1 and 3.2, it fixes activerecord 3.1 and 3.2 compatibility, there is
no need to call caller in Association#load_target, it runs much faster
in rails 3.1 and 3.2, the following is the benchmark result</p>
<div class="highlight"><pre><code class="language-text" data-lang="text">bullet 2.3.0 with rails 3.2.2
user system total real
Querying & Iterating 1000 Posts with 10000 Comments and 100 Users 16.460000 0.190000 16.650000 ( 16.968246)
bullet 2.3.0 with rails 3.1.4
user system total real
Querying & Iterating 1000 Posts with 10000 Comments and 100 Users 14.600000 0.130000 14.730000 ( 14.937590)</code></pre></div>
<p>Enjoy the new <a href="https://github.com/flyerhzm/bullet">bullet</a> gem!</p>
multiple_mailers - send emails by different smtp accounts2012-03-21T00:00:00+00:00http://blog.huangzhimin.com/2012/03/21/multiple_mailers-send-emails-by-multiple-smtp-accounts<p>I use gmail to send email notifications on my website, it's really easy
to build based on actionmailer</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="no">ActionMailer</span><span class="o">::</span><span class="no">Base</span><span class="o">.</span><span class="n">smtp_settings</span> <span class="o">=</span> <span class="p">{</span>
<span class="ss">:address</span> <span class="o">=></span> <span class="s1">'smtp.gmail.com'</span><span class="p">,</span>
<span class="ss">:port</span> <span class="o">=></span> <span class="mi">587</span><span class="p">,</span>
<span class="ss">:domain</span> <span class="o">=></span> <span class="s1">'railsbp.com'</span><span class="p">,</span>
<span class="ss">:authentication</span> <span class="o">=></span> <span class="ss">:plain</span><span class="p">,</span>
<span class="ss">:user_name</span> <span class="o">=></span> <span class="s1">'notification@railsbp.com'</span><span class="p">,</span>
<span class="ss">:password</span> <span class="o">=></span> <span class="s1">'password'</span>
<span class="p">}</span></code></pre></div>
<p>But I found it does not allow to setup 2 different smtp accounts, e.g. I
want to send notification email with notification@railsbp.com and send
exception notifier email with exception.notifier@railsbp.com, after
googling, I hacked my mailer classes with</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">class</span> <span class="nc">NotificationMailer</span> <span class="o"><</span> <span class="no">ActionMailer</span><span class="o">::</span><span class="no">Base</span>
<span class="k">if</span> <span class="no">Rails</span><span class="o">.</span><span class="n">env</span><span class="o">.</span><span class="n">production?</span>
<span class="k">class</span> <span class="o"><<</span><span class="nb">self</span>
<span class="k">def</span> <span class="nf">smtp_settings</span>
<span class="n">options</span> <span class="o">=</span> <span class="no">YAML</span><span class="o">.</span><span class="n">load_file</span><span class="p">(</span><span class="s2">"</span><span class="si">#{</span><span class="no">Rails</span><span class="o">.</span><span class="n">root</span><span class="si">}</span><span class="s2">/config/mailers.yml"</span><span class="p">)</span><span class="o">[</span><span class="no">Rails</span><span class="o">.</span><span class="n">env</span><span class="o">][</span><span class="s1">'exception_notifier'</span><span class="o">]</span>
<span class="vc">@@smtp_settings</span> <span class="o">=</span> <span class="p">{</span>
<span class="ss">:address</span> <span class="o">=></span> <span class="n">options</span><span class="o">[</span><span class="s2">"address"</span><span class="o">]</span><span class="p">,</span>
<span class="ss">:port</span> <span class="o">=></span> <span class="n">options</span><span class="o">[</span><span class="s2">"port"</span><span class="o">]</span><span class="p">,</span>
<span class="ss">:domain</span> <span class="o">=></span> <span class="n">options</span><span class="o">[</span><span class="s2">"domain"</span><span class="o">]</span><span class="p">,</span>
<span class="ss">:authentication</span> <span class="o">=></span> <span class="n">options</span><span class="o">[</span><span class="s2">"authentication"</span><span class="o">]</span><span class="p">,</span>
<span class="ss">:user_name</span> <span class="o">=></span> <span class="n">options</span><span class="o">[</span><span class="s2">"user_name"</span><span class="o">]</span><span class="p">,</span>
<span class="ss">:password</span> <span class="o">=></span> <span class="n">options</span><span class="o">[</span><span class="s2">"password"</span><span class="o">]</span>
<span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></div>
<p>then add a new config file config/mailers.yml</p>
<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="l-Scalar-Plain">production</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">common</span><span class="p-Indicator">:</span> <span class="nl">&common</span>
<span class="l-Scalar-Plain">address</span><span class="p-Indicator">:</span> <span class="s">'smtp.gmail.com'</span>
<span class="l-Scalar-Plain">port</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">587</span>
<span class="l-Scalar-Plain">domain</span><span class="p-Indicator">:</span> <span class="s">'rails-bestpractices.com'</span>
<span class="l-Scalar-Plain">authentication</span><span class="p-Indicator">:</span> <span class="s">'plain'</span>
<span class="l-Scalar-Plain">notification</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain"><<</span><span class="p-Indicator">:</span> <span class="nv">*common</span>
<span class="l-Scalar-Plain">user_name</span><span class="p-Indicator">:</span> <span class="s">'notification@rails-bestpractices.com'</span>
<span class="l-Scalar-Plain">password</span><span class="p-Indicator">:</span> <span class="s">'password'</span>
<span class="l-Scalar-Plain">exception.notifier</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain"><<</span><span class="p-Indicator">:</span> <span class="nv">*common</span>
<span class="l-Scalar-Plain">user_name</span><span class="p-Indicator">:</span> <span class="s">'exception.notifier@rails-bestpractices.com'</span>
<span class="l-Scalar-Plain">password</span><span class="p-Indicator">:</span> <span class="s">'password'</span></code></pre></div>
<p>that allows me to setup one smtp account per actionmailer class, keep in
mind that you should only hack smtp_settings for what environment you
really want to send emails (here is production), if you don't check
Rails.env, it will send email even in development and test environments.</p>
<p>Now it works fine, I can send emails by as many smtp accounts as I like, but
it looks ugly, I don't like hacking codes all over my mailer classes. So I
abstract it to a new gem <a href="https://github.com/flyerhzm/multiple_mailers">multiple_mailers</a>, like the hack above, you
should define config file config/mailers.yml and for each mail class,
what you only need is to declare its mailer account name</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">class</span> <span class="nc">NotificationMailer</span> <span class="o"><</span> <span class="no">ActionMailer</span><span class="o">::</span><span class="no">Base</span>
<span class="n">mailer_account</span> <span class="s2">"notification"</span>
<span class="k">end</span>
<span class="k">class</span> <span class="nc">ExceptionNotifier</span>
<span class="k">class</span> <span class="nc">Notifier</span> <span class="o"><</span> <span class="no">ActionMailer</span><span class="o">::</span><span class="no">Base</span>
<span class="n">mailer_account</span> <span class="s2">"exception.notifier"</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></div>
passenger with http_gzip_static_module2012-02-27T00:00:00+00:00http://blog.huangzhimin.com/2012/02/27/passenger-with-http-gzip-static-module<p>Rails 3.1 has been released for a long time, asset pipeline becomes more
and more popular, I also upgraded my rails website.</p>
<p>I used nginx + passenger for my rails projects, but nginx only supports
dynamic gzip support (compress in runtime), there is a
http_gzip_static_module for nginx, which can make full use of rails
asset pipeline.</p>
<p>I don't like the way to customize my Nginx installation during passenger
installation, I found there is a <a href="https://github.com/FooBarWidget/passenger/pull/35">pull request</a> to add
http_gzip_static_module, so I changed to source code of passenger gem,
then installed nginx as default. :-)</p>
rake arguments2011-12-13T00:00:00+00:00http://blog.huangzhimin.com/2011/12/13/rake-arguments<p>Long ago I began to write some rake tasks, it's simple but doesn't have
an instruction about how to add arguments to a rake task. What I did
before is to use ruby environment variables.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">task</span> <span class="ss">:try_argument</span> <span class="k">do</span>
<span class="no">ENV</span><span class="o">[</span><span class="s1">'GLOBAL_ARGUMENT1'</span><span class="o">]</span> <span class="ow">or</span> <span class="no">ENV</span><span class="o">[</span><span class="s1">'GLOBAL_ARGUMENT2'</span><span class="o">]</span>
<span class="k">end</span>
<span class="no">GLOBAL_ARGUMENT1</span><span class="o">=</span><span class="n">xxx</span> <span class="no">GLOBAL_ARGUMENT2</span><span class="o">=</span><span class="n">yyy</span> <span class="n">rake</span> <span class="n">try_argument</span></code></pre></div>
<p>As you seen, I have to set the global environment variable to pass the
arguement to a rake task.</p>
<p>But there is another way to pass the arguments to rake task via []</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">task</span> <span class="ss">:try_argument</span><span class="p">,</span> <span class="o">[</span><span class="ss">:key1</span><span class="p">,</span> <span class="ss">:key2</span><span class="o">]</span> <span class="k">do</span> <span class="o">|</span><span class="n">t</span><span class="p">,</span> <span class="n">args</span><span class="o">|</span>
<span class="n">args</span><span class="o">.</span><span class="n">with_defaults</span><span class="p">(</span><span class="ss">:key1</span> <span class="o">=></span> <span class="n">value1</span><span class="p">,</span> <span class="ss">:key2</span> <span class="o">=></span> <span class="n">value2</span><span class="p">)</span>
<span class="n">args</span><span class="o">[</span><span class="ss">:key1</span><span class="o">]</span> <span class="ow">or</span> <span class="n">args</span><span class="o">[</span><span class="ss">:key2</span><span class="o">]</span>
<span class="k">end</span>
<span class="n">rake</span> <span class="n">try_argument</span><span class="o">[</span><span class="n">xxx</span><span class="p">,</span> <span class="n">yyy</span><span class="o">]</span></code></pre></div>
<p>and if there is dependent task, you should define it like</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">task</span> <span class="ss">:try_argument</span><span class="p">,</span> <span class="o">[</span><span class="ss">:key1</span><span class="p">,</span> <span class="ss">:key2</span><span class="o">]</span> <span class="o">=></span> <span class="ss">:environment</span> <span class="k">do</span> <span class="o">|</span><span class="n">t</span><span class="p">,</span> <span class="n">args</span><span class="o">|</span>
<span class="n">args</span><span class="o">.</span><span class="n">with_defaults</span><span class="p">(</span><span class="ss">:key1</span> <span class="o">=></span> <span class="n">value1</span><span class="p">,</span> <span class="ss">:key2</span> <span class="o">=></span> <span class="n">value2</span><span class="p">)</span>
<span class="n">args</span><span class="o">[</span><span class="ss">:key1</span><span class="o">]</span> <span class="ow">or</span> <span class="n">args</span><span class="o">[</span><span class="ss">:key2</span><span class="o">]</span>
<span class="k">end</span>
<span class="n">rake</span> <span class="n">try_argument</span><span class="o">[</span><span class="n">xxx</span><span class="p">,</span> <span class="n">yyy</span><span class="o">]</span></code></pre></div>
<p>It looks like the difference between hash arguments and normal arguments.</p>
<p>Both of them have disadvantage:</p>
<p>ENV arguments also changes the system env variables
normal arguments do not make sense when calling, difficult to remember
the meanings of arguments.</p>
<p>Both work fine, it depends on you to use which one.</p>
passenger with redis2011-12-12T00:00:00+00:00http://blog.huangzhimin.com/2011/12/12/passenger-with-redis<p>Today I encountered an issue that passenger forks too many workers
than what we set (6) on qa servers. I used strace, the passenger worker
is blocked by failed to writing to a socket, like</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="k">select</span><span class="o">(</span>15, <span class="o">[]</span>, <span class="o">[</span>13<span class="o">]</span>, <span class="o">[]</span>, <span class="o">[</span>58, 915000<span class="o">])</span></code></pre></div>
<p>fd 13 is a socket.</p>
<p>I also tried netstat and found the status for some redis socket
connections are CLOSE_WAIT.</p>
<p>So I judged this is the problem the ruby redis clients are not closed
correctly. This reminds me that passenger fork() nature, I checked our
source codes, unfortunately, we didn't do anything special for passenger
fork.</p>
<p>This is the <a href="https://github.com/ezmobius/redis-rb/wiki/redis-rb-on-Phusion-Passenger">link</a> tells you how to close the redis connection after
passenger forks a worker. After deploy the new codes to qa servers,
passenger never forks more workers than we expected. But the workers
still hang up according strace result, that means some workers keep
inactive status, they won't be able to handle any requests. Wooops...</p>
<p>I looked through the redis-rb source codes, we used redis 2.0.5, it
didn't handle TIMEOUT error and always retry writing to redis.
Fortunately, the latest redis version is 2.2.2 and it already fixed this
issue, retry 3 times, if still failed, the release the connection.</p>
<p>Now it works fine, no unexpected additional passenger workers and no
unexpected inactive workers.</p>
avoid committing git conflicts2011-11-14T00:00:00+00:00http://blog.huangzhimin.com/2011/11/14/avoid-committing-git-conflicts<p>I made a mistake when merging branch last week, I forgot to remove a
conflict syntax "<<<<<< HEAD" and push it to remote repository. It
breaks other one's development. So stupid to make such mistake.</p>
<p>To avoid making such mistake anymore, I write a git hook
.git/hooks/pre-commit to check conflict syntax "<<<<<<" and ">>>>>>"</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="c1">#!/usr/bin/env ruby</span>
<span class="sb">`git diff-index --name-status HEAD`</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="p">)</span><span class="o">.</span><span class="n">each</span> <span class="k">do</span>
<span class="o">|</span><span class="n">status_with_filename</span><span class="o">|</span>
<span class="n">status</span><span class="p">,</span> <span class="n">filename</span> <span class="o">=</span> <span class="n">status_with_filename</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="sr">/\s+/</span><span class="p">)</span>
<span class="k">next</span> <span class="k">if</span> <span class="n">status</span> <span class="o">==</span> <span class="s1">'D'</span>
<span class="no">File</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">file</span><span class="o">|</span>
<span class="k">while</span> <span class="n">line</span> <span class="o">=</span> <span class="n">file</span><span class="o">.</span><span class="n">gets</span>
<span class="k">if</span> <span class="n">line</span><span class="o">.</span><span class="n">include?</span><span class="p">(</span><span class="s2">"<<<<<<<"</span><span class="p">)</span> <span class="o">||</span> <span class="n">line</span><span class="o">.</span><span class="n">include?</span><span class="p">(</span><span class="s2">">>>>>>>"</span><span class="p">)</span>
<span class="nb">puts</span> <span class="s2">"ERROR: </span><span class="si">#{</span><span class="n">filename</span><span class="si">}</span><span class="s2"> is conflict"</span>
<span class="nb">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></div>
<p>It will prevent you from committing conflicts.</p>
after_commit2011-11-06T00:00:00+00:00http://blog.huangzhimin.com/2011/11/06/after_commit<p>We are using RabbitMQ as our message queue system, ruby client is
workling. This week we encountered a strange issue, we create a
notification, and define an after_create callback to ask workling to
find that notification and then push the notification to twitter or
facebook, it works fine except that sometimes it will raise an error
said "can't find the notification with the specified ID"</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">class</span> <span class="nc">Notification</span> <span class="o"><</span> <span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Base</span>
<span class="n">after_create</span> <span class="ss">:asyns_send_notification</span>
<span class="o">.</span><span class="n">.</span><span class="o">.</span><span class="n">.</span><span class="o">.</span><span class="n">.</span>
<span class="k">def</span> <span class="nf">async_send_notification</span>
<span class="no">NotificationWorker</span><span class="o">.</span><span class="n">async_send_notification</span><span class="p">({</span><span class="ss">:notification_id</span> <span class="o">=></span> <span class="nb">id</span><span class="p">})</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">class</span> <span class="nc">NotificationWorker</span> <span class="o"><</span> <span class="no">Workling</span><span class="o">::</span><span class="no">Base</span>
<span class="k">def</span> <span class="nf">send_notification</span><span class="p">(</span><span class="n">params</span><span class="p">)</span>
<span class="n">notification</span> <span class="o">=</span> <span class="no">Notification</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="n">params</span><span class="o">[</span><span class="ss">:notification_id</span><span class="o">]</span><span class="p">)</span>
<span class="o">.</span><span class="n">.</span><span class="o">.</span><span class="n">.</span><span class="o">.</span><span class="n">.</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></div>
<p>It's wierd the notification_id is passed to NotificationWorker, that
means the notification is already created, the notification is supposed
to be existed.</p>
<p>After talking with MySQL DBA, we find the problem is the find sql is
executed before insert transaction is committed.</p>
<p>Let me describe it step by step.</p>
<ol>
<li>Notification sends "Transaction Begin" command</li>
<li>Notification sends "INSERT" command</li>
<li>Notification gets "next sequence value" as new object id</li>
<li>Notification sends "new object id" to NotificationWorker</li>
<li>NotificationWorker sends "SELECT" command to find notification object</li>
<li>Notification sends "Transaction Commit" command</li>
</ol>
<p>As you seen, at step 5, the new notification is not existed in the mysql
database yet, so the error "Not found" will be raised.</p>
<p>To solve this issue, we can use after_commit callback.</p>
<p>In rails 2.x, we should install after_commit gem, in rails 3.x,
after_commit callback is supported by default.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">class</span> <span class="nc">Notification</span> <span class="o"><</span> <span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Base</span>
<span class="n">after_commit_on_create</span> <span class="ss">:asyns_send_notification</span>
<span class="o">.</span><span class="n">.</span><span class="o">.</span><span class="n">.</span><span class="o">.</span><span class="n">.</span>
<span class="k">def</span> <span class="nf">async_send_notification</span>
<span class="no">NotificationWorker</span><span class="o">.</span><span class="n">async_send_notification</span><span class="p">({</span><span class="ss">:notification_id</span> <span class="o">=></span> <span class="nb">id</span><span class="p">})</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></div>
<p>So Notification asks NotificationWorker to run only after the
transaction is committed.</p>
reset_counters in rails2011-10-31T00:00:00+00:00http://blog.huangzhimin.com/2011/10/31/reset_counter-in-rails<p>I thought reset_counters method is to reset a counter_cache column to be
0, but it is not. After trying several times, I finally realize that
reset_counters is to update the value of counter_cache column to the
exact count of associations. The usecase of reset_counters is when you
add the counter_cache in migration and update the counter_cache value,
like</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">up</span>
<span class="n">add_column</span> <span class="ss">:posts</span><span class="p">,</span> <span class="ss">:comments_count</span>
<span class="no">Post</span><span class="o">.</span><span class="n">all</span><span class="o">.</span><span class="n">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">post</span><span class="o">|</span>
<span class="no">Post</span><span class="o">.</span><span class="n">reset_counters</span><span class="p">(</span><span class="n">post</span><span class="o">.</span><span class="n">id</span><span class="p">,</span> <span class="ss">:comments</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span></code></pre></div>
<p>it will add comments_count column to posts table, and calculate the
comments count for each post, and set it to posts' comments_count
column.</p>
<p>I didn't find a method to reset the counter_cache column to be 0, why?
Because counter_cache is used to cache the association count, it will be
incremented and decremeneted automatically, you should never reset it 0.
If you find you need to reset counter_cache to 0, that means it's a
wrong usage of counter_cache.</p>
use rspec filter to speed up tests2011-10-21T00:00:00+00:00http://blog.huangzhimin.com/2011/10/21/rspec-filter<p>Rspec 2 introduce a very efficient way to test only one test or one test
suit, it's filter_run.</p>
<p>You should first add filter_run in rspec/spec_helper.rb</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">config</span><span class="o">.</span><span class="n">filter_run</span> <span class="ss">:focus</span> <span class="o">=></span> <span class="kp">true</span></code></pre></div>
<p>Then you can tell rspec to test only one test you are focused by</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">it</span> <span class="s2">"should focus now"</span><span class="p">,</span> <span class="ss">:focus</span> <span class="o">=></span> <span class="kp">true</span> <span class="k">do</span>
<span class="o">.</span><span class="n">.</span><span class="o">.</span>
<span class="k">end</span></code></pre></div>
<p>rspec will only test this spec, :focus => true can be applied on
describe/context as well.</p>
<p>One problem is that if there is no :focus => true on your tests, rspec
will do nothing, but most of time we are expecting to test all specs if
no focus is true, so you should add a line to spec_helper as well.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">config</span><span class="o">.</span><span class="n">run_all_when_everything_filtered</span> <span class="o">=</span> <span class="kp">true</span></code></pre></div>
<p>As the name implies, rspec will test all specs if no focus filter.</p>
<p>Another you may interest that you can also define filter_run_excluding</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">config</span><span class="o">.</span><span class="n">filter_run_excluding</span> <span class="ss">:slow</span> <span class="o">=></span> <span class="kp">true</span></code></pre></div>
<p>rspec will run all specs except what specs are marked as slow.</p>
rubykaigi presentation2011-07-17T00:00:00+00:00http://blog.huangzhimin.com/2011/07/17/rubykaigi-presentation<p>My presentation in RubyKaigin 2011 today.</p>
<div style="width:425px" id="__ss_8617337"><strong style="display:block;margin:12px 0 4px"><a href="http://www.slideshare.net/flyerhzm/rails-best-practicesrubykaigi" title="Rails best practices_rubykaigi">Rails best practices_rubykaigi</a></strong><object id="__sse8617337" width="425" height="355"><param name="movie" value="http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=railsbestpracticesrubykaigi-110717075124-phpapp02&stripped_title=rails-best-practicesrubykaigi&userName=flyerhzm" /><param name="allowFullScreen" value="true"/><param name="allowScriptAccess" value="always"/><embed name="__sse8617337" src="http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=railsbestpracticesrubykaigi-110717075124-phpapp02&stripped_title=rails-best-practicesrubykaigi&userName=flyerhzm" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="425" height="355"></embed></object><div style="padding:5px 0 12px">View more <a href="http://www.slideshare.net/">presentations</a> from <a href="http://www.slideshare.net/flyerhzm">Richard Huang</a>.</div></div>
<p>and the video is here: <a href="http://www.ustream.tv/recorded/16051491">http://www.ustream.tv/recorded/16051491</a></p>
beijing ruby线下活动2011-03-27T00:00:00+00:00http://blog.huangzhimin.com/2011/03/27/beijing-ruby-event<p>周日在北京的ruby线下活动的ppt</p>
<div style="width:425px" id="__ss_7409659"> <strong style="display:block;margin:12px 0 4px"><a href="http://www.slideshare.net/flyerhzm/rails-best-practices-7409659" title="使用Rails best practices做代码审查">使用Rails best practices做代码审查</a></strong> <object id="__sse7409659" width="425" height="355"> <param name="movie" value="http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=railsbestpractices-110327215428-phpapp02&stripped_title=rails-best-practices-7409659&userName=flyerhzm" /> <param name="allowFullScreen" value="true"/> <param name="allowScriptAccess" value="always"/> <embed name="__sse7409659" src="http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=railsbestpractices-110327215428-phpapp02&stripped_title=rails-best-practices-7409659&userName=flyerhzm" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="425" height="355"></embed> </object> <div style="padding:5px 0 12px"> View more <a href="http://www.slideshare.net/">presentations</a> from <a href="http://www.slideshare.net/flyerhzm">Richard Huang</a> </div> </div>
Upgrade Mongoid - Multiple databases2011-03-22T00:00:00+00:00http://blog.huangzhimin.com/2011/03/22/upgrade-mongoid-multiple-database<p>My recent post <a href="http://www.huangzhimin.com/2011/01/14/use-different-mongodb-instances-in-mongoid/">Use different mongodb instances in mongoid</a> tells you how to use multiple databases, it looks good, but mongoid began to support multiple databases itself from mongoid.2.0.0.rc.1, much better than my hack.</p>
<p>It's really easy to use, first, you should define multiple databases in mongoid.yml like</p>
<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="l-Scalar-Plain">development</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain"><<</span><span class="p-Indicator">:</span> <span class="nv">*defaults</span>
<span class="l-Scalar-Plain">host</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">localhost</span>
<span class="l-Scalar-Plain">database</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">main_mongo_instance</span>
<span class="l-Scalar-Plain">databases</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">other_mongo_instance_name</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">database</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">other_mongo_instance</span>
<span class="l-Scalar-Plain">host</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">localhost</span></code></pre></div>
<p>As you seen, besides the common database param, I have defined a new param databases, you should define the mongo instance name with database and host name, and of course, you can define as many mongo instances as you need.</p>
<p>Then, you can choose which mongo instance to use in mongoid model.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">class</span> <span class="nc">User</span>
<span class="kp">include</span> <span class="no">Mongoid</span><span class="o">::</span><span class="no">Document</span>
<span class="n">set_database</span> <span class="ss">:other_mongo_instance_name</span>
<span class="k">end</span></code></pre></div>
<p>set_database method tells mongoid that the model will use another mongo instance instead of the main mongo instance, here we use the name other_mongo_instance_name which should exactly be the same with the name defined in mongoid.yml. If you don't say anything, it will use the main_mongo_instance.</p>
<p>So all the users data will be stored to other_mongo_instance_name, and the other data will be stored to main_mongo_instance. Great!</p>
Upgrade Mongoid - update_attribute2011-03-21T00:00:00+00:00http://blog.huangzhimin.com/2011/03/21/upgrade-mongoid-update_attribute<p>Before mongoid 2.0.0.rc.6, there is no update_attribute method for Mongoid::Document, it makes me unhappy. As in ActiveRecord world, I always use update_attribute to change one attribute and use update_attributes to change two or more attributes.</p>
<p>It's a good news that mongoid introduces the update_attribute method from 2.0.0.rc.6, now I can follow my practice in mongoid.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">post</span><span class="o">.</span><span class="n">update_attribute</span><span class="p">(</span><span class="ss">:title</span> <span class="o">=></span> <span class="s2">"New Post"</span><span class="p">)</span>
<span class="n">post</span><span class="o">.</span><span class="n">update_attributes</span><span class="p">(</span><span class="ss">:title</span> <span class="o">=></span> <span class="s2">"New Post"</span><span class="p">,</span> <span class="ss">:body</span> <span class="o">=></span> <span class="s2">"New Body"</span><span class="p">)</span></code></pre></div>
Upgrade Mongoid - Many to many association2011-03-08T00:00:00+00:00http://blog.huangzhimin.com/2011/03/08/upgrade-mongoid-many-to-many-associations<p>Before mongoid 2.0.0.rc1, there is no default support for many to many association. So we use join document (aka join table in relational database) to implement the many to many association.</p>
<p>For example, we have two documents users and accounts, one user has many accounts and one account contains many users, to establish the many to many relationship between users and accounts, we create a new document named user_accounts, the document looks like</p>
<div class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="p">{</span><span class="s1">'_id'</span><span class="o">:</span> <span class="s1">'4d76d3a70bdb822d08000001'</span><span class="p">,</span> <span class="s1">'user_id'</span><span class="o">:</span> <span class="nx">BSON</span><span class="o">::</span><span class="nx">ObjectId</span><span class="p">(</span><span class="s1">'4d76d3b80bdb822d080015b3'</span><span class="p">),</span> <span class="s1">'account_id'</span><span class="o">:</span> <span class="nx">BSON</span><span class="o">::</span><span class="nx">ObjectId</span><span class="p">(</span><span class="s1">'4d76d3b90bdb822d080015b7'</span><span class="p">)}</span></code></pre></div>
<p>and the models are defined as follows</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">class</span> <span class="nc">User</span>
<span class="kp">include</span> <span class="no">Mongoid</span><span class="o">::</span><span class="no">Document</span>
<span class="n">references_many</span> <span class="ss">:user_accounts</span>
<span class="k">end</span>
<span class="k">class</span> <span class="nc">Account</span>
<span class="kp">include</span> <span class="no">Mongoid</span><span class="o">::</span><span class="no">Document</span>
<span class="n">references_many</span> <span class="ss">:user_accounts</span>
<span class="k">end</span>
<span class="k">class</span> <span class="nc">UserAccount</span>
<span class="kp">include</span> <span class="no">Mongoid</span><span class="o">::</span><span class="no">Document</span>
<span class="n">referenced_in</span> <span class="ss">:user</span>
<span class="n">referenced_in</span> <span class="ss">:account</span>
<span class="k">end</span></code></pre></div>
<p>Are you familiar with it, it's what activerecord did for many to many association.</p>
<p>I'm glad that mongoid began to support many to many association after mongoid 2.0.0.rc1, the new syntax is "referenes_and_referenced_in_many".</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">class</span> <span class="nc">User</span>
<span class="kp">include</span> <span class="no">Mongoid</span><span class="o">::</span><span class="no">Document</span>
<span class="n">references_and_referenced_in_many</span> <span class="ss">:accounts</span>
<span class="k">end</span>
<span class="k">class</span> <span class="nc">Account</span>
<span class="kp">include</span> <span class="no">Mongoid</span><span class="o">::</span><span class="no">Document</span>
<span class="n">references_and_referenced_in_many</span> <span class="ss">:users</span>
<span class="k">end</span></code></pre></div>
<p>We don't need the join document any more. The implementation of mongoid is different with activerecord, it uses array attribute to store the relationship at both sides. Like</p>
<p>These are user documents</p>
<div class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="p">{</span><span class="s1">'_id'</span><span class="o">:</span> <span class="s1">'4d76d3a90bdb822d08000009'</span><span class="p">,</span> <span class="nx">account_ids</span><span class="o">:</span> <span class="p">[</span><span class="nx">BSON</span><span class="o">::</span><span class="nx">ObjectId</span><span class="p">(</span><span class="s1">'4d76d3aa0bdb822d0800001b'</span><span class="p">),</span> <span class="nx">BSON</span><span class="o">::</span><span class="nx">ObjectId</span><span class="p">(</span><span class="s1">'4d76d3aa0bdb822d0800001d'</span><span class="p">),</span> <span class="nx">BSON</span><span class="o">::</span><span class="nx">ObjectId</span><span class="p">(</span><span class="s1">'4d76d3aa0bdb822d08000017'</span><span class="p">)]}</span>
<span class="p">{</span><span class="s1">'_id'</span><span class="o">:</span> <span class="s1">'4d76d3a80bdb822d08000005'</span><span class="p">,</span> <span class="nx">account_ids</span><span class="o">:</span> <span class="p">[</span><span class="nx">BSON</span><span class="o">::</span><span class="nx">ObjectId</span><span class="p">(</span><span class="s1">'4d76d3aa0bdb822d08000017'</span><span class="p">),</span> <span class="nx">BSON</span><span class="o">::</span><span class="nx">ObjectId</span><span class="p">(</span><span class="s1">'4d76d3a90bdb822d08000015'</span><span class="p">)]}</span></code></pre></div>
<p>And these are account documents</p>
<div class="highlight"><pre><code class="language-javascript" data-lang="javascript"><span class="p">{</span><span class="s1">'_id'</span><span class="o">:</span> <span class="s1">'4d76d3aa0bdb822d08000017'</span><span class="p">,</span> <span class="nx">user_ids</span><span class="o">:</span> <span class="p">[</span><span class="nx">BSON</span><span class="o">::</span><span class="nx">ObjectId</span><span class="p">(</span><span class="s1">'4d76d3a80bdb822d08000005'</span><span class="p">),</span> <span class="nx">BSON</span><span class="o">::</span><span class="nx">ObjectId</span><span class="p">(</span><span class="s1">'4d76d3a90bdb822d08000009'</span><span class="p">),</span> <span class="nx">BSON</span><span class="o">::</span><span class="nx">ObjectId</span><span class="p">(</span><span class="s1">'4d76d3a90bdb822d0800000d'</span><span class="p">)]}</span>
<span class="p">{</span><span class="s1">'_id'</span><span class="o">:</span> <span class="s1">'4d76d3aa0bdb822d0800001b'</span><span class="p">,</span> <span class="nx">user_ids</span><span class="o">:</span> <span class="p">[</span><span class="nx">SON</span><span class="o">::</span><span class="nx">ObjectId</span><span class="p">(</span><span class="s1">'4d76d3a90bdb822d08000009'</span><span class="p">),</span> <span class="nx">BSON</span><span class="o">::</span><span class="nx">ObjectId</span><span class="p">(</span><span class="s1">'4d76d3a90bdb822d08000011'</span><span class="p">)]}</span></code></pre></div>
<p>As mongodb support the Array type, it is really easy to maintain the many to many relationship.</p>
<p>Btw, if you use</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">references_many</span> <span class="ss">:name</span><span class="p">,</span> <span class="ss">:stored_as</span> <span class="o">=></span> <span class="ss">:array</span></code></pre></div>
<p>before, you will receive a runtime error. You should use</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">references_and_referenced_in_many</span> <span class="ss">:name</span></code></pre></div>
<p>instead.</p>
Upgrade Mongoid - Hash arguments for group2011-03-01T00:00:00+00:00http://blog.huangzhimin.com/2011/03/01/upgrade-mongoid-hash-arguments-for-group<p>You will receive a warning for the group method call after upgrading mongoid.</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash">Collection#group no longer take a list of paramters. This usage is deprecated.</code></pre></div>
<p>exactly this is because mongo gem changes the group method definition.</p>
<p>Before</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">key</span> <span class="o">=</span> <span class="o">[</span><span class="s2">"ad_id"</span><span class="o">]</span>
<span class="n">conditions</span> <span class="o">=</span> <span class="p">{</span> <span class="s1">'ad_id'</span> <span class="o">=></span> <span class="p">{</span> <span class="s1">'$in'</span> <span class="o">=></span> <span class="n">ad_ids</span> <span class="p">}</span> <span class="p">}</span>
<span class="n">initial</span> <span class="o">=</span> <span class="p">{</span> <span class="s2">"impressions"</span> <span class="o">=></span> <span class="mi">0</span><span class="o">.</span><span class="mi">0</span><span class="p">,</span> <span class="s2">"clicks"</span> <span class="o">=></span> <span class="mi">0</span><span class="o">.</span><span class="mi">0</span> <span class="p">}</span>
<span class="n">reduce</span> <span class="o">=</span> <span class="s2">"a reduce javascript function"</span>
<span class="no">AdStat</span><span class="o">.</span><span class="n">collection</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">conditions</span><span class="p">,</span> <span class="n">initial</span><span class="p">,</span> <span class="n">reduce</span><span class="p">)</span><span class="o">.</span><span class="n">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">e</span><span class="o">|</span>
<span class="o">.</span><span class="n">.</span><span class="o">.</span><span class="n">.</span><span class="o">.</span><span class="n">.</span>
<span class="k">end</span></code></pre></div>
<p>After</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">key</span> <span class="o">=</span> <span class="o">[</span><span class="s2">"ad_id"</span><span class="o">]</span>
<span class="n">conditions</span> <span class="o">=</span> <span class="p">{</span> <span class="s1">'ad_id'</span> <span class="o">=></span> <span class="p">{</span> <span class="s1">'$in'</span> <span class="o">=></span> <span class="n">ad_ids</span> <span class="p">}</span> <span class="p">}</span>
<span class="n">initial</span> <span class="o">=</span> <span class="p">{</span> <span class="s2">"impressions"</span> <span class="o">=></span> <span class="mi">0</span><span class="o">.</span><span class="mi">0</span><span class="p">,</span> <span class="s2">"clicks"</span> <span class="o">=></span> <span class="mi">0</span><span class="o">.</span><span class="mi">0</span> <span class="p">}</span>
<span class="n">reduce</span> <span class="o">=</span> <span class="s2">"a reduce javascript function"</span>
<span class="no">AdStat</span><span class="o">.</span><span class="n">collection</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="ss">:key</span> <span class="o">=></span> <span class="n">key</span><span class="p">,</span> <span class="ss">:conditions</span> <span class="o">=></span> <span class="n">conditions</span><span class="p">,</span> <span class="ss">:initial</span> <span class="o">=></span> <span class="n">initial</span><span class="p">,</span> <span class="ss">:reduce</span> <span class="o">=></span> <span class="n">reduce</span><span class="p">)</span><span class="o">.</span><span class="n">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">e</span><span class="o">|</span>
<span class="o">.</span><span class="n">.</span><span class="o">.</span><span class="n">.</span><span class="o">.</span><span class="n">.</span>
<span class="k">end</span></code></pre></div>
<p>This is the usage of hash arguments, it makes the group calling more readable.</p>
Upgrade Mongoid - Default Type for Field2011-01-28T00:00:00+00:00http://blog.huangzhimin.com/2011/01/28/upgrade-mongoid-default-type-for-field<p>If you have watched the episode about <a href="http://railscasts.com/episodes/238-mongoid">mongoid</a> from railscast, ryanb removed the default type String for field, like</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">class</span> <span class="nc">Article</span>
<span class="n">field</span> <span class="ss">:name</span><span class="p">,</span> <span class="ss">:type</span> <span class="o">=></span> <span class="nb">String</span>
<span class="n">field</span> <span class="ss">:content</span><span class="p">,</span> <span class="ss">:type</span> <span class="o">=></span> <span class="nb">String</span>
<span class="k">end</span></code></pre></div>
<p>can be written as</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">class</span> <span class="nc">Article</span>
<span class="n">field</span> <span class="ss">:name</span>
<span class="n">field</span> <span class="ss">:content</span>
<span class="k">end</span></code></pre></div>
<p>but it is not valid from mongoid.2.0.0.rc.1 again, the default type of field is changed from String to Object, that means we should explicitly set the type for each field.</p>