ixti's personal scratchpad

Delegate in Ruby and Rails

2014-07-18T00:00:00+02:00

I believe most of developers in Rails community use delegate method to define delegators, because (according to documentation) it looks nicer and it’s API more human readable. I disagree and encourage you to look at Forwardable.

First of all I would like to talk about “beauty”. Since Ruby 1.9.1, Forwardable provides a decorated def_instance_delegator with API similar to Rails one:

class Boss
  extend Forwardable

  # delegate single method
  delegate :schedule => :secretary

  # or a bunch of them
  delegate [:calls_filter, :mails_filter] => :secretary
end

Compare that to:

class Boss
  # delegate single method
  delegate :schedule, :to => :secretary

  # or a bunch of them
  delegate [:calls_filter, :mails_filter], :to => :secretary
end

Looks pretty similar, huh? So, what you will miss if you will use Ruby’s def_instance_delegator (or it’s helper delegate)? The list is pretty short:

Silent return of nil when target is nil and delegator defined with :allow_nil => true option.
Automatic prefixing of bunch of delegated methods

So, why I prefer def_instance_delegator (or def_delegator shortcut) instead? One pretty thing about Forwardable’s delegators is that you can define custom generated delegator method name:

module Workers
  class Base
    extend Forwardable

    # define `option(key)` delegatred to `@options.fetch(key)`
    def_delegator :@options, :fetch, :option

    # mimic Rails' `:prefix => true` option of delegate method
    def_delegator :client, :address, :client_address
  end
end

Now let’s talk about difference in wraper that is generated. Let’s start with Rails:

class Invoice
  delegate :address, :to => :client, :prefix => true

  # the above equals of writing following:

  def client_address(*args, &block)
    _ = client
    if !_.nil? || nil.respond_to?(:address)
      _.address(*args, &block)
    else
      raise DelegationError, "Invoice#client_address delegated to client.address, but client is nil: #{self.inspect}"
    end
  end
end

Now, let’s see what Forwardable variant will generate for us:

class Invoice
  def_delegator :client, :address, :client_address

  # the above equals of writing following:

  def client_address(*args, &block)
    begin
      client.__send__(:address, *args, &block)
    rescue Exception
      $@.delete_if { |s| %r"#{Regexp.quote(__FILE__)}"o =~ s } unless Forwardable::debug
      ::Kernel::raise
    end
  end
end

As you can see main difference is that Forwardable version:

does not checks whenever target responds to delegated message or not, and simply sends that message with given args and block if any.
removes it’s own trace lines from backtrace

Notice, that due to Forwardable uses __send__ on target, it ignores defined message visibility, so you can delegate to protected and private methods of a target. Keep that in mind.

Forwardable can be used to define methods on instances as well:

some_object.extend Forwardable
some_object.delegate "puts" => "STDOUT"
some_object.puts "Tada!"

But, if you want to define delegators on instance you should better use another delegation mixin: SingleForwardable (defined in forwardable). Difference between two is that Forwardable tries to define method via module_eval and then fall-backs to instance_eval, where SingleForwardable uses only instance_eval. That actually allows you to define delegators on class, so instead of writing something like this:

class Implementation
  def self.service
    puts "serviced!"
  end
end

class Facade
  class << self
    extend Forwardable
    def_delegator :Implementation, :service
  end
end

Facade.service # => "serviced!"

You can use SingleForwardable instead right on class:

class Implementation
  def self.service
    puts "serviced!"
  end
end

class Facade
  extend SingleForwardable
  def_delegator :Implementation, :service
end

Facade.service # => "serviced!"

The two variants above generate absolutely the same result. Notice that if you want to use both Forwardable and SingleForwardable at the same time you can use full names of helpers: def_instance_delegator and def_single_delegator. But, in my opinion first variant (with class << self and Forwardable) looks way more readable after all ;))

Share Sidekiq's redis pool with other part of your app

2014-03-26T00:00:00+01:00

If you are running Sidekiq, then, most likely you would like to share it’s redis connection with other bits of your application. For example, you want to implement a throttler for some Rails action, or you want to use some gem, that needs redis, e.g. redis-semaphore. Well, it’s easier than it might sound…

This snippet was extrcated from one of Rails applications I have honor to work on, but it can be used in any other application. There’s only one bit that relies on Rails: MyApplication::Redis#url - it uses Rails.root.

So you should put this snippet somewhere in your app. On Rails, a good choice will be config/initializers/00_redis.rb.

module MyApplication
  module Redis
    class Pool < ::ConnectionPool
      attr_accessor :namespace

      def initialize(options = {})
        super(options.delete :pool) { Redis.new options }
      end

      def with_namespace(ns)
        clone.tap { |o| o.namespace = ns }
      end

      def checkout(*args, &block)
        conn = super(*args, &block)

        if conn && namespace
          return ::Redis::Namespace.new namespace, :redis => conn
        end

        conn
      end

      def wrap
        Wraper.new self
      end

      class Wrapper < ::ConnectionPool::Wrapper
        def initialize(pool)
          @pool = pool
        end
      end
    end

    def pool
      @pool ||= Pool.new config
    end

    private

    def config
      {
        :url      => url,
        :driver   => :hiredis,
        :timeout  => 10.0,
        :pool     => { :size => pool_size }
      }
    end

    def pool_size
      case
      when ::Sidekiq.server? then Sidekiq.options[:concurrency] + 2
      else                        5
      end
    end

    def url
      @url ||= YAML.load(Rails.root.join("config", "redis.yml").read)
        .fetch(Rails.env) { fail "No Redis URL given in config/redis.yml" }
    end
  end


  def self.redis(*args, &block)
    return Redis.pool.wrap unless block_given?
    Redis.pool(*args, &block)
  end
end

Now, it’s time to configure Sidekiq. It’s pretty trivial, instead of passing redis options, we will pass it a ConnectionPool instance:

Sidekiq.configure_client do |config|
  config.redis = ::MyApplication::Redis.pool.with_namespace "sidekiq"
end

Sidekiq.configure_server do |config|
  config.redis = ::MyApplication::Redis.pool.with_namespace "sidekiq"
end

And finally, with our Pool::Wrapper we can use it in redis-semaphore:

semaphore = Redis::Semaphore.new("foobar", :redis => MyApplication.redis)

Mimic dtach with tmux

2013-09-08T00:00:00+02:00

dtach is a tiny program that emulates detach feature of screen. You can use it to run, for example WeeChat in background without suspending it. And tmux is an awesome feature-rich alternative to screen. So why one would want to use tmux in place of dtach? Personally I do because it allows to change X window title, thus I can make sure WeeChat is always opened on 8th workspace in my XMonad without any extra hacks.

So let’s summarize what I want from tmux:

no history
no hotkeys expect the one to “detach”
no status line

To achieve that I’m gonna create a config for tmux and a launcher something like this:

#!/bin/sh
# file ~/.local/bin/weechat
tmux start
tmux -f ~/.tmux/weechat.conf new -d -A -s weechat weechat-curses \; attach

Pretty simple enough, now all we need is a config for tmux:

# Unbind default prefix, so that there will be no hotkeys available and bind
# hotkeys we are interested in:

unbind C-b

bind-key -n C-\   detach
bind-key -n C-F10 kill-session


# Disable (hide) status line completely

set-option -g status off


# Force tmux to resize a window based on the smallest client actually
# viewing it, not on the smallest one attached to the entire session.

set-window-option -g aggressive-resize on


# 256 Colours

set-option -g default-terminal "screen-256color"


# Set history limit

set-option -g history-limit 0


# Window titling for X

set-option -g set-titles on
set-option -g set-titles-string '#W'

And that’s all. To simplify this process I’ve released a Ruby gem dmux. Alternatively you can (if you by some reason aware of Ruby) use initial Shell script of mine:

#!/bin/sh


set -e


APP=$(echo $* | sed -e "s/^  *|  *$//g")
SID=$(echo "$APP" | md5sum | cut -f1 -d" ")
RUN="tmux -L dmux"
CFG="$HOME/.dmux.conf"


if [ -z "$APP" ]; then
  echo "USAGE: dmux <command>" >&2
  exit 1
fi


test -e "$CFG" || cat > "$CFG" <<CONFIG
# Unbind default prefix, so that there will be no hotkeys available and bind
# hotkeys we are interested in:

unbind C-b

bind-key -n C-\   detach
bind-key -n C-F10 kill-session


# Disable (hide) status line completely

set-option -g status off


# Force tmux to resize a window based on the smallest client actually
# viewing it, not on the smallest one attached to the entire session.

set-window-option -g aggressive-resize on


# 256 Colours

set-option -g default-terminal "screen-256color"


# Set history limit

set-option -g history-limit 0


# Window titling for X

set-option -g set-titles on
set-option -g set-titles-string '#W'
CONFIG


session_exists () {
  $RUN has -t "$SID" 2>/dev/null
}


create_session () {
  $RUN new -d -A -s "$SID" "$APP" \; source "$CFG" \; attach
}


attach_session () {
  $RUN attach -t "$SID"
}


session_exists && attach_session || create_session

Just save it as dmux somewhere within your $PATH environment variable and don’t forget to chmod +x it.

XMonad.Action.GridSelect spawnSelected with nice titles

2013-09-07T00:00:00+02:00

There’s an awesome module available for XMonad called GridSelect. It contains very pretty window selectors and such. And one of it’s awesome functions is spawnSelected that aimed to build a grid menu with apps to run. Usage is pretty simple: spawnSelected conf ["xterm", "smplayer", "gvim"]. But what if you want to have nice titles rather than command names?

If you’ll take a look on that function source it zips given list with itself and feeds to gridlist function, which returns snd element of chosen tuple. That means you can easily write your own definitions that will allow to provide nice names for menu elements. So it will look something like this:

Let’s import module first:

import XMonad.Actions.GridSelect

Then we can define our version of spawnSelected like this:

spawnSelected' :: [(String, String)] -> X ()
spawnSelected' lst = gridselect conf lst >>= flip whenJust spawn
  where conf = defaultGSConfig

Now all you need to do is to add a hotkey to run your menu, something like this:

, ((modm, xK_r), spawnSelected'
  [ ("Mozilla Firefox", "firefox")
  , ("GVim",            "gvim")
  , ("MCabber",         "xterm -e detach -A /tmp/mcabber -z mcabber")
  , ("WeeChat",         "xterm -e detach -A /tmp/weechat -z weechat-curses")
  , ("Terminal",        "xterm")
  ])

Get MongoDB existing indexes

2013-04-15T00:00:00+02:00

When your application on an early development stage, your indexes are changing very fast. And most of the time you only add more and more indexes, until you come to the point that either MongoDB fails auto-guessing a proper index to use or indexes are taking too much RAM. This is a simple snippet that allows get a list of existing indexes (except built-in) of all collections in your MongoDB…

db.getCollectionNames().forEach(function(coll) {
  db[coll].getIndexes().forEach(function(index) {
    if ("_id_" !== index.name) {
      print("db." + coll + ".ensureIndex(" + tojson(index.key) + ")");
    }
  });
});

You can either run this script inside mongo shell, or save this into show-indexes.js file and execute it as follows:

$ cat show-indexes.js | mongo --quiet my-database

Using Jekyll plugins on GitHub Pages

2013-01-28T00:00:00+01:00

Few people asked me how they can use jekyll-assets on GitHub pages. There are lots of info out there on this topic, but I would like to aggregate some kind of best practices here. Main thing to understand here is that GitHub renders pages using Jekyll with safe mode enabled. Custom plugins are disabled when safe mode is on. In other words this instruction applies to any third-party plugin…

As I said above, GitHub Pages does not allows you to use custom plugins, so the only way to use Jekyll with custom plugins on GitHub Pages is compiling sources locally and pushing compiled results only. In other words, it becomes absolutely indifferent where to publish your website GitHub Pages or any other hosting of static files.

Following instructions are based on Octopress deployment how-to, so all credits goes to community of that awesome project.

User & Organization Pages

User & Organization Pages live in a special repository dedicated to only the Pages files. This repo must use username/username.github.com naming scheme.

GitHub will use master branch of such repo to build and publish the Pages. That leads us into having master branch with compiled web-site and source branch with our website sources.

Prepare repository

Repo preparation is very simple, just create a source branch in your repo:

$ git checkout -b source master
$ git push -u origin source

Now as you have created source branch you can make it default on GitHub:

Automate publishing

Once repo is ready you can render your website and push compiled sources into master branch. But doing it manually is a pain, so let’s add simple rake task. Create (if you don’t have one yet) a Rakefile and add following into it:

require "rubygems"
require "tmpdir"

require "bundler/setup"
require "jekyll"


# Change your GitHub reponame
GITHUB_REPONAME = "ixti/ixti.github.com"


desc "Generate blog files"
task :generate do
  Jekyll::Site.new(Jekyll.configuration({
    "source"      => ".",
    "destination" => "_site"
  })).process
end


desc "Generate and publish blog to gh-pages"
task :publish => [:generate] do
  Dir.mktmpdir do |tmp|
    cp_r "_site/.", tmp

    pwd = Dir.pwd
    Dir.chdir tmp

    system "git init"
    system "git add ."
    message = "Site updated at #{Time.now.utc}"
    system "git commit -m #{message.inspect}"
    system "git remote add origin git@github.com:#{GITHUB_REPONAME}.git"
    system "git push origin master --force"

    Dir.chdir pwd
  end
end

Now you can simply call rake publish to compile and publish your web-site to GitHub Pages.

Project Pages

Unlike User and Org Pages, Project Pages are kept in the same repo as the project they are for. These pages are almost exactly the same as User and Org Pages, with one main difference: gh-pages branch is used instead of master to build and publish Pages.

There’s no extra repo preapration steps needed. All that you’ll need is a similar, rake task with tiny changes in it:

require "rubygems"
require "tmpdir"

require "bundler/setup"
require "jekyll"


# Change your GitHub reponame
GITHUB_REPONAME = "ixti/jekyll-assets"


namespace :site do
  desc "Generate blog files"
  task :generate do
    Jekyll::Site.new(Jekyll.configuration({
      "source"      => ".",
      "destination" => "_site"
    })).process
  end


  desc "Generate and publish blog to gh-pages"
  task :publish => [:generate] do
    Dir.mktmpdir do |tmp|
      cp_r "_site/.", tmp

      pwd = Dir.pwd
      Dir.chdir tmp

      system "git init"
      system "git add ."
      message = "Site updated at #{Time.now.utc}"
      system "git commit -m #{message.inspect}"
      system "git remote add origin git@github.com:#{GITHUB_REPONAME}.git"
      system "git push origin master:refs/heads/gh-pages --force"

      Dir.chdir pwd
    end
  end
end

Now you can run rake site:publish to compile and publish your web-site to GitHub Pages. Take a look on Jekyll’s own Rakefile as well for alternative implementation of rake site:publish.

Unleash Mr.Hyde! Introduction of Jekyll-Assets.

2012-12-30T00:00:00+01:00

Not so long ago I have released Jekyll-Assets plugin, that adds Rails-alike assets pipeline for Jekyll or Octopress powered blogs. As later Brandon Mathis mentioned, a reference to “rails-alike” gives almost zero understanding of what plugin actually does. In few words, with this plugin you can write your assets in languages like CoffeeScript, SASS, LESS, automagically minify and concatenate assets, and more ;)) So this is an introduction to some of core features and how to enable jekyll-assets for your blog…

As I said before jekyll-assets is a plug-in for Jekyll and Octopress. Setup process is the same for both of them, so examples will be shown on a Jekyll-based blog, but instructions are quiet similar for Octopress users with few differences in “default” paths you have to use.

For visual demonstration I’ll be using an example blog. It is under git control, so you can always see history of it’s evolution.

First of all let’s take a quick look at what we are going to do. Assume we have a very simple blog with some posts. And we’re gonna style it up with Twitter’s Bootstrap framework. And we are going to add some search by tags feature using typeahead jQuery plugin from Bootstraps' toolbox. Also we are going to write our own assets in SASS and CoffeeScript.

Preparations

As we are going to have our javascript assets to be written in CoffeeScript and, we will need ExecJS-supported runtime on system to invoke it.

That means that you’ll need to have Node and CoffeeScript Node module installed. Also, probably, you would like to have UglifyJS Node module installed as well to have minification (alternatively you can use YUI compressor for this purpose).

Initial state

Before we start, let’s overview initial structure of our blog:

.
├── _assets/
│   └── javascripts/
│       └── vendor/
├── _config.yml
├── Gemfile
├── Gemfile.lock
├── index.html
├── _layouts/
│   ├── default.html
│   └── post.html
├── _posts/
│   ├── 2012-01-18-it-s-the-final-countdown-the-only.md
│   └── ...
└── README.md

Installation

First of all we’ll need to add some gems into Gemfile:

#
# jekyll-assets plugin itself
#

gem "jekyll-assets"

#
# Additional gems for jekyll-assets
#

gem "coffee-script" # We want to write our javascripts in CoffeeScript
gem "uglifier"      # And we want our javascripts to be minified with UglifyJS
gem "sass"          # And we want to write our stylesheets using SCSS/SASS

Let’s slow down and take a closer look on the required gems above:

jekyll-assets: This is plug-in itself, nothing to say more
coffee-script: This gem will allow us to write javascripts using CoffeeScript language
uglifier: This gem will allow us to use UglifyJS for compiled javascripts minification
sass: This gem will allow us to write our stylesheets using SASS or SCSS and also it will allow us compress our compiled stylesheets.

After you have put these lines into your Gemfile, run bundle to install dependencies. Now you are ready to enable plug-in. That’s really simple, just add following line into your _plugins/ext.rb file (create it if you don’t have such one yet):

require "jekyll-assets"

That’s all. Installation is complete. Now let’s add some assets.

Some basic assets

By default, jekyll-assets will search for assets under following self-describing paths (relative to your sources root):

_assets/images/
_assets/stylesheets/
_assets/javascripts/

So let’s create these directories and some base assets.

First of all we will put noise.png (nice background lovely generated with noise generator) under _assets/images directory.

Now let’s add some vendors:

we’ll save jQuery as _assets/javascripts/vendor/jquery.js
and Modernizr as _assets/javascripts/vendor/modernizr.js
and all SASS files of SASS port of Bootstrap under _assets/stylesheets/vendor/bootstrap directory.

OK. Now let’s ~~put a smile on that face~~ style our blog a little bit. To do so we’ll create a simple _assets/stylesheets/app.css.sass file first with something like this:

body
  background-color: #eee
  background-image: url(asset_path('noise.png'))

Notice that despite the fact that noise.png is kept under different directory we refer it using so-called “logical path” which is relative to the paths where jekyll-assets will look for assets. Let’s include bootstrap, the easiest way would be to add a line like this one to the op of our main stylesheet file:

//= require vendor/bootstrap/bootstrap

But first of all that looks ugly, secondly, we don’t need “full” bootstrap. So we will require vendor/bootstrap instead, will create this file as _assets/stylesheets/vendor/bootstrap.css.scss and will include only those parts we interested in:

@import "bootstrap/variables";
@import "bootstrap/mixins";

// CSS Reset
@import "bootstrap/reset";

// Grid system and page structure
@import "bootstrap/scaffolding";
@import "bootstrap/layouts";

// Base CSS
@import "bootstrap/type";
@import "bootstrap/forms";

// Components: common
@import "bootstrap/dropdowns";

// Components: Nav
@import "bootstrap/navs";
@import "bootstrap/navbar";

// Components: Misc
@import "bootstrap/labels-badges";

// Utility classes
@import "bootstrap/utilities";

So, let’s mixin helpers of Bootstrap into our main stylesheet and make it even more funky. Here’s what our app.css.sass will look like after all:

//= require vendor/bootstrap


@import "vendor/bootstrap/mixins"


body
  background-color: #eee
  background-image: url(asset_path('noise.png'))


#content
  background-color: #fff
  border: 1px solid #ccc

  @include border-radius(6px)
  @include box-shadow(0 1px 3px rgba(0,0,0,.1))

  margin: 25px auto
  padding: 25px
  width: 940px


#js-tag-search
  display: none


.js
  body
    @include opacity(0)

    &.is-ready
      @include opacity(100)
      @include transition(opacity 1s)

  #js-tag-search
    display: block

Now, when our styles are ready, let’s add some basic JavaScript file as well. Let’s create _assets/javascripts/app.js.coffee with following content:

#= require vendor/jquery

$ ->
  $("body").addClass "is-ready"

Hooray! We’re ready to use these assets in our layout. But before we’ll continue here’s an overview of our assets:

./_assets/
├── images/
│   └── noise.png
├── javascripts/
│   ├── app.js.coffee.erb
│   └── vendor/
│       ├── jquery.js
│       └── modernizr.js
└── stylesheets/
    ├── app.css.sass
    └── vendor/
        ├── bootstrap/
        │   ├── _accordion.scss
        │   └── ...
        └── bootstrap.css.scss

Using assets

Here’s how our layout will look like:

<!DOCTYPE html>
<html>
  <head>
    <title>My Blog</title>
    {% stylesheet app %}
    <script type="text/javascript">{% asset vendor/modernizr %}</script>
  </head>
  <body>
    <div id="content">
      <div class="container">
        {{ content }}
      </div>
    </div>

    {% javascript app %}
  </body>
</html>

I believe that this example is simple enough to understand, but here'a a quick overview of what is happening there:

{% stylesheet app %} will generate a <link> tag with URL to our app.css (_assets/stylesheets/app.css.sass)
{% asset vendor/modernizr %} will return compiled body of vendor/modernizr (_assets/javascripts/vendor/modernizr.js)
{% javascript app %} will generate <script> tag with URL to our app.js (_assets/javascripts/app.js.coffee)

Notice, that we may avoid file extension when we use stylesheet or javascript liquid tags, so the above are shortcuts for:

{% stylesheet app.css %}
{% javascript app.js %}

Also, notice, that despite that our asset files are in fact have .coffee extension, we call this file as app.js anyway. So we can say that we use “logical” extension. To better understand this concept, all these files will be treated as foobar.js:

foobar.js.coffee
foobar.js.coffee.erb
foobar.coffee
foobar.coffee.erb

Same thing about *.css. This is so called pipeline. So foobar.js.coffee.erb will be first rendered with ERB, then it will be compiled with CoffeeScript. And CoffeeScript is a JavaScript type, so we can avoid .js extension, but we leave it for readability.

Here’s list of Liquid tags you might want to use in your templates:

{% javascript <asset> %}: Generates <script> tag for given asset
{% stylesheet <asset> %}: Generates <link rel="stylesheet"> tag for given asset
{% asset_path <asset> %}: Generates URL for an asset. Useful for images, e.g. <img src="{% asset_path me.jpg %}" alt="Alexey V Zapparov" />
{% asset <asset> %}: Returns compiled source of an asset. You can use it to embed some asset, e.g. Modernizr.

Helpers available inside assets

You can use following helpers/methods inside your ERB, SASS and SCSS files:

asset_data_uri(asset_logical_name): Returns encoded Base64 data URI. Use it to embed images into your stylesheets.
asset_path(asset_logical_name): Returns URL of compiled asset.

Let’s go crazy now!

As final step with asets let’s make our app.js more dynamic and add list of our posts with inks, titles and tags to allow search throught them with JavaScript. For such crazy thing you will need ERB, rename app.js.coffee to app.js.coffee.erb and now you can write something like this inside of it:

console.log <%= JSON.dump site.posts.map { |p| p.to_liquid["title"] } %>

Notice, that to use JSON you will ned to require it first. It’s a standard Ruby library s no need for extra step despite adding follwoing line into your _plugins/ext.rb file:

require "json"

Enough talking, let’s create something useful. We’ll add a navigation bar (from Bootstrap) with search field into our layout like so:

<div class="navbar navbar-static-top">
  <div class="navbar-inner">
    <div class="container">
      <a href="/" class="brand">My Blog</a>

      <form id="js-tag-search" class="form-search navbar-form pull-right">
        <input type="text" class="input-medium search-query"
               placeholder="Type tags to search" />
      </form>
    </div>
  </div>
</div>

Similar to stylesheets, we will save all jQuery plugins from Bootstrap into _assets/javascripts/vendor/bootstrap:

./_assets/
├── images/
│   └── noise.png
├── javascripts/
│   ├── app.js.coffee.erb
│   └── vendor/
│       ├── bootstrap/
│       │   ├── bootstrap-affix.js
│       │   └── ...
│       ├── jquery.js
│       └── modernizr.js
└── stylesheets/
    ├── app.css.sass
    └── vendor/
        ├── bootstrap/
        │   ├── _accordion.scss
        │   └── ...
        └── bootstrap.css.scss

Then we’ll add require bootstrap-typeahead directive into our main file with some initialization code, and finally our app.js.coffee.erb will look like:

#= require vendor/jquery
#= require vendor/bootstrap/bootstrap-typeahead

$ ->
  $("body").addClass "is-ready"

  $("#js-tag-search input").typeahead
    source: <%=
      JSON.dump site.posts.map { |p|
        [p.to_liquid["title"], p.url, p.tags.join(",")]
      }
    %>
    sorter: (items) ->
      $.map items, (o) ->
        JSON.stringify o
    matcher: (item) ->
      0 <= item[2].indexOf @query
    updater: (item) ->
      item = JSON.parse item
      window.location = item[1]
      null
    highlighter: (item) ->
      item = JSON.parse item
      "<a href=\"#{item[1]}\">#{item[0]}</a>"

That’s all! You can start jekyll server now to see it in action.

The world is not enough

It’s time to say jekyll-assets to minify our compiled assets. It’s as easy as adding few lines into your _config.yml file:

assets:
  compress:
    css:  sass
    js:   uglifier

For detailed information about possible configuration options refer jekyll-assets or contact me.

Mass-restore files from corrupted media with sleuthkit

2012-10-25T00:00:00+02:00

Not so long ago, I mentioned that one of my external hard drives become little bit too slow, and was not able to remove some files, and system was always proposing me to run file system check. But I was always in hurry, until one day, when I decided to move files to another drive. Unfortunately, by some reason, I decided to run fsck before copying file on another drive. When I came in the morning my drive was not working properly anymore…

So, I lost all my ~~porn~~ photos. It was impossible to mount it, and the syslog was full of messages like Buffer I/O error on device sdb. Googling left me with sad feeling that I will never get my data recovered. But I was wrong and thanks to all developers of safecopy, testdisk, sleuthkit and autopsy.

Grabbing image of a failing drive

As you might already guessed, dd was useless for me. So here’s where might safecopy comes for the rescue :)) This brilliant application helped me to get as much data from drive as possible without any extra work:

$ cd /media/storage/recover-my-broken-drive
$ safecopy --stage1 /dev/sdb sdb-data

...

$ safecopy --stage2 /dev/sdb sdb-data

...

$ safecopy --stage3 /dev/sdb sdb-data

Use man safecopy for details. :))

Bringing data back alive

So the next day I was able to run testdisk on the result image file. And it was able to determine my filesystem. Even more, I was able to mount this image, but it was working really buggy. So I have investigated that image with autopsy. I needed all files under one particular directory and all it’s sub-directories. Unfortunately autopsy does not gives you an option to recover “all files from that directory”. But it gives you an inode of each file and directory. That was what I was looking for. Now, knowing the inode of my diretory I was able to get the list of all nested entries (files, dirs, etc):

$ fls -f ext2 -p -r ./sdb-data 8650754

r/r 8651063:    INCITS+ISO+IEC+14882-2003.pdf
...

You can read details on the output format on a [fls wiki page][fls-wik]. But in short I needed all dirs/files that are not deleted, so the final command was:

$ fls -f ext2 -p -r ./sdb-data 8650754 \
  | grep -v '^..-' | grep -v '^... \*' > files.lst

Now I had a “clean” list of files and directories I need to restore. To restore a directory, all we have to do is to, surprise-surprise, mkdir it ;)) but to recover a file, we need to output it’s content. To get contents of a file from the image byt it’s inode, I used another tool from the sleuthkit box: icat. Grabbing each file manually is dead-boring task, so here’s a small BASH script that did all the dirty work for me:

IMAGE=$1
LIST=$2
DEST=$3
HT=`printf '\x09'`

cat $LIST | while read line; do
    filetype=`echo "$line" | awk {'print $1'}`
    filenode=`echo "$line" | awk {'print $2'}`
    filenode=${filenode%:}
    filename=`echo "$line" | cut -f 2 -d "$HT"`

    if [ $filetype == "r/r" ]; then
        echo "$filename"
        mkdir -p "`dirname "$DEST/$filename"`"
        icat -f ext2 -r -s $IMAGE "$filenode" > "$DEST/$filename"
    fi
done

And again, man icat for details ;))

That’s all!

Hope this will help somebody. And I hope I will never need it by myself. ;))

UPDATE (Tue 26 Nov 2013 11:22:34 PM CET)

If files extractor above doesn’t work for you, you might want try version, kindly proposed by Dan O'Day in comments:

IMAGE=$1
LIST=$2
DEST=$3

while IFS=$' \t:' read filetype filenode filename; do
    if [ "$filetype" = "r/r" ]; then
        echo "$filename"
        mkdir -p "`dirname "$DEST/$filename"`"
        icat -f ntfs -r -s $IMAGE "$filenode" > "$DEST/$filename"
    fi
done < $LIST

Bye, Bye, Toto!

2012-10-22T00:00:00+02:00

Finally, I have replaced my toto-based blog with real monster - Mr. Jekyll. Thanks, Tom Preston-Werner AKA mojombo for this awesome project. So, I hope from now on, blogging will become easier for me. And I’ll be posting more frequently once I’ll finish some of the outstanding issues with my current blog setup…

~~I need a good categories/tags generation plugin (all existing are not good enough for me, so I guess I’ll end up writing my own).~~
~~I need to recreate atom.xml feed generator.~~
~~Would be awesome to integrate QR code with URL generator from old blog into this one.~~ (Rejected: QRCode with RMagick was way too slow)
~~Integrate back disquss and move existing threads to the new locations.~~
~~Add rake new_post~~

Base64 encode/decode of UTF8 in browser with JS

2011-11-11T00:00:00+01:00

While working on live demo of JS-YAML, our pure JavaScript port of exciting PyYAML. I found that there’s absolutely no ready solutions for encoding and decoding Base64 in JavaScript for unicode. Tons of Base64 encoders/decoders that will blow out your unicode string like я обожаю ириьски into something absolutely useless. Tons of utf8 encoders/decoders that just do not work. So I wrote my own…

Let’s start with encoding/decoding Unicode strings. The easiest part here is decoding:

function utf8Decode(bytes) {
  var chars = [], offset = 0, length = bytes.length, c, c2, c3;

  while (offset < length) {
    c = bytes[offset];
    c2 = bytes[offset + 1];
    c3 = bytes[offset + 2];

    if (128 > c) {
      chars.push(String.fromCharCode(c));
      offset += 1;
    } else if (191 < c && c < 224) {
      chars.push(String.fromCharCode(((c & 31) << 6) | (c2 & 63)));
      offset += 2;
    } else {
      chars.push(String.fromCharCode(((c & 15) << 12) | ((c2 & 63) << 6) | (c3 & 63)));
      offset += 3;
    }
  }

  return chars.join('');
}

This function is straight forward and dead simple, so we won’t talk about it too much. It receives array of unicode bytes and returns a simple string. Now, the most interesting part is encode unicode stream into array of bytes.

I saw lots of variants and all they was not working properly. Mostly because authors forget that String#length on unicode string in browser is little bit unpredictable (otherwise you would not read this post). So we can simply throw away about 95% of such implementations. I even saw attempt to use String#split to calculate length of the string - nice but useless try…

The most viable solution I saw was using regular expressions. The idea was good, the implementation was shit - in fact, author was simply lack of regular expression usage practice. Anyway, everybody miss the most simple solution… We have native function encodeURI which converts a string to be used as part of the URI, and that string contains bytes representation of non-ASCII chars n form of %XX, where XX is a hexadecimal code. So we can use such string to build desired array of bytes. And here we are:

function utf8Encode(str) {
  var bytes = [], offset = 0, length, char;

  str = encodeURI(str);
  length = str.length;

  while (offset < length) {
    char = str[offset];
    offset += 1;

    if ('%' !== char) {
      bytes.push(char.charCodeAt(0));
    } else {
      char = str[offset] + str[offset + 1];
      bytes.push(parseInt(char, 16));
      offset += 2;
    }
  }

  return bytes;
}

OK. now the most easiest part - Base64 encode/decode. I would not talk too much about it, as it’s straight forward task, and in fact almost every solution I found on the Internet was working (more or less). So I used the most easy to read and understand one - the part of Mozilla’s XML-RPC client with few tuning I made to meet my requirements of code style mostly. Here’s the result:

(function (global) {
  'use strict';

  var log = function () {},
      padding = '=',
      chrTable = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz' +
                 '0123456789+/',
      binTable = [
        -1,-1,-1,-1, -1,-1,-1,-1, -1,-1,-1,-1, -1,-1,-1,-1,
        -1,-1,-1,-1, -1,-1,-1,-1, -1,-1,-1,-1, -1,-1,-1,-1,
        -1,-1,-1,-1, -1,-1,-1,-1, -1,-1,-1,62, -1,-1,-1,63,
        52,53,54,55, 56,57,58,59, 60,61,-1,-1, -1, 0,-1,-1,
        -1, 0, 1, 2,  3, 4, 5, 6,  7, 8, 9,10, 11,12,13,14,
        15,16,17,18, 19,20,21,22, 23,24,25,-1, -1,-1,-1,-1,
        -1,26,27,28, 29,30,31,32, 33,34,35,36, 37,38,39,40,
        41,42,43,44, 45,46,47,48, 49,50,51,-1, -1,-1,-1,-1
      ];

  if (global.console && global.console.log) {
    log = function (message) {
      global.console.log(message);
    };
  }

  // internal helpers //////////////////////////////////////////////////////////

  function utf8Encode(str) {
    var bytes = [], offset = 0, length, char;

    str = encodeURI(str);
    length = str.length;

    while (offset < length) {
      char = str[offset];
      offset += 1;

      if ('%' !== char) {
        bytes.push(char.charCodeAt(0));
      } else {
        char = str[offset] + str[offset + 1];
        bytes.push(parseInt(char, 16));
        offset += 2;
      }
    }

    return bytes;
  }

  function utf8Decode(bytes) {
    var chars = [], offset = 0, length = bytes.length, c, c2, c3;

    while (offset < length) {
      c = bytes[offset];
      c2 = bytes[offset + 1];
      c3 = bytes[offset + 2];

      if (128 > c) {
        chars.push(String.fromCharCode(c));
        offset += 1;
      } else if (191 < c && c < 224) {
        chars.push(String.fromCharCode(((c & 31) << 6) | (c2 & 63)));
        offset += 2;
      } else {
        chars.push(String.fromCharCode(((c & 15) << 12) | ((c2 & 63) << 6) | (c3 & 63)));
        offset += 3;
      }
    }

    return chars.join('');
  }

  // public api ////////////////////////////////////////////////////////////////

  function encode(str) {
    var result = '',
        bytes = utf8Encode(str),
        length = bytes.length,
        i;

    // Convert every three bytes to 4 ascii characters.
    for (i = 0; i < (length - 2); i += 3) {
      result += chrTable[bytes[i] >> 2];
      result += chrTable[((bytes[i] & 0x03) << 4) + (bytes[i+1] >> 4)];
      result += chrTable[((bytes[i+1] & 0x0f) << 2) + (bytes[i+2] >> 6)];
      result += chrTable[bytes[i+2] & 0x3f];
    }

    // Convert the remaining 1 or 2 bytes, pad out to 4 characters.
    if (length%3) {
      i = length - (length%3);
      result += chrTable[bytes[i] >> 2];
      if ((length%3) === 2) {
        result += chrTable[((bytes[i] & 0x03) << 4) + (bytes[i+1] >> 4)];
        result += chrTable[(bytes[i+1] & 0x0f) << 2];
        result += padding;
      } else {
        result += chrTable[(bytes[i] & 0x03) << 4];
        result += padding + padding;
      }
    }

    return result;
  }

  function decode(data) {
    var value, code, idx = 0,
        bytes = [],
        leftbits = 0, // number of bits decoded, but yet to be appended
        leftdata = 0; // bits decoded, but yet to be appended

    // Convert one by one.
    for (idx = 0; idx < data.length; idx++) {
      code = data.charCodeAt(idx);
      value = binTable[code & 0x7F];

      if (-1 === value) {
        // Skip illegal characters and whitespace
        log("WARN: Illegal characters (code=" + code + ") in position " + idx);
      } else {
        // Collect data into leftdata, update bitcount
        leftdata = (leftdata << 6) | value;
        leftbits += 6;

        // If we have 8 or more bits, append 8 bits to the result
        if (leftbits >= 8) {
          leftbits -= 8;
          // Append if not padding.
          if (padding !== data.charAt(idx)) {
            bytes.push((leftdata >> leftbits) & 0xFF);
          }
          leftdata &= (1 << leftbits) - 1;
        }
      }
    }

    // If there are any bits left, the base64 string was corrupted
    if (leftbits) {
      log("ERROR: Corrupted base64 string");
      return null;
    }

    return utf8Decode(bytes);
  }

  global.base64 = {encode: encode, decode: decode};
}(window));

These are all you need to Base64 encode/decode unicode in browser properly. Hope this will help and save you some time. :))

You can download full solution from the bits of this article below. The usage is pretty simple. Require it, e.g. like this:

<script src="js/base64.js"></script>

and then use it like this:

var enc = base64.encode('я обожаю ириськи');
console.log(enc);
// -> '0Y8g0L7QsdC+0LbQsNGOINC40YDQuNGB0YzQutC4'

UPDATE (2014-12-03)

This was written long time ago, and since then there’s an awesome and maintained package by Dan Kogai. Also I encourage you to take a look at MDN’s Base64 encoding and decoding article as well.