QuarkRuby - Home

Consume non rails-style REST API's

2008-03-11T11:17:00Z

ActiveResource is a great concept which consumes rails-style REST API but unfortunately most of the REST API's are not rails-style. This means that very frequently you will end up modifying ActiveResource to consume non rails-style REST API's. This article is about understanding ActiveResource and how to tweak/extend it to consume non rails-style REST API's. We will mainly concentrate on reading data i.e. the GET method.

Introduction
Consume non rails-style REST API
Custom HTTP GET method tweaks
Data Format

Introduction

Let me recall the purpose of ActiveResource as stated in ActiveResource README :

Active Resource attempts to provide a coherent wrapper object-relational mapping for REST web services. It follows the same philosophy as Active Record, in that one of its prime aims is to reduce the amount of code needed to map to these resources. This is made possible by relying on a number of code- and protocol-based conventions that make it easy for Active Resource to infer complex relations and structures.

Or, Model classes are mapped to remote REST resources by Active Resource much the same way Active Record maps model classes to database tables

The CRUD Mapping to REST (or ActiveRecord Mapping to ActiveResource) :

Create	POST
Read	GET -- our target
Update	PUT
Delete	DELETE

A minimalistic ActiveResource Model class looks as follows:

class Product < ActiveResource::Base
  self.site = "http://www.quarkrank.com/"
end
# Now, one can query quarkrank.com's api to get all products, or complete details for a particular product by simply doing a find:
# Product.find(:all) => http://www.quarkrank.com/products.xml
# Product.find("canon-powershot-sd1000") => http://www.quarkrank.com/products/canon-powershot-sd1000.xml

Nested Resources: Some resources depend on other resources for e.g. comments on a blog would always depend on of the blog post. The comments can't exist independently. So, url for comments would be something like: www.myblog.com/posts/a_post/comments. Which means that url for finding comments would require an blog_post id.

And if one is accessing nested resources, model class would look like:

class Review < ActiveResource::Base
  # here reviews exist for a given product only.
  self.site = "http://www.quarkrank.com/products/:product_id/"
end
# Now, you can ask for reviews on canon-powershot-sd1000 by doing find:
# Review.find(:all, :params=>{:product_id=>'canon-powershot-sd1000'})

For better understanding further, I would really appreciate if you could go through Ryan's presentation on ActiveResource and Railscasts ActiveResource episodes for better understanding of ActiveResource basics.

Method Call Flow in a Get Request
Lets say, we are doing a find query on some ActiveResource Model.
Note: (phrases in braces denote the actual method calls being made)

 find
  |  
  |-- find single or all items(find_single/find_every)
        |
        |-- create_url (element_path/collection_path)
        |
        |-- get_response_from_url (connection.get)
               |
               |-- make_http_request_to_url (http.get)
               |
               |-- handle_response(response) -> exceptions are raised here if we get 4xx/5xx response codes.
               |
               |-- get_body(response.body)
               |
               |-- decode_output, from xml/json to ruby hash (format.decode)
        |
        |-- convert_hash_to_active_resource_object (instantiate_record/instantiate_collection)

Consume non rails-style REST API

As we plan to talk about the GET operation, lets get deeper into the following steps :

Creating URL for remote resources

Sometimes, we might need to change the REST style url generation. At time of writing this article, most of the popular API's do not follow the rails restful url generation. So, the first step is to create the URL before a third party resource call is made. The URL is constructed using element_path or collection_path methods, depending on whether the response has 1 element or a set of elements respectively.
So, here is the code and little explanation of element_path method.

# code of element_path function
def element_path(id, prefix_options = {}, query_options = nil)
  prefix_options, query_options = split_options(prefix_options) if query_options.nil?
  # path to the resource, which we want to access is evaluated in this statement: 
  "#{prefix(prefix_options)}#{collection_name}/#{id}.#{format.extension}#{query_string(query_options)}"
end

Explanation: Lets look at each of the variable/method used in last statement above.

prefix(prefix_options) depends on self.site variable and value(s) of nested resource variable.
   =>Evaluates the "fixed" prefix path to the resource (if any, mentioned in self.site variable) and/or
      in case you are using nested style queries, replaces variables with their values
== Examples:
1. self.site = "http://www.quarkrank.com/folders/api"
prefix(prefix_options) => "/folders/api/"
2. self.site = "http://www.quarkrank.com/folders/:folder_id"
find(1,:folder_id=>5)
prefix(prefix_options) => "/folders/5/"

1234	collection_name => evaluates to pluralize form of classname== Examples:1. class Comment < ActiveResource::Base;endcollection_name => "comments"

12	id => id of the item we are querying, usually mentioned in find (example: User.find(5))format_extension => what format request you are making request for (default is xml).

query_string(query_options) => generates get styled query string from remaining params.
== Examples:
1. self.site = "http://www.quarkrank.com/folders/api"
    find(1)
    query_string(query_options) => ""
2. self.site = "http://www.quarkrank.com/folders/:folder_id"
    find(1,:folder_id=>5, :filename=>"nakul")
    query_string(query_options) => "?filename=nakul"

## collection_path method is quite similar

So, in case you want to modify the element_path, just redefine the method in your model class with custom definition. Please look into ActiveYoutube class code as an example.

Make a GET request

After creating url, request is send using Net::HTTP (ssl requests are supported).
Note that, private method "request" is called for making a Net::HTTP request, which logs the request being made and response from api server. This logging might lead to exception, because of the following line in the code.

12	# in case, result.message contains some characters like "%A", this leads to exceptionlogger.info "--> #{result.code} #{result.message} (#{result.body ? result.body : 0}b %.2fs)" % time if logger

So, in case you are getting exception try to switch off the logs.

Handling (Custom) Response

ActiveResource relies on response code to figure out errors/success/redirection but this might not always be true. Most of the API's do not respect this. Its quite common to see possible errors like unauthorized access, forbidden access, server error etc in success response.
For example, on unauthorized access, Flickr's API returns 200 OK response code with xml response describing the failure. (Facebook API also belongs to this league)

How to handle these errors? : ActiveResource currently doesn't supports callback hooks like after_find etc. So, one cannot hook the custom handlers for response handling. While, work is in progress for having callbacks support in ActiveResource, but till then we need to handle them on our own. So, for Flickr API, one solution will look like:

## define a ActiveResource::Flickr  class, which raises exception if response is not OK.
  class ActiveFlickr<ActiveResource::Base
    class << self
      alias :old_find :find
      def find(*arguments)
        output = old_find(*arguments)
        if output.respond_to? :err
          case output.err.code.to_i
            when 100
              raise(ActiveResource::UnauthorizedAccess.new(output.err, output.err.msg))
            when 112
              raise(ActiveResource::MethodNotAllowed.new(output.err, output.err.msg))
            else
              raise(ConnectionError.new(output.err, "Unknown response code: #{outout.err.code}"))
          end
        end
      end
   end

# now other ActiveResource models would inherit from ActiveFlickr, rather ActiveResource::Base.

## also ActiveResource Exceptions, currently doesn't logs/prints the "message", which is passed as second argument. 
# You might want to modify the behavior to print error message also.
module ActiveResource
  class ConnectionError
    def to_s
      str = "Failed with #{response.code} #{response.message if response.respond_to?(:message)}\n"
      str += @message unless @message.nil?
      str
    end
  end
end

Parse Response

Next step is to decode the XML/JSON response into ruby object. Decoding is done in get/post method call in connection.rb. XML to hash conversion is done using XmlSimple with some modifications.
There is not much documentation on this conversion but more enthusiastic people can look at from_xml method in: vendor/rails/activesupport/lib/active_support/core_ext/hash/conversions.rb

Create ActiveResource object from parsed response

Convert appropriate elements into ActiveResource objects. Its done using 'load' method in ActiveResource::Base, which takes ruby object as input and maps it into ActiveResource object.

Other things to keep in mind

In last step of find(:all) method call, i.e. conversion of ruby object to ActiveResource Object, instantiate_collection method is called on ruby object. Here, ActiveResource expects an array. This may not be true for many of API's like Amazon, Youtube. So, you might need to rewrite this function:

class ActiveResource::Base
  def self.instantiate_collection(collection, prefix_options = {})
    unless collection.kind_of? Array      [instantiate_record(collection, prefix_options)]
    else
      collection.collect! { |record| instantiate_record(record, prefix_options) }
    end
  end
end

Custom HTTP GET method tweaks

Since simple CRUD/lifecycle methods cannot accomplish every task, ActiveResource supports defining your own custom REST methods. Sometimes we will be using CustomHTTP requests for executing a custom action for a particular resource.

Example: Getting comments for a particular blog post. A sample request could be: www.myblog.com/post/active_resouce/coments.xml. Here, we find a particular blog post and then ask for comments on it. So, ActiveResouce code would be:

# Type1: 
BlogPost.find('active_resouce').get(:comments)
# More examples:
Person.find(1).put(:promote, :position => 'Manager') # PUT /people/1/promote.xml
Person.find(1).delete(:deactivate) # DELETE /people/1/deactivate.xml

Or, sometimes, we might just want the list of active users on website right now.

12	#Type2Person.get(:active) # GET /people/active.xml

find method makes a call to get "id"

The "Type1" custom REST requests actually makes 2 remote requests:

Find the resource for which we want to make a custom request: This is used to find the id of the resource to be used in next step
The actual custom rest request

Here, sometimes we might want to skip step1 if we already know the "id" of the resource. So, one can define different find method which just sets the id param to be used to custom rest request and call get method:

  def find_custom(arg)
    object = self.new
    object.id = arg
    object
  end
# Example: For youtube videos, if we want comments for a particular video, we would do 
# Video.find_custom("ZTUVgYoeN_o").get(:comments)

"get" method doesn't converts hash into objects.

ActiveResource::CustomMethods get request sometimes does not converts the decoded ruby object (from xml) to activeresource objects. You can modify the behavior to get activeresource object

  def get(method_name, options = {})
    self.class.new.load(connection.get(custom_method_element_url(method_name, options), self.class.headers))
  end

  def self.get(method_name, options = {})
    object_array = connection.get(custom_method_collection_url(method_name, options), headers)
    if object_array.class.to_s=="Array"
      object_array.collect! {|record| self.class.new.load(record)}
    else
      self.class.new.load(object_array)
    end
  end

Data Format

Currently 2 formats are supported by ActiveResource: JSON and XML Do you want to use another format? Pretty easy, you need to define 4 methods: extension, mime_type, encode and decode. Encoding is for converting hash into required format and Decoding is for decoding the response in this format into a hash object. Example of JSON format

module ActiveResource
  module Formats
    module JsonFormat
      extend self
      
      def extension
        "json"
      end
      
      def mime_type
        "application/json"
      end
      
      def encode(hash)
        hash.to_json
      end
      
      def decode(json)
        ActiveSupport::JSON.decode(json)
      end
    end
  end
end

Thanks to Brian Nochugi for his frequent discussion/doubts on ActiveResource. It helped us to properly formulate the article.

Active Youtube

2008-02-12T18:57:00Z

ActiveYoutube is a gem to access YouTube API using ActiveResource. This gem wraps code from our previous post on extending ActiveResource to access YouTube. There have been minor changes, which are :

Namespace in class names: Video, User, StandardFeed and Playlist classes have been moved to "Youtube" module, to prevent any conflicts with your ActiveRecord models.
CustomMethods related change: In last version, only response from "find" was converting "entry" object to array of "entry" object. Now, the same behavior is implemented for custom http calls like Video.find().get(:comments)
Small patch for better namespacing: Its basically some code from the rails trunk on ActiveResource, for better handling of namespaces while creating ActiveResource objects.

Gem Installation:

sudo gem install active_youtube

Example Usage:

#### Video
  ## search for videos on 'ruby'
  search = Youtube::Video.find(:first, :params => {:vq => 'ruby', :"max-results" => '5'})
  puts search.entry.length

  ## video information of id = ZTUVgYoeN_o
  vid = Youtube::Video.find("ZTUVgYoeN_o")
  puts vid.group.content[0].url

  ## video comments
  comments = Youtube::Video.find_custom("ZTUVgYoeN_o").get(:comments)
  puts comments.entry[0].link[2].href

  ## searching with category/tags
  results = Youtube::Video.search_by_tags("Comedy")
  puts results[0].entry[0].title

#### STANDARDFEED
  ## retrieving standard feeds
  most_viewed = Youtube::Standardfeed.find(:most_viewed, :params => {:time => 'today'})
  puts most_viewed.entry[0].group.content[0].url

#### USER
  ## user's profile - guthrie
  user_profile = Youtube::User.find("guthrie")
  puts user_profile.link[1].href

#### PLAYLIST
  ## get playlist - multiple elements in playlist
  playlist = Youtube::Playlist.find("EBF5D6DC4589D7B7")
  puts playlist.entry[0].group.content[0].url

Scrapi enhancements

2008-01-30T14:11:00Z

We have been using ruby library Scrapi quite a lot for HTML Scraping in QuarkRank and other projects. Most of the times, I want to extract/scrape specific information from a page and directly dump it into the database. There were a few processes which were regularly repeated in my code, so as to make my code more DRY, I have enhanced Scrapi so that manipulations of extracted information becomes easier.

Lets consider an example, for each of the top 250 movies at IMDB, I want to extract and store in DB the following properties :

imdb_id
name
release date
rating
tagline
runtime
director

So, the process would be :

Get the list of top 250 movies and their URLs.
For each URL, scrape the corresponding properties.
For each property, do cleaning/formatting (like parse Release Date into Date object) if required.
Build a hash out of extracted information.
Create Movie object using hash in above step.

The code for the above process would be something like :

require 'scrapi'
require 'open-uri'

# Step 1: Get list of top 250 movies and their URLs.
#  - Go to http://www.imdb.com/chart/top.
#  - Open up firebug, with firequark installed.
#  - Right click on any movie from the list and select "Inspect This" option from menu. In firebug you will see one selected element.
#  - Right click on this element in firebug, and select option "get css selector" (put 250 in popup input box).
#  - This returns "font>a", which is css selector for getting all 250 movies from this page.
links = (Scraper.define do
  array :urls
  process "font>a", :urls=>"@href"
  result :urls
end).scrape(open("http://www.imdb.com/chart/top").read)

# Step 2: For each movie, get its properties
links.each do |url|
  nurl = "http://www.imdb.com" + url
  hash = (Scraper.define do
    process "h5:content('Release Date:')", :date=>:element
    process "div.general>b:content('User Rating:')", :rating=>:element
    process "h5:content('Tagline:')", :tagline=>:element
    process "h1", :title=>:text
    result :date, :rating, :title, :tagline
  end).scrape(open(nurl).read)

  # Step 3: Cleaning and hash building of properties.
  object = {}
  date_val = hash.date.next_sibling.text
  object[:date] = Date.parse(date_val[0..date_val.rindex('(')||-1]).to_s
  object[:rating] = hash.rating.next_element.text.gsub(/\/.*/,"")
  object[:title] = hash.title[0..hash.title.rindex('(')||-1]
  object[:tagline] = hash.tagline.next_sibling.text.strip if hash.tagline
  object[:imdb_id] = url.gsub(/\/title\//,"").gsub(/\//,"")

  # save the hash to database
  Movie.create(object)
end

Problems in the above code:

Inserting each struct member/value pair into hash explicitly
I would love to do manipulations inline, i.e. when extracting an element. Particularly, in cases where I need to do small/minor modifications like, getting the text corresponding to next element of selected element or converting a string into date object

Following two enhancements in Scrapi allow me to get rid of problems and simplify my code + efforts :

Fix1: Assign value of block manupilation to associated variable.
So, I could write rules like:

1234	# get the text content of element next to "div#footer" into "footer_text" variableprocess "div#footer", :footer_text do \|x\| x.next_element.textend

This is the patch which provides above functionality:

Index: lib/scraper/base.rb
===================================================================
--- lib/scraper/base.rb (revision 268)
+++ lib/scraper/base.rb (working copy)
@@ -296,7 +296,8 @@
           extracts.each do |extract|
             extract.bind(self).call(element)
           end
-          true
+          #true
+          map.keys[0]
         end
       end
 
@@ -499,6 +500,9 @@
           extractor = selector.pop
         elsif selector.last.is_a?(Hash)
           extractor = extractor(selector.pop)
+        elsif selector.last.is_a?(Symbol)
+          block_key = selector.pop
+          attr_accessor block_key
         end
         if block && extractor
           # Ugly, but no other way to chain two calls bound to the
@@ -509,11 +513,24 @@
           extractor2 = instance_method(:__extractor)
           remove_method :__extractor
           extractor = lambda do |element|
-            extractor1.bind(self).call(element)
+            #extractor1.bind(self).call(element)
+            key = extractor1.bind(self).call(element)
             extractor2.bind(self).call(element)
           end
         elsif block
-          extractor = block
+          #extractor = block
+          operator = "="
+          if !@arrays.nil? and @arrays.include? block_key.to_sym
+            extractor = lambda do |element|
+              send(block_key.to_s+"=",[]) if send(block_key.to_s).nil?
+              send(block_key.to_s+"=", send(block_key.to_s) + [block.call(element)])
+            end
+          else
+            extractor = lambda do |element|
+              send(block_key.to_s+"=", block.call(element))
+            end
+          end
         end
         raise ArgumentError, "Missing extractor: the last argument tells us what to extract" unless extractor
         # And if we think the extractor is the last argument,

Fix2: Struct to Hash conversion

class Struct
  def to_hash
    hash = {}
    members.each do |key|
      hash[key.to_sym] = self.send(key)
    end
    hash
  end
end

Using above two fixes, my refactored code becomes cleaner and shorter:

links.each do |url|
  nurl = "http://www.imdb.com" + url
  hash = (Scraper.define do
    process "h5:content('Release Date:')", :date do |elem|
       date_val = elem.next_sibling.text
       Date.parse(date_val[0..date_val.rindex('(')||-1])
      end
    process "div.general>b:content('User Rating:')", :rating do |elem|
        elem.next_element.text.gsub(/\/.*/,"")
      end
    process "h5:content('Tagline:')", :tagline do |elem|
        elem.next_sibling.text
      end
    process "h1", :title do |elem|
        elem.text[0..(elem.text.rindex('(')||0)-1].strip
      end
    result :date, :rating, :title, :tagline
  end).scrape(open(nurl).read)
  object = hash.to_hash
  object[:imdb_id] = url.gsub(/\/title\//,"").gsub(/\//,"")
  Movie.create(object)
end

NOTE: When you run above code snippets, you will get error "text" not defined for HTML::Tag class. Copy this code in your file, which defines this function for HTML::Text and HTML::Tag class.

module HTML
  class Text
    def text
      @content
    end
  end
  class Tag
    def text
      return @content if respond_to? :content
      data=""
      children.each do |child|
        next if child.respond_to?:name and child.name=="script"
        data += child.text
      end
      data
    end
  end
end

I wish this is helpful to the Scrapi community.

ActiveResource and YouTube

2008-01-15T12:49:00Z

This article is about consuming YouTube API in your Ruby/Rails project using ActiveResource. Moreover, this article is an example of how to extend ActiveResource to consume non rest-style API.

Benefits of using our extension to ActiveResource :

ActiveResource provides a ActiveRecord style interface.
You can modify our extension according to your interface requirement.
No not need to use and rely on Ruby library for YouTube REST API.

YouTube references

Note : The extension is using ActiveResource 2.0.2

ActiveResource extension

require 'rubygems'
require 'activeresource'
    
class ActiveYouTube < ActiveResource::Base
  class << self
  ## Remove format from the url.
  def element_path(id, prefix_options = {}, query_options = nil)
    prefix_options, query_options = split_options(prefix_options) if query_options.nil?
    "#{prefix(prefix_options)}#{collection_name}/#{id}#{query_string(query_options)}"
  end
    
  ## Remove format from the url.
  def collection_path(prefix_options = {}, query_options = nil)
    prefix_options, query_options = split_options(prefix_options) if query_options.nil?
    "#{prefix(prefix_options)}#{collection_name}#{query_string(query_options)}"
  end

  ## For a collection call, ActiveResource formatting is not 
  ## compliant with YouTube's output.
  def instantiate_collection(collection, prefix_options = {})
    unless collection.kind_of? Array
      [instantiate_record(collection, prefix_options)]
    else
      collection.collect! { |record| instantiate_record(record, prefix_options) }
    end
  end

  ## To convert output into proper standard.
  ## If single element is present in output then entry is not an array.
  ## So this method will ensure entry is always an array.
  alias :old_find :find
  def find(*args)
    output=old_find(*args)
    if output.respond_to?:entry and !(output.entry.kind_of? Array)
      output.entry=[output.entry]
    end
    output
  end

  ## When using ActiveResource::CustomMethods, ActiveResource first tries to retrieve the id using find() 
  ## and then makes a get() call using that id. 
  ## But, youtube returns the url of this item as id, which we don't want. This method overrides the behavior.
  ## Example: comments = Video.find_custom("ZTUVgYoeN_o").get(:comments)
  def find_custom(arg)
    object = self.new
    object.id = arg
    object
  end

  ##  Following method from ActiveResource::CustomMethods extends the capabilities of activeresource for non-standard urls ;-)
  ##  The objects returned from this method are not automatically converted into ActiveResource instances - they are ordinary Hashes. 
  ##  Modifications below ensures that you get ActiveResource instances.
  def get(method_name, options = {})
    object_array = connection.get(custom_method_collection_url(method_name, options), headers)
    if object_array.class.to_s=="Array"
      object_array.collect! {|record| self.class.new.load(record)}
    else
      self.class.new.load(object_array)
    end
  end
  end

  ## Instance Methods: (modifying the ActiveRecord::CustomMethods).
 
  ## This modification is same as defined in above method 
  def get(method_name, options = {})
    self.class.new.load(connection.get(custom_method_element_url(method_name, options), self.class.headers))
  end

  ## Modifying the url formation to make it Youtube API complaint
  def custom_method_element_url(method_name, options = {})    
    "#{self.class.prefix(prefix_options)}#{self.class.collection_name}/#{id}/" +
    "#{method_name}#{self.class.send!(:query_string, options)}"
  end
end

Points to keep in mind
- When using the ActiveResource::CustomMethods (get/post) make sure you are using "find_custom" and not "find"
- Disable the logging of ActiveResource, since the following line leads to exception if logger is enabled.

12	## line 111, connection.rblogger.info "--> #{result.code} #{result.message} (#{result.body ? result.body : 0}b %.2fs)" % time if logger

Example usage

#### Create classes for YouTube resources.
class Video < ActiveYouTube
  self.site = "http://gdata.youtube.com/feeds/api"

  ## To search by categories and tags
  def self.search_by_tags (*options)
    from_urls = []
    if options.last.is_a? Hash
      excludes = options.slice!(options.length-1)
      if excludes[:exclude].kind_of? Array
        from_urls << excludes[:exclude].map{|keyword| "-"+keyword}.join("/")
      else
        from_urls << "-"+excludes[:exclude]
      end
    end
    from_urls << options.find_all{|keyword| keyword =~ /^[a-z]/}.join("/")
    from_urls << options.find_all{|category| category =~ /^[A-Z]/}.join("%7C")
    from_urls.delete_if {|x| x.empty?}
    self.find(:all,:from=>"/feeds/api/videos/-/"+from_urls.reverse.join("/"))
  end
end

class User < ActiveYouTube
  self.site = "http://gdata.youtube.com/feeds/api"
end

class Standardfeed < ActiveYouTube
  self.site = "http://gdata.youtube.com/feeds/api"
end

class Playlist < ActiveYouTube
  self.site = "http://gdata.youtube.com/feeds/api"
end



##### Examples #######

#### VIDEO
  ## search for videos
  search = Video.find(:first, :params => {:vq => 'ruby', :"max-results" => '5'})
  puts search.entry.length

  ## video information of id = ZTUVgYoeN_o
  vid = Video.find("ZTUVgYoeN_o")
  puts vid.group.content[0].url

  ## video comments
  comments = Video.find_custom("ZTUVgYoeN_o").get(:comments)
  puts comments.entry[0].link[2].href

  ## searching with category/tags
  results = Video.search_by_tags("Comedy")
  puts results[0].entry[0].title
  # more examples:
  # Video.search_by_tags("Comedy", "dog")
  # Video.search_by_tags("News","Sports","football", :exclude=>["Comedy","soccer"])
  # Video.search_by_tags("News","Sports","football", :exclude=>"soccer")


#### STANDARDFEED
  ## retrieving standard feeds
  most_viewed = Standardfeed.find(:most_viewed, :params => {:time => 'today'})
  puts most_viewed.entry[0].group.content[0].url

#### USER
  ## user's profile - guthrie
  user_profile = User.find("guthrie")
  puts user_profile.link[1].href

  ## user's playlist - john
  user_playlist = User.find_custom("john").get(:playlists)
  puts user_playlist.link[1].href

  ## user's upload or favorites
  rick_video = User.find_custom("rick").get(:uploads)
  puts rick_video.entry[0].group.content[0].url

  ## user's subscription
  user_subscriptions = User.find_custom("guthrie").get(:subscriptions)
  puts user_subscriptions.to_yaml

#### PLAYLIST
  ## get playlist - multiple elements in playlist
  playlist = Playlist.find("EBF5D6DC4589D7B7")
  puts playlist.entry[0].group.content[0].url

  ## get playlist - single element in playlist
  playlist = Playlist.find("45C563323B344971")
  puts playlist.entry[0].group.content[0].url

Writing a web widget

2008-01-07T17:25:00Z

Web widgets are widely used across the Internet but still lacks good documentation. From online advertisement to videos to blogs, widgets are highly used. Some of the popular widgets being Google Adsense, Youtube, MyBlogLog widgets and Twitter badges.

Note: This page will be slow to load because of many widget examples.

Before you start
Flash widget
HTML widget
JavaScript widget
1. Passive widgets
2. Active widgets
Examples
Key issues

To formally define : The web widget is a portable chunk of code that can be installed and executed within any separate HTML-based web page by an end user without requiring additional compilation. A widget adds some content to that page that is (mostly) not static. Generally widgets are third party originated, though they can be home made. Widgets are also known as modules, snippets, and plug-ins.

This article is my journey of understanding and making a widget myself. I have tried to make things look simple and insightful by taking a lot of examples.

Before you start

Clearly identify the information which will be distributed by your widget or the information you would like your client to embed in their webpage. Then find the most suitable mechanism to send this information. Mechanism can be one of the following :

Flash (need not be static i.e. it can take parameters) as a widget.
Simple HTML page
JavaScript
JSON API

Flash widget

This is a highly used method for making widgets. Most of the flashy irritating advertisements seen on many websites are flash widgets. A good usage being youtube videos and twitter badge type-1. Flash widget code includes an "embed" or "object" HTML tag with the source to flash file. Extra parameters may be passed to flash file to customize flash content. And remember, ActionScript is always there for more magic :-)

HTML widget

Instead of flash embed an HTML page or HTML code at client's webpage using iframe.

<iframe src="http://www.google.com"  height="100px" width="90%" frameborder="0"></iframe>

Mostly used when content is static for all users (like brand advertisements). Flash and HTML widgets are the most simplest to develop but can be of limited use in many cases.

JavaScript widget

JavaScript is the most powerful tool to make widgets. Top examples being, Google Adsense and MyBlogLog recent visitors widget. There are two ways to develop widgets via scripting:

Passive widget : The widget code is a script tag which links to external JavaScript file and this file has the code to directly print the HTML markup on client's webpage. AideRSS provides a passive widget.

12	// widget code<script src="http://www.quarkruby.com/assets/script-raw-dump.js"></script>

// script-raw-dump.js contains the following code
document.write("<ul>");
document.write("<li>Hello world</li>");
document.write("<li>Hello world2</li>");
document.write("</ul>");
document.write("Powered by <img src='quarkruby' alt='quarkruby'/>");

Output seen on webpage will be:

Active widget : As the name says, they do work/processing/calculations before just printing HTML code. The widget code contains a script tag to download one or more external JavaScript files, which does work (getting client information, passing information around, etc.) and then HTML is dumped at client side. Google adsense, Mybloglog's widget all fall in this category.

Active widgets can be very powerful if, each client is given a unique identifier by the widget source for user identification. This allows widget providing site to send/collect user-related data. Active widget's can be made in two ways :

Using simple JavaScript

<script type="text/javascript">
unique_key_each_host = "quarkruby_123";
// some other variables ..
</script>
<script src="http://www.quarkruby.com/assets/static-javascript-variable.js"></script>

What the code does:
- Get the JavaScript as mentioned in last line above
- This JavaScript file when executed, will read the values of above defined variables + possibly collect other information

// static-javascript-variable.js contains the following code
(function(){
  var url = "http://www.quarkruby.com/show_data";
  var current_location = window.location.hostname;
  url += "?" + "host=" + current_location + "&width="+200 + "&key="+unique_key_each_host;
  // sample url generated : http://www.quarkruby.com/show_data?host=quarkruby.com&width=200&key=quarkruby_123
  document.write("<iframe src='" + url + "'></iframe>");
})()

This creates an iframe element at client's webpage and set the source to an external url (defined in static-javascript-variable.js file) passing above calculated parameters. The output fetched is an HTML page which is inserted iframe created in last line of code

JSON widget
For using JSON API's or JSON data. Twitter badge type 2 is a good example of widget using JSON calls.

1
2
3

<div id="quarkruby_widget"></div>
<script type="text/javascript" src="http://www.quarkruby.com/js/static-javascript-dom.js"></script>
<script text="text/javascript" src="http://www.quarkruby.com/js/json-data-callback.js"></script>

Explanation :
- Download the script files mentioned in last 2 lines. The latter one fetches the JSON data and calls the callback function.

 // sample static-javascript-dom.js file 
  function printHelloWorld(args) {
    var outputhtml = "<h3>List of Visitors: </h3><ul>";
    for(var x=0;x<args.length;x++) {outputhtml += "<li>" + args[i]+"</li>");}
    outputhtml += "</ul>";
    document.getElementById('quarkruby_widget').innerHTML = outputhtml;
  }

  //sample json-data-callback.js
  printHelloWorld(["Nakul", "Ritesh"]);

- The first JavaScript contains the callback function definition (possibly other functions also). These will process the JSON data, generate HTML code and insert into DOM at "#quarkruby_widget" element using innerHTML method or DOM manipulation by creating/adding/appending elements.

Examples

Youtube

<object width="425" height="355">
  <param name="movie" value="http://www.youtube.com/v/dMH0bHeiRNg&rel=1"></param>
  <param name="wmode" value="transparent"></param>
  <embed src="http://www.youtube.com/v/dMH0bHeiRNg&rel=1" type="application/x-shockwave-flash" wmode="transparent" width="425" height="355"></embed>
</object>

- Here, a static youtube video is loaded in flash player.

Output :
<object height="355" width="425"><param></param><param></param><embed src="http://www.youtube.com/v/qbO1zNglwvY&rel=1" height="355" width="425"></embed></object>

Twitter Flash widget

<div style="width:176px;text-align:center">
  <embed src="http://twitter.com/flash/twitter_badge.swf"  flashvars="color1=16594585&type=user&id=8484232"  quality="high" width="176" height="176" name="twitter_badge" align="middle" allowScriptAccess="always" wmode="transparent" type="application/x-shockwave-flash" pluginspage="http://www.macromedia.com/go/getflashplayer" />
  <br>
  <a style="font-size: 10px; color: #FD3699; text-decoration: none" href="http://twitter.com/nakula">follow nakula at http://twitter.com</a>
</div>
<br/>

- This embeds a flash file passing it color of widget, user_id as variables. So, after flash file is loaded, it makes request for user specific data (mentioned in user_id variable) and loads it later.

Output :

AideRSS
Widget code :
<script src="http://www.aiderss.com/widget/top_content/year/100" type="text/javascript"></script>
Here the widget codes gets the JavaScript, which directly does a document.write at client's webpage.

Output :

Google Adsense

This is one of those widget which gathers a lot of information before it generates output.

<script type="text/javascript"><!--
google_ad_client = "pub-xyzvalue";
google_alternate_color = "FFFFFF";
google_ad_width = 468;
google_ad_height = 60;
google_ad_format = "468x60_as";
google_ad_type = "text";
google_ad_channel = "";
google_color_border = "FFFFFF";
google_color_bg = "FFFFFF";
google_color_link = "000000";
google_color_text = "333333";
google_color_url = "666666";
//--></script>
<script src="http://pagead2.googlesyndication.com/pagead/show_ads.js"></script>

It works as follows:

Fetches the JavaScript file mentioned in last line.
This file then uses the defined parameters and gathers value for hidden parameters (so that resulting ads are more local and content specific).
Finally creates an iframe element at client's webpage and set the source to the external url passing above calculated parameters. The output is an HTML page which is inserted into this iframe.

Twitter badge type2

<div id="twitter_div">
<h2 class="twitter-title">Twitter Updates</h2>
<ul id="twitter_update_list"></ul></div>
<script type="text/javascript" src="http://twitter.com/javascripts/blogger.js"></script>
<script text="text/javascript" src="http://twitter.com/statuses/user_timeline/username.json?callback=twitterCallback2&count=5"></script>

The second JavaScript link is the JSON call to their API, which fetches the JSON data and calls back "twitterCallback2" function which is defined in blogger.js file. Then blogger.js downloads the JavaScript code which processes the JSON data and inserts the final output into "twitter_div" element.

Output :

Amazon

<SCRIPT charset="utf-8" type="text/javascript" src="http://ws.amazon.com/widgets/q?ServiceVersion=20070822&MarketPlace=US&ID=V20070822/US/xyz/8001/f5c7593b-5be8-4c15-345d-e8f70675f476"> </SCRIPT> <NOSCRIPT><A HREF="http://ws.amazon.com/widgets/q?ServiceVersion=20070822&MarketPlace=US&ID=V20070822%2FUS%2Fxyz-20%2F8001%2Ff5c75233b-5be8-4c15-916d-e8f70675f476&Operation=NoScript">Amazon.com Widgets</A></NOSCRIPT>

The JavaScript file mentioned in script tag is fetched which contains the JSON data + link to more JavaScript files. These additional JavaScript files are fetched and then JSON data is processed and using DOM manipulation output is inserted into webpage.

Output :
<noscript>Amazon.com Widgets</noscript>

MyBloglog recent visitors widget

<script type="text/javascript" src="http://pub.mybloglog.com/comm2.php?mblID=2007123816343566&c_width=180&c_sn_opt=y&c_rows=5&c_img_size=f&c_heading_text=Recent+Readers&c_color_heading_bg=005A94&c_color_heading=ffffff&c_color_link_bg=E3E3E3&c_color_link=005A94&c_color_bottom_bg=005A94"></script>

- Fetch this file, which generates HTML code + CSS data for the final output
- Fetch one more JavaScript file linked in the above mentioned file which is used for tracking the users.

Output :

Key issues

Slowing down loading of client's site.:
What are you talking? Yes, widgets can bring down client's site who is embedding widget code in their page because:

script tag stops page rendering until code inside it is executed and rendered.
Script tag blocks parallel downloads :(

If the widget source goes down or is slow, the client's webpage won't load unless the widget code is executed and rendered. To avoid this problem consider following workarounds :

Use "defer" attribute with script (firefox doesn't support this though :-( ). This tells the browser to continue rendering and do not block the other downloads. Moreover, this can only be used if there is no "document.write" statement in your script. See more details here

Put timeout on script attachment (source: WEDJE)
Here, external scripts in widget code are not downloaded during page load, but attached after sometime. So, this does not affects loading/rendering of site. Hence, after sometime, the external scripts are downloaded, attached/executed thereafter. In example mentioned below, Twitter type-2 badge is modified to download the external scripts after 1 sec. So, if you embed this code in some html page, you will see the output after delay of 1sec

<div id="twitter_div">
<h2 class="twitter-title">Twitter Updates</h2>
<ul id="twitter_update_list"></ul></div>
<div id="twitter_script"></div>
<script type="text/javascript">
var script1 = document.createElement("script");
script1.src="http://twitter.com/javascripts/blogger.js";
setTimeout("document.getElementById('twitter_script').appendChild(script1)",1000);
var script2 = document.createElement("script");
script2.src="http://twitter.com/statuses/user_timeline/username.json?callback=twitterCallback2&count=5";
setTimeout("document.getElementById('twitter_script').appendChild(script2)",1000);
</script>

Avoid conflict's with client's JavaScript:
Widget code and external loaded Javascript files should contain minimum global variables, so as to prevent any chances of conflict with client's JavaScript which is embedding the widget. If your widget does not take any parameters from client, one should try to use anonymous JavaScript classes.
Let client apply its own CSS to widget:
This helps in customizing the style of widget as per the design/looks of client's webpage. Google allows this even via iframes by using global variables, Amazon allows this while making widget, Mybloglog and twitter doesn't use iframes so your CSS directly applies to the widget's HTML.
Minimize number of request:
This is to load your widget faster which applies to any HTML page
Do you want to prevent content editing?:
If you do not want to allow content editing of the widget, then use iframes since they cannot be manipulated by JavaScript outside the iframe element.
Using API
If you are using an API, then callback method must be provided by that API. This is just to keep in mind if you want to use some external API and display the content dynamically.

Quarkshop : next-generation shopping

2007-12-19T18:56:00Z

QuarkShop is a next-generation shopping experience to find product of your choice based on opinions across the Internet. You give your preference and Quarkshop will fetch the best matched products for YOU.

For launch, currently we have following products :

Camera : http://www.quarkshop.com/cameras
Mp3 player : http://www.quarkshop.com/mp3players
Flat panel TV : http://www.quarkshop.com/flatpaneltvs
Cell phone : http://www.quarkshop.com/cellphones

Search products

You can give your preference by selecting features you LIKE and we will find the best matched products. Selection of preference can be used along with other navigation parameters like brand, price, etc. These features are automatically extracted from reviews.

Then click submit, and you will see the three best matched products with their most related information. There is also an option to see more products. But we think that top 3 products is what consumer care about after they have given their preference, this makes searching for products simple and fast.

Choose your features

Top 3 products

Comparison : QuarkGraph

You can compare the top three products on the spot!. When you click on Compare Top Three button, you will see QuarkGraph, the graphical comparison showing scores of feature for each product. The graph is just not an image but you can play with it.

Features & Snippets : Voice of consumer

Features are keywords that are reviewed in a review. When you read a review you are always looking for such keywords and people's comment about this keywords. Quarkshop gives an option to choose features/keywords that you like and it will give you the best matched products. You can also read sentence from actual reviews giving opinion on keywords (Snippets).

Feedback

We would really appreciate if you could give your feedback. It would help us to know where we stand and what is required to make useful shopping experience.

introducing QuarkRank

2007-12-17T10:32:00Z

We're back to blogging after taking a leave for more than a month. We have been very busy developing QuarkRank, a summarized reviews repository. It is a result of more than 18 months of dedicated research on Natural language processing, HTML Scrapping and User interface. Finally, we are happy to make this product live!!!

Currently, the repository is accessible using RESTful API or Widget. Moreover, its absolutely free!

About QuarkRank

"From product reviews, restaurant reviews, hotel reviews, to others, QuarkRank provides the information for making decisions at the point of purchase. Proprietary technology lies at the core of QuarkRank's ability to automatically summarize the opinions of millions of consumer reviews on the internet."

QuarkRank is an intelligent engine which crawls the web for opinions on various products/services and automatically summarize them feature-by-feature using its natural language processing technique.

Best and worst of Apple Ipod Touch 8gb

QuarkGraph: QuarkRank data displayed using amcharts at Quarkshop

QuarkRank will help consumers to quickly educate themselves, based on the most unbiased information possible, without spending hours reading review online.

If you use QuarkRank data, your customers will feel confident in making purchase decision at your site, without going to competitors, and at the same time reducing the return rate of impulse purchases.

QuarkRank provides

Reviews gathered and combined at one place! for various products and services.
QuarkRank, no. of reviews for a product/service
Top 5 feature, Worst 3 feature, All features that people talk about a product/service. For example : sound quality, design, screen of a Mp3 player.
Feature score, buzz and SNIPPETS from reviews which have opinion about a feature.

No need to waste time analyzing reviews at Cnet and Amazon anymore!

Where can you use it?

Boost online shopping experience of your users.
- Best and Worst about a product.
- Graphical feature-by-feature comparison.
- Power your navigation by giving feature as an option to choose.
- Summarized form of reviews.
Add product widget to your blog/article.
On your social profile, add widget of your favourite or owned products.
Show feature-by-feature opinions and comparison for products at your retail store.

API

QuarkRank provides a RESTful API to access our huge repository of summarized reviews. Send us simple HTTP requests and it will send back basic XML responses, which means you can interact with our API from any language.

It provides data in XML and JSON format. There is no limit is using the api. For detailed information, visit : ActiveResource can be used to access QuarkRank's RESTful API in Ruby on Rails. Note : You need to apply this tiny patch to ActiveResource.
ActiveResource code:

class Product < ActiveResource::Base
  self.site = "http://username:password@quarkrank.com"

  def self.list options={:category=>"camera"}
    find(:all, :params=>options)
  end

  def self.show sku, site=nil, all_features=false
    params = {}
    params[:site] = site unless site.nil?
    params[:all_features]="true" if all_features
    find(sku, :params=>params)
  end

  def self.search query
    find(:all,:params=>{:search=>query})
  end

end

class Snippet < ActiveResource::Base
  self.site = "http://username:password@www.quarkrank.com/products/:product_id"

  def self.snippets product, name
    find name.gsub(" ","%20"),:params=>{:product_id=>product}
  end
end

Widget

QuarkRank provides two kinds of customizable widget.

Top 5 features
Interactive widget for features and review snippets of a product

More informatin at:

Technologies and tools used

A lot of them ....

HTML scrapping : NLP, Scrapi, Firequark, CSS Selectors. Implemented in Ruby.
Text-mining : Statistics, Text parsing, chunking, cleaning and many enhancements.
API and QuarkRank site : Ruby on Rails, REST, Acts_as_solr, Request routing.

Coming next

QuarkShop : a mashup of QuarkRank, Cnet, Amazon, Shopping.com and Yahoo.com.

Why I moved from Prototype to jQuery

2007-11-06T15:37:00Z

jQuery is a JavaScript library which follows unobtrusive paradigm for application development using JavaScript. jQuery inherently supports Behavior driven development and is based on traversing HTML documents using CSS Selectors. On the other hand, Prototype is a JavaScript library for Class driven development which makes life easier working with JavaScript. Prototype library has a good support in Ruby on Rails via helper functions.

I have always used Prototype library for most of my projects until I was introduced to jQuery three months back ... and it enchanted me.

The reasons

Behavior driven development (BDD)

Using jQuery, behavior(s) of HTML elements is defined separately from HTML code similar to defining style of an element using CSS. Lets look at a simple example to display alert box on click of an element

1234	$(element).click(function(){ alert("warning"); //fill stub for confirming this action from user});

A complex example: All elements of class "speciallinks" should emit the following behavior:

change their href link to "javascript:void(0);"
generate logs on click
onhover should change background color.

$("div.speciallinks").attr("href","javascript:void(0)")
    .click(function() {console.log("div.speciallinks clicked");})
    .hover(function(){$(this).addClass("hovered");}, 
             function(){$(this).removeClass("hovered");});

MVC + J
MVC framework divides web development into 3 separate parts Model-View-Controller and Ruby on Rails is a great MVC framework. The View part of MVC framework comprises of HTML, CSS and JavaScript which is bulky, combining three different parts of GUI development into one. Moreover, using Prototype helpers in Ruby on Rails views messes up HTML and JavaScript together. Here, jQuery fits in nicely (due to BDD) to separate JavaScript(J) from Views(V) to visualize the framework as MVC + J which I find very powerful especially working with Ajax.

Using jQuery, I keep all my HTML files are clean and clear as all the JavaScript code is kept in .js files defining behavior of HTML elements.
Chaining of actions
Chaining of actions for a HTML element follows DRY principals and increases readability of JavaScript code. If I want to add a bunch of operations on a single/multiple elements, it can be as simple as:
1234
$("div.message").show() .append("<p>Action has been executed successfully") .addClass("flash");// chained functions can be on separate lines :)
Now, this is possible because every method within jQuery returns the query object itself on which further methods can be applied to form a chain of methods on "selected" HTML elements.
CSS Selector rocks!
CSS Selectors are very powerful when playing with HTML DOM. jQuery is based on CSS Selectors to identify elements in a HTML document, which avoids tedious job of managing id attribute for each of my HTML tags. Most of id attributes can be avoided using right CSS Selector. Prototype does supports CSS Selectors via $$ function, but it doesn't fully leverages the power of CSS Selectors. I find Prototype working best with element's id attribute.
No more checks for presence of an element
In prototype, I always need to check if an element exists before applying an action to it. For example: I want to display user specific content in a div{id='user-box'} only if user is logged in (this div will exist on rendered page only if user is logged in). In Prototype I will do :
1234
if ($('user-box')!=null) { // this if block is redundant with jQuery $("user-box").style.backgroundColor = "red";}
Aids development process
Using jQuery HTML code is clean and nearly untouched. My web designer can easily modify html and stylesheets without learning the Prototype library.

Examples

Here are few examples highlighting the difference in using jQuery and Prototype. (code in the examples may not be the shortest way possible.)

jQuery	Prototype
Example1: Tabbing system with 3 tabs of class "tabs" and ids=>["tabs1","tabs2","tabs3"] respectively. When user clicks on the link "show first tab", #tab1 should only be displayed.
1 $("div.tabs").hide(); 2 $("div#tabs1").show();	1 $$("div.tabs").invoke("hide"); 2 //or 3 $$("div.tabs").each(function(x){ 4 Element.hide(x); 5 }); 6 // then.. 7 $("tabs1").show();
Example2: Modify CSS of some element
1 $("#user-box").css("background-color","red")	1 var a = $("user-box"); 2 a.style.cssText += 3 "background-color:red;";

For more examples checkout, Remy Sharp's presentation

I miss prototype

On Ajax request, when a div is updated with a partial and this partial contains elements with JavaScript behavior, the behavior needs to be activated/reactivated. In jQuery I do not have to write a lot of code for this, but still I have to keep this in mind every time a partial is loaded with Ajax. In Prototype, each HTML element use to contain the corresponding JavaScript code along with it in the partial.
Ruby on Rails helpers for Prototype library are awesome and makes life a lot easier working with JavaScript and Ajax.
Prototype library provides Ruby like syntax in JavaScript for functions on arrays, objects, etc which further makes development easier and faster.
With jQuery, element behavior gets activated only after DOM has been built and JavaScript files have been downloaded. For slow connections, this can be painful as there is a delay in activation of behavior.

Conclusion:

Jquery and prototype are both great libraries. For me, jQuery's philosophy (type less, do more, keeping things intuitive and unobstrusive) makes a big difference. But, I miss prototype core javascript additions. Waiting for jquery-rails integration. Some steps have been taken toward this goal. I would love to listen some of your personal experiences in shifting.

P.S: For using both libraries together, check this.

Sessions and cookies in Ruby on Rails

2007-10-21T17:06:00Z

An important issue rarely talked about with little documentation on Internet. So, here we go ... a guide to session and cookies in Rails. Session and cookies are an integral part of any good web application and rails has a good support for them. Continuing with our DRY approach, this guide contains link to cool articles with good description wherever necessary.

Introduction
Sessions
Cookies

Introduction

HTTP is a stateless protocol which creates problem in uniquely tracking a visitor to a web application. The process of managing the state between browser and server is through the use of session IDs which uniquely identifies a client browser.

Session IDs can be stored and communicated in one of the following ways :

Embedded in URL
In form field
Using cookies.

Information stored between multiple client browser request is called Session Data. Session data for each visitor can be stored at the server or in cookies. Upon client request to server, session data is extracted from session storage using session ID send by client browser. A good common example for session data is user information for authentication.

In the present times, its hard to imagine a good web application not using Sessions.

A wonderful article on implementation techniques of Session ID.

- Sessions -

Session in rails

Session in rails is a hash-like structure which allows you to store data across requests. Sessions can hold any kind of data object (with some limitations) because they store data using Data Marshalling (aka Data Serialization or Data Deflating).

Rails way of implementing session:

session_id is a 32 hex character MD5 hash based upon time, random number and constant string. It is stored in cookie at client browser. Rails provides transparent support for session_id.
Session storage discussed below.

Remember, you can insert or access values from session similar to hash ... but session is NOT a hash. Most of the other hash methods will not work with sessions.

Working with session in rails Data Serialization in Ruby CGI::Session creates a new instance of session everytime a new user visits your site.

Configure your sessions

Configure key, prefix, expiry and domain of your session. Switch on/off session at controller and action level. session_path and session_secure

Refer to next section on options for session storage.

Note: By default session_id is stored as key in cookies. For multiple rails application from same domain its a good practice to set 'session_key' to avoid conflicts.

Storage options

Ruby on Rails provides you with many session storage option.

PStore
ActiveRecordStore
CookieStore
DRbStore
FileStore
MemoryStore

CookieStore is available only in edge rails. PStore is the default option for stable release, whereas its CookieStore as default for edge rails.

Good description on session stores. Cookie-based session storage Improve performance : use SQLSessionStore instead of ActiveRecordStore Comparison of session storage option

Session storage limitations

Following objects cannot be stored in session storage:

Bindings
Singleton
I/O objects
Procedure objects

Do not store model objects in session

Change in model objects saved in session would also change table row corresponding to model object.
Breaks validations.
On change in class definition, model objects in session will go out of sync.
http://railscasts.com/episodes/13 http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/268624
A general belief is marshalling/unmarshalling of session on each request is expensive. This might be wrong - read Eric Hodel's comment at :
http://www.caboo.se/articles/2007/2/21/new-controversial-...-cookies#comment-1331

Incase you are storing model objects in session, upon change in class definition you will need to delete sessions and then restart your applications.

More limitations :

Store only user-specific data. Session data will become stale if other users can modify/update it. For e.g. if you store blog comments in session then session of active users need to be explicitly updated when new comment in made.
Critical information should be stored in database and not sessions as you might loose information if client cookie is lost/deleted. For e.g. user purchase information on a shopping site.
Session are not meant for storing massive objects or tons of information, use your application database instead.

Session and Security

A detailed look into security issues concerning sessions. Lookout into sections named as 'Session Hijacking' and 'Common Failings' Minimize session attacks Using XSS attacks a hacker can steal user's session-id ... be careful.

HowTo

Implement session expiration

A good description A concise version

Delete stale sessions

Every session created in session storage (ActiveReocord, PStore ...) is not deleted upon session expiration or browser close by client. Which means you will have to run a cron job to delete old sessions else your session storage will shoot up in size.

As a good practice disable sessions for those part of your web application which does not require sessions. This will avoid creation and storing unnecessary sessions in your storage.

Find out active users

If you want to find no. of active users, simply use updated_at column of sessions table.

Access session data using session_id

This might be helpful if another application in same domain wants to access your session.

Miscellaneous

Use model method to access row in sessions table corresponding to current session. For e.g. session.model.id or session.model.updated_at
Smart plugin for better session experience in rails specially for session expiration.
http://agilewebdevelopment.com/plugins/limited_sessions
Flash messages to communicate between actions are stored in sessions. So, if you switch off your sessions, flash messages will stop working.
Do not let bots make unnecessary sessions
http://jroller.com/obie/entry/wrestling_with_the_bots http://gurge.com/blog/2007/01/08/turn-off-rails-sessions-for-robots/
Do not write unchanged sessions back to database -- improves performance
http://pastie.caboo.se/53086
Non-cookie session :
If the client has cookies disabled, the session_id must be included as a parameter of all requests sent by the client to the server. The CGI::Session class will transparently add the session_id as a hidden input field to all forms generated. No built-in support is provided for other mechanisms, such as URL re-writing.
If you care about browsers which do not support cookies, checkout the following plugins (disclaimer : I have never used these plugins :P )
Hidden Field Session : http://agilewebdevelopment.com/plugins/hidden_field_session
No cookie session support : http://www.edgesoft.ca/blog/read/2

Beware, session information in URL might be dangerous. Someone might post a link to a product on a board and everyone following this link is logged with all user data available.
PStore in Windows : Marshal Data Too Short Error
http://wiki.rubyonrails.org/rails/pages/MarshalDataTooShort

- Cookies -

Cookie on rails

Cookies are stored at client browser and is sent back to server on each request. Rails provides a hash-like structure ActionController#cookies in controller to manage cookies. Session in rails is implemented using cookies.

Description, options and example Note :

Only string can be stored in cookies.
session_id is by default stored in cookies at client browser.

cookies vs. request.cookies

Both cookies and request.cookies are used to access cookie information in your rails application. They are very different in their behavior which can be daunting for beginners. Examples below explains the basic difference between them.

## cookies and request.cookies are different
cookies.class #=> ActionController::CookieJar
request.cookies #=> Hash

## how to access value from cookies and request.cookies
# First set some value in cookies
cookies[:key] = "value"

cookies["key"] #=> "value"
cookies[:key] #=> "value"
# both the output are of type String

request.cookies["key"] #=> "value"
request.cookies["key"].class #=> CGI::Cookie

request.cookies[:key].empty? #=> true
request.cookies[:key].class #=> Array

CookieJar

cookies in rails is of type CookieJar. CookieJar manages incoming and outgoing cookie information and works as follows.

def index
  cookies[:key] = "val"
  puts cookies[:key]
  redirect_to :second_index
end

def second_index
  cookies[:key] = "newval"
  puts cookies[:key]
end

## Output :
## Open index page for the first time
# nil
# val

## Open index page the second time
# newval
# val

So,
cookies[] gives you value from the incoming cookie.
cookies[]= sets value in the outgoing cookie.

Reference :

Miscellaneous

Checkout cookie based session store explained above.

Testing cookies Open ticket - making CookieJar behave like a Hash. Performance and cookies : a good read

Ruby on Rails Security Guide

2007-09-20T05:02:00Z

Ruby on Rails does a decent job in handling security concerns in the background. You will have to configure your application to avoid few security attacks while plugins would be required for many security concerns which are not at all or poorly managed by rails.

In this article I have described the security issues related to a ruby on rails web application. I have followed DRY by linking to articles with good explanation and solutions to security concerns wherever required. This guide can also be used as a quick security check for your current web application.

Authentication
Model
Controller
View
Miscellaneous
Security plugins directory

Authentication

Authentication is the foremost requirement of most of the web applications to authenticate and give privileges to their users. Apart from normal authentication mechanism rails have plugins for OpenID, CAS and Access Control. Build your own authentication system only if your requirements are very unique or you do not trust other implementations.

Plugin - Restful Authentication (recommended) - easy to use and you can tweak it according to your requirements. Build your own authentication. You should rarely need to do this ... Restful Authentication is quite flexible. OpenID - a universal authentication system to avoid use of multiple username and password on the Internet. OpenID is getting quite famous now-a-days. Access Control : To easily proivde different priviliges to your users. There are a lot of cool plugins available for access control. Centralized Authentication Server - is used to implement single login/password for your users across multiple application. It can also be used for a single sign-on system. For example, Gmail and Google Reader have a single sign-on between them. Use Google Authentication API to let your users login using their google username and password. More Plugins :

- Model -

SQL Injection

The problem arises when metacharacters are injected into your queries to database. Rails has a very good support to avoid SQL injection if you follow conventions in issuing queries to your database.

Description : Alternate Solution - use hash for specifying conditions in #find

Activerecord Validation

To validate the contents of model object before records are created/modified in the database. Activerecord validations are very useful over database data-type constraints to ensure values entered into the database follow your rules. You might have javascript validations for forms but javascript can easily be switched off. Use javascript validations only for better user experience.

Description : Conditional validation using :on and :if options. Checkout this cool video Be careful using validates_uniqueness_of, it has problems when used with :scope option. Open bug tickets : Use :allow_blank to pass validations if value is nil or empty string Testing Validations - do read the comments in this article Useful Tips

Its easy to manage 'nil' values using :allow_nil, its quite handy. For ex: set :allow_nil => true in validates_uniqueness_of to check uniqueness of non-nil values and ignore nil values
validates_presence_of is not required if you are using validates_format_of, unless regular expression accepts empty string.

Creating records directly from parameters

While creating database records directly from form params, a malicious user can add extra fields into the params and manually submit the web page which will set values of fields which you do not want user to set.

Description : Alternate Solution - Trim the parameters to keep the required keys and remove the others.

- Controller -

Exposing methods

Use private and protected in controller for methods which should not be actions. Actions are pubic methods and can be invoked from the browser.

hide_action : If non-action controller methods must be public, hide them using hide_action. Be careful of bypassing private and protected using meta-programming

Authorize parameters

Always authorize user request. By tweaking form parameters or url a user can send request to view/modify other users information if there is no proper authorization of parameters.

For example :

## To find information of an order which belongs to a particular user.

#Incorrect :
@order = Order.find(order_id)

#Correct :
@order = @user.orders.find(order_id)

Do not ignore hidden fields - a user can easily modify their value, so suspect them similar to params[:id]

Filter sensitive logs

Prevent logs of sensitive unencrypted data using #filter_parameter_logging in controller. The default behavior is to log request parameters in production as well as development environment, and you would not like logging of password, credit card number, etc.

Video Tutorial

Cross Site Reference(or Request) Forgery (CSRF)

In a CSRF attack, the attacker makes victim click on a link of his choice which would contain a GET/POST request and causes web application to take malicious action. The link could be embedded in a iframe or an img tag. Its recommended to use secret token while communicating with user to avoid this attack.

Its little complex to understand this attack. So, only those readers who are very enthusiastic to know about it, please read the Description below. Rest can directly move ahead to use the plugin.

Description : Use Get and Post appropiately (note : Both get and post are vulnerable to CSRF) Example - Gmail CSRF security flaw Plugin - CSRF Killer (recommended) - it requires edge rails

Minimize session attacks

If an attacker has session-id of your user, he can create HTTP requests to access user account. An attacker can get session-id by direct access to user machine or is able to successfully run malicious scripts at user machine. In this section we will talk about how to avoid or minimize the risk if attacker has user session-id. Following steps are helpful:

Store IP Address, but creates problem if user moves from one network to another.
Create a new session everytime someone logs in.
Expire session on user logout, user is idle for a time period or on closing of browser/tab. For maximum security expire sessions on all the three conditions.

Code for session expiry on timeout

## Timeout after inactivity of one hour.
MAX_SESSION_PERIOD = 3600

before_filter :session_expiry

def session_expiry
   reset_session if session[:expiry_time] and session[:expiry_time] < Time.now

   session[:expiry_time] = MAX_SESSION_PERIOD.seconds.from_now
   return true
end

Plugin - Session Expiration for session expiry on timeout Do not put expiry time in the cookie unless your cookie information is properly encrypted. If not, use server side session expiry. Persistent session / login in rails - global setting in enviornment.rb

ActionController::Base.session_options[:session_expires] = <i>say after two years</i>

Persistent session / login in rails - to give your users a feature - remember me

Stop spam on your website from DNS Blacklist

Avoid access to your website from IP addresses which are present in DNS Blacklist(DNSBL).

Plugin - DNSBL check

Caching authenticated pages

Page caching does bypass any security filters in your application. So avoid caching authenticated pages and use action or fragment caching instead.

- View -

Cross site scripting(XSS) attack

Cross Site Scripting is a technique found in web applications which allow code injection by malicious web users into the web pages viewed by other users. An attacker can steal login of your user by stealing his cookie. The most common method of attack is to place javascript code on a website that can receive the session cookie. To avoid the attack, escape HTML meta characters which will avoid execution of malicious Javascript code. Ruby on Rails has inbuilt methods like escape_html() (h()), url_encode(), sanatize(), etc to escape HTML meta characters.

Description Can we avoid tedious use of h() in views? Sanitize() is used to escape script tags and other malicious content other than html tags. Avoid using it ... its unsecure. Use white_list instead. White_list plugin

Anti-spam form protection

Use Captcha or Javascript based form protection techniques to ensure only human can submit forms successfully.

When using Captcha do ensure the following :

Images are rendered on webpage using send_data and are not stored at the server, because its not required to store images and are redundant.
Avoid using algorithm used by standard Catpcha plugins as they can easily be hacked, instead tweak an existing algorithm or write your own.
Use a Captcha which does not store secret code or images in filesystem, as you will have trouble using Captcha with multiple servers.

Tutorial - a nice article on concepts of captcha Plugin - ReCaptcha (recommended) Plugin - BrainBuster - a logic captcha based on simple puzzles, math and word problems. By default, it has limited set of problems and you would have to come up with large set of your own problems. Plugin - Simple Captcha (not recommended) as it breaks all the must have features of a good Captcha implementation. For less critical systems like blogs, a more user-friendly option can be use of CSS based technique or JavaScript based plugin unlike Captcha. Both JavaScript and CSS based techniques can only avoid spam from dumb or general bots. If an hacker specifically targets your site or bot is smart enough, you are dead, so be careful. Captcha with Multiple Servers

Hide mailto links

Mailto links in a webpage can be attacked by e-mail harvesting bots. Use the plugin CipherMail to generate a 1024 bit random key and obfuscate the mailto link.

Plugin - CipherMail

Use password strength evaluators

A lot of people have used password strength evaluators simply because its used by google in their registration form. You can use it to help your users register with strong password. But I don't think its a must have security addon. Uptill now I have not found a good algorithm to assess strength of a password, but some of them are reasonable.

Also, if there is an open source tool or algorithm for evaluating password strength, it can easily be broken. So, you might consider tweaking the algorithm or building one from scratch.

Tools

- Miscellaneous -

Transmission of Sensitive information

Use SSL to encrypt sensitive data between transfer from client to server. SSL hits server performace, so you might consider using SSL only for few pages which transfer sensitive data to and fro.

Plugin ssl_requirement Mongrel, rails, apache and SSL Controller in SSL subdomain Sample SSL code in rails

File upload

Be very careful when you allow your users to upload files and make them available for other users to download.

Description Must read - Section 26.7 of Agile web development with rails - 2nd edition In place file upload 3 plugins for file upload reviewed at :

Secure your setup / environment

Proper Mysql configuration

Use good passwords

Security plugins directory

Note : I will keep this security guide updated. Any additions/improvements are welcome.

Advanced acts_as_solr

2007-09-14T04:15:00Z

This article extends our acts_as_solr : search and faceting tutorial and talks about how to manage rails associations, solr indexes and more with acts_as_solr.

Rebuild Solr index
Import existing Solr index or your custom Solr schema.xml
Highlight search term
Rails associations and acts_as_solr
Tips

Rebuild Solr Index

rebuild_solr_index is a class method to re-build your model indexes on import of external data.

For large tables rebuilding Solr index might be time consuming process because each row is comitted to solr index. Rather saving each row individually, you can save them in batch. Using batch_size argument, in #rebuild_solr_index(say for example 100), solr index would be updated with 100 rows in 1 commit rather updating solr index for each of them.

If you do not want to index the complete data but only few rows based on conditions, use optional arguments batch_size and finder in rebuild_solr_index.

batch_size : to send this many rows to solr for index update. Default value is 1.
finder : it is used for conditional indexation. It takes a method as argument which returns objects to be indexed. Default method is :
123
def finder(ar, options) ar.find (:all, options.merge({:order => self.primary_key}))end

Import existing Solr index or your custom Solr schema.xml

Assumption: Column names in table definition are same as the names of the fields using which index has been/will be created.

Add your indexed fields to vendor/plugins/acts_as_solr/solr/solr/conf/schema.xml (continuing with example of our previous tutorial, we add name1, brand, resolution to schema.xml).
Remember, to copy them to text field using solr's copyfield directive. text is the default field/column in which search is made, so using copyField we add all data to default search field. The new schema.xml looks like...

<field name="text" type="text" indexed="true" stored="true" multiValued="true"/> 
<field name="name1" type="text" indexed="true" stored="false" />
<field name="brand" type="string" indexed="true" stored="false" />
<field name="resolution" type="sfloat" indexed="true" stored="false" />

<copyField source="*_facet" dest="text"/>
<copyField source="brand" dest="text"/>
<copyField source="name1" dest="text"/>

Apply our patch (download). For indexing and searching, acts_as_solr suffixes dataType (e.g. int, float, string etc) of column to the column name, this patch tells acts_as_solr not to do this since we have explicitly created entries for these columns in schema file:
12
cd /path/to/rails/dirpatch -p0 custom-schema.patch

Modify your acts_as_solr definition:

123	class Camera acts_as_solr :custom_schema=>true, :fields=>[:name,:brand,:resolution]end

... and it works!

Highlight search terms

Goal is to highlight search term(s) in search results. Follow the steps below :

Modify solrConfig.xml to enable highlighting. Apply our patch (download) as:
12
cd /path/to/rails/dirpatch -p0 highlight-solrconfig.patch
acts_as_solr does not stores any of the field values as it is in its index. We need to modify the configuration file vendor/plugins/acts_as_solr/solr/solr/conf/schema.xml to enable storage of data, which is required to return the highlighted text data. For example, we modify configuration file to store the fields having text dataType to get highlighted matching terms for name1 field.
1234
<dynamicField name="*_t" type="text" indexed="true" stored="false"/><dynamicField name="*_t" type="text" indexed="true" stored="true"/>
Apply one more patch in the same way as the previous one. This patch enables acts_as_solr to handle the option of highlighting in find_by_solr queries and parses the results returned by solr. And now we are ready to roar...

Lets see it in action with an example

@results = Camera.find_by_solr("canon powershot", :highlight=>{:fields=>"name1"})
@results.highlights
## returns hash of objects id and hash of the columns with matched text data
{1=>{:name1=>"<em>Canon</em> <em>Powershot</em> sd1000"} ... }

#You want to display names with highlighted terms:
@results.docs.each do |doc|
   <%= @results.highlights[doc.id][:name1]  -%><br/>
end

Options for highlighting (You can also refer them here)
1. fields: columns to highlight terms for (default is none, and no highlighted text)
2. prefix: You do not want to have "<em>" tag around matched search term(s), tell your tag here. (eg. "<span class='highlight'>")
3. suffix: Ending tag for matched term(s)
4. max_snippets: stop searching the column after max_snippets matched terms found
Currenlty acts_as_solr does not implements these options at each column level, so you cannot ask to apply some of the options to only one (or few) columns. For e.g. you cannot ask acts_as_solr to use <em> tags for one field and <span class='highlight'> for other field.

Note: If you want both the functionalities Custom schema and Highlighting together, use these two patches. Patch 1 and Patch 2.

Rails associations and acts_as_solr

Lets say I have two models, User (columns: username, password) and UserProfile (columns: full name, location, favorite food .... and many more). These two models have one-to-one association (:has_one, :belongs_to) between them and I have two problems :

When searching the User model, I also want to search UserProfile model.
I want to allow faceting based on few columns in UserProfile lets say location and favorite food.

Fabio Confalonieri has written a small hack for the solution to both the problems along with good explanation.

Tips:

Wanna see the statistics for created index?
Download this xsl file as luke.xsl to vendor/plugins/acts_as_solr/solr/solr/conf/xslt/ and then open url: http://localhost:8982/solr/admin/luke?wt=xslt&tr=luke.xsl
If your search results are not good enough because the text is not indexed properly, it might be due to presence of acronyms or words having apostrophes or html content in your data. Solr provides you with many options for analyzing and modifying text before it is indexed and searched. Read these options at Solr's Wiki page. Remember, you need to make these modifications in schema.xml file.
Note : The default behaviour of acts_as_solr is to index model objects automatically upon save or update of a record.

Firequark : quick html screen scraping

2007-09-05T19:42:00Z

Introduction
Why Firequark?
Usage - screencast
Installation
Documentation
Todo

Firequark is an extension to Firebug to aid the process of HTML Screen Scraping. Firequark automatically extracts css selector for a single or multiple html node(s) from a web page using Firebug (a web development plugin for Firefox). The css selector generated can be given as an input to html screen scrapers like Scrapi to extract information. Firequark is built to unleash the power of css selector for use of html screen scraping.

HTML screen scraping is a common technique of extracting information about specific and useful elements from a web page. Independent of programming language, for extracting an element from a web page one need to know its exact location or a key to uniquely identify the element. There are two approaches for uniquely identifying an element: using XPath or CSS Selectors.

Firebug has an inbuilt functionality of generating XPath for an html element. Ilya Grigorik has written a good article on using XPath for HTML screen scraping. Whereas, Firequark extends Firebug for generating CSS Selector for elements on a web page.

Example case : Lets take a practical example where you want to scrape Amazon.com. My goal is to get product name, price and rating for all the products from the Amazon point-and-shoot camera catalog page. I will use this example in screencast and explanation below.

XPath vs. CSS Selector
When XPath is already provided by Firebug then why do I need CSS Selector? Xpath is great for scaping XML documents but for (x)html documents, it runs into many issues like :

Various parsers generate different xpath for the same element depending on their handling of broken markup with badly nested tags, errors in html pages, custom tags etc.
Firefox adds <tbody> tag in table nodes, independent of whether html page has tbody tag or not, which makes it difficult to figure out to keep <tbody> tag or not.

CSS Selector does not suffer from these problems because css selector of an html node is based on properties of self and neighboring nodes. Attributes of element and its neighbors like id, class, etc are used to find css selector for an element.

Find CSS Selector manually

Its difficult to find css selector for a html node. Its a trial and error method until you find a right combination of rule (css selector). In the worst case the css selector would be the xpath itself. In our example case, its more difficult to manually find one unique css selector for 24 camera products and one each for their attributes like camera name, price, rating, etc (4 in total).

Bundle Scraping

I am big fan of Scrapi, a html scraping toolkit in Ruby because it supports bundle scraping. Bundle scraping refers to extraction of multiple attributes of an object from a web page in one parse. Bundle scraping is well-defined at Qscraper: a hpricot interface to scrapi.

Continuing with the example case, object is a camera product (there are 24 such objects on the page) and price, rating, product name, etc are attributes of camera object. One way of extracting attributes is to separately get list of product name, price and rating from the web page and then combine these list. But, how do you combine them? What if one of the camera product is not rated or amazon does not provide its price?

Firequark is really powerful in solving this problem. Using Firequark, first get one css selector to identify all the camera objects on the page which contain the attribute information inside them (name, price and rating). Set camera object to parent and find css selector of attributes relative to the parent. Give these css selectors as an input to html screen scraper supporting bundle scraping like Scrapi and bingo! Our screencast below explains this case in detail.

Click here to view the screencast [format:avi, size:6.6MB]

Note : Avoid using CSS Selector functionality on first 2-3 products on a page because top 2-3 products are usually displayed differently like top sellers or top rated with same id which causes problems in getting good & simple css selectors.

Even in our screencast we analyze the 4th product on the page to get a simple css selector

Click here to install, this will overwrite your current Firebug installation (built on Firebug v1.05).

(FF3 users) Click here to install, this will overwrite your current Firebug installation (build on firebug 1.2).

Note: in case you are doing firebug development over 1.05 then please backup your work before installing Firequark.

Firequark adds four new functions to each node element in html source tab of Firebug. In html source tab of Firebug, when you click on an html node:

Current menu	New menu

Get U CSS Selector: Get css selector for the selected element which uniquely identifies that element.
Get CSS Selector: Get one css selector for group of elements on a web page. When you select this option, it will prompt you for how many such elements exist on this page. In our example case, right click on camera object node, in the prompt box put 24, as there are 24 camera objects on the page, press enter and you will get one unique css selector to extract 24 camera objects.
Mark as parent: Used in bundle scraping to mark an object as parent. It is followed by Get U CSS Selector for attributes of parent, which will return you with css selector of attributes relative to parent object.
Unmark parentnode: To unmark object as parent.

The current algorithm to find css selector is not sophisticated but solves my purpose very well.
Firquark is based on CSS Selector 2. We want to move to CSS Selector 3 because it has more selectors.

HTML::Tag class in scrapi

2007-08-28T16:06:00Z

While working with scrapi, I found there is no external documentation for HTML::Tag class. This article is to ensure no one says this again.

In scrapi, HTML::Node represents a html node which can be of 2 types: HTML::Text and HTML::Tag for a text node and html tag node respectively. Here is a code snippet in which scrapi returns the html node as a HTML::Tag object.

require 'rubygems'
require 'scrapi'
require 'open-uri'

scraper = (Scraper.define do
   process_first "span.metaspan", :node=>:element
   result :node
end).scrape(open("http://www.quarkruby.com"){|f| f.read})
# scraper.class
=> "HTML::Tag"

METHOD	DESCRIPTION
tag?	true if this is HTML::Tag . HTML::Tag and HTML::Text have many different methods so its an important check.
name	returns name of the tag
attributes	returns hash of attributes name/value pairs
children	returns array of children nodes, which may contain both HTML::Text and HTML::Tag class objects
next_sibling	returns next adjacent node, which can be HTML::Text or HTML::Tag
next_element	returns next HTML::Tag node
to_s	returns html content of this node and its children
find	match on subtree of this node (not root node).
position	byte index of matched node in html content
line	line number of matched node in html content
parent	returns parent HTML::Tag
match	matches the given node with given arguments (explained with examples below)
Next two methods are not a part of HTML::Tag class but I frequently find them useful. Their code is available below
inner_html	javascript's innerHTML :)
text	returns the text content for this node and its childrens

Some examples on how to use match

#test if the node is a "span" tag
  node.match :tag => "span"

#test if the node's parent is a "div"
  node.match :parent => { :tag => "div" }
#test if any of the node's ancestors are "table" tags
  node.match :ancestor => { :tag => "table" }

#test if any of the node's immediate children are "em" tags
  node.match :child => { :tag => "em" }
  
#test if any of the node's descendants are "strong" tags
  node.match :descendant => { :tag => "strong" }

#test if the node has between 2 and 4 span tags as immediate children
  node.match :children => { :count => 2..4, :only => { :tag => "span" } }

#get funky: test to see if the node is a "div", has a "ul" ancestor
#and an "li" parent (with "class" = "enum"), and whether or not it has
#a "span" descendant that contains # text matching /hello world/:
  node.match :tag => "div",
             :ancestor => { :tag => "ul" },
             :parent => { :tag => "li",
             :attributes => { :class => "enum" } },
             :descendant => { :tag => "span",
             :child => /hello world/ }

# Sometimes, you can get a HTML::Text node in return
# if its a text node, following are ways of using match on that node.
  node.match (conditions)
# * if +conditions+ is a string, it must be a substring of the node's  content
# * if +conditions+ is a regular expression, it must match the node's content
# * if +conditions+ is a hash, it must contain a <tt>:content</tt> key that
#   is either a string or a regexp, and which is interpreted as described above.

(above examples are shamelessly copied from html/node.rb, scrapi)

module HTML
   class Tag < Node
      def text
         return @content if respond_to? :content
         data=""
         0.upto(children.size-1) {|i|
             next if children[i].respond_to?:name and children[i].name=="script"
             data += children[i].text
         }
         return data
      end
     
      def inner_html 
         data = ""
         children.each {|e| data += e.to_s}
         data
       end
   end
end

Assaf Arkin has written a well-documented cheetsheet on scrapi. Surprisingly, I found few selectors left unexplained..

E,F : i.e. alternate selectors (matching any of E or F)
E:content('hello world'): content match
E[foo=bar][foo2=bar2]: element E with attribute foo and foo2 having values bar and bar2 respectively

Domain Forwarding or URL Redirection

2007-08-25T09:45:00Z

also known as URL Forwarding or Domain Redirection. Its a technique of making webpage available through many URLs.

Checkout wikipedia article on URL redirection for uses of redirection.

In Short,

Client Side Fowarding : URL in client browser changes.
Server Side Redirection : URL in client browser does NOT change. User remains on same website/domain.
Server Side Forwarding or DNS Forwarding : URL in client browser does NOT change. User is moved to NEW website.

All the above methods are explained below in detail. I will be using Ruby on Rails for illustration.

Client-Side Redirection

The server sends a 301 or 302 redirect response with a Location: header containing the new URL. The client browser makes another request to the new URL. This is the most common form of domain forwarding.

302 is for temporary redirection. It is used when the requested resource resides temporarily under a different URL. For Example: Latest article at my site are available at www.mysite.com/latest/ which would temporarily redirect to www.mysite/article/356.html and then to www.mysite/article/357.html

# 302 is default response status of redirect_to
# in controller latest
def index
  redirect_to "article/356"
end


# or redirect to another website
def index
redirect_to "http://www.newsite.com"
end

301 is used for permanent redirection which means that the resource has moved permanently and people should update their information for this URL.

1234	def index headers["Status"] = "301 Moved Permanently" redirect_to "http://www.newsite.com"end

Server side Redirection
The server transparently rewrites the request URL to another URL. User remains on the same website and the URL on the client browser does not change.

##<strong>Two ways</strong>
# 1. Use routes.rb
# in routes.rb write following to redirect incoming request to 'your_controller/create_new'
map.connect 'my_controller/create', :controller => 'your_controller', :action => 'create_new' 

# 2. render :action
# in my_controller
def create
  render :controller => 'your_controller', :action => 'create_new'
end

Server side Forwarding

The server transparently rewrites the requrest URL to another URL. The user is moved to a NEW website and the URL on the client browser does not change. Also known as url masking or url cloaking or url shadowing. The most common method is by Frame Redirects

Note : Apache "core" does NOT perform ANY sort of redirection. Add on modules like mod_redirect to get Client-Side Redirection, mod_rewrite to get Server-Side Redirection, and mod_proxy to get Server-Side Forwarding. So, the ability to redirect is NOT built-in; the modules happen to be in common distros and configuration is baked into httpd.config and .htaccess.

(shamelessly copied from Domain Forwarding in IIS)

DNS Forwarding
The output is exactly similar to Server side Forwarding but approach is different. The user is moved to NEW website and the URL on the client browser does not change. You need to set appropiate CNAME entry in your domainname record at your DNS server. Many hosting service providers or registrars provide you with this service by the name 'Domainname forwarding'. If you have your own DNS server, you can easily set CNAME entry. For example :

; my website is www.mysite.com. I want to redirect blog.mysite.com to www.myblog.com
; I will add following record in the DNS setting of www.mysite.com
blog       IN      CNAME  www.myblog.com.
; NOTE : do not forget to put '.' in the end

For detailed information about CNAME, please refer to Canonical Name Record (CNAME)

QScraper : hpricot interface to scrapi

2007-08-23T08:28:00Z

QScraper is a wrapper over scrapi to provide Hpricot like interface.

Motivation: Hpricot interface is simple and easy to use while scrapi is more powerful because of bundle scraping and anonymous classes. I was using hpricot for quick testing and checking but scrapi for project implementation. To avoid working with two html scrapers, I wrote this wrapper over scrapi.

Bundle Scraping: It refers to extraction of multiple attributes of an element from a web page in a single parse. Most screen scraping tools extract only multiple elements but not multiple attributes of an element. Lets take an example of blog scraping, each blog post would be an element and I would like to extract multiple attributes of blog post like info about author, published on, title and content. Rather than making individual calls like doc.search(author_selector), doc.search(published_selector) etc., I would like to do doc.find(author_selector, date_selector, title ...).

require 'rubygems'
require 'scrapi'
require 'open-uri'

class QScraper
  attr_accessor :document
  attr_accessor :rule_list

  # source can be a node or file
  # Constructor for this class ..
  # Takes a source (file or html::node) as argument.
  # Example : qScraper = QScraper.new("index.html")
  def initialize source, local_file=true
    case source
      when URI, HTML::Node
        @document = source
      when String
        @document = (local_file && File.exists?(source)) ?  File.read(source) : source
      else
        raise ArgumentError, "Can only scrape URI, String or HTML::Node"
    end
  end

  # search string selector in given source and 
  # returns all the matching elements as HTML::Tag object
  # Selector is a css selector 
  # Example:
  # qScraper.search("div.highlight")
  def search selector
    output = (Scraper.define do
      array :elems
      process selector, :elems =>:element
      result :elems
    end).scrape(@document)
    output
  end

  # search the first node matching this selector
  def search_first selector
    output = (Scraper.define do
      process_first selector, :elem=>:element
      result :elem
    end).scrape(@document)
    output
  end

  # Adds a rule for parsing, you can add many rules and 
  # then parse them by making a call to parse_rules
  # Example:
  # q.add_rule({"br#lgpd"=>{:n1ame=>"@id"}})
  def add_rule selector, extractors
    @rule_list ||= {}
    @rule_list.merge!({selector=>extractors})
  end

  # Example:
  # q.add_rules({"table.searchResults2>tbody>tr>td:nth-of-type(3)>a"=>
  #                               {:href=>"@href",:name=>:text},
  #              "table.searchResults2>tr>td:nth-of-type(3)>a.cat"=>
  #                               {:reviewcount=>:text}})
  def add_rules selector_extractors_hash
    @rule_list ||= {}
    @rule_list.merge! (selector_extractors_hash)
  end

  # parses the given rule set .. specified by add_rule methods.
  def parse_rules parser=nil
    result_string = "result "
    @rule_list.values.each do |j| 
        j.keys.each {|i| 
           result_string << ":" << i.to_s << ", "
        }
    end
    result_string = result_string[0..-3]
    array_string = result_string.sub(/result/,"array")
    create_process_string = ""
    @rule_list.keys.each {|i|
      create_process_string << "process \"" << i << "\", " 
      create_process_string << @rule_list[i].inspect.gsub(/[{}]/,"") << "\n"
    }
    res = String.module_eval %{Scraper.define do
      #{array_string}
      #{create_process_string}
      #{result_string}
    end}
    parser ||= :tidy
    f = res.scrape(@document,:parser=>parser)
  end
end

# Small extension to scrapi's HTML::Tag class 
# for "inner_html" method.
#
module HTML
  class Tag < Node
    def inner_html 
      data = ""
      children.each {|e| data += e.to_s}
      data
    end
  end
end

  # Example page to parse : http://www.google.co.in/search?q=scrapi
  doc = QScraper.new(open("http://www.google.co.in/search?q=scrapi"){|f| f.read})

  # Extracting titles elements of search results 
  # returned objects are of HTML::Tag type
  result = doc.search("a.l")

  # now extracting title and link for each of the search results.
  doc.add_rules({"a.l"=>{:url=>"@href",:title=>:text}})
  result = doc.parse_rules
  #=> result.url returns an array of 10 url links
  #=> result.title returns the array of titles

Hpricot	Scrapi (with Qscraper)
doc=Hpricot(string/file/uri)	doc=QScraper(string/file/uri)
doc.search("css selector") [or doc/"css selector"]	doc.search("css selector")
doc.at("css selector")	doc.search_first("css selector")
(doc/"css selector").inner_html	doc.search_first("css selector").inner_html
(doc/"css selector").to_html	doc.search_first("css selector").to_s
allows html page editing	--N/A--
--N/A--	Allows bundled scraping (example below)

TODO: Add XPath support in scrapi.

QuarkRuby - Home

Consume non rails-style REST API's

Table of Contents

Introduction

Consume non rails-style REST API

Creating URL for remote resources

Make a GET request

Handling (Custom) Response

Parse Response

Create ActiveResource object from parsed response

Other things to keep in mind

Custom HTTP GET method tweaks

find method makes a call to get "id"

"get" method doesn't converts hash into objects.

Data Format

Active Youtube

Gem Installation:

Example Usage:

Scrapi enhancements

ActiveResource and YouTube

ActiveResource extension

Example usage

Writing a web widget

Table of Contents

Before you start

Flash widget

HTML widget

JavaScript widget

Examples

Key issues

Quarkshop : next-generation shopping

Search products

Comparison : QuarkGraph

Features & Snippets : Voice of consumer

Feedback

introducing QuarkRank

About QuarkRank

QuarkRank provides

Where can you use it?

API

Widget

Technologies and tools used

Coming next

Why I moved from Prototype to jQuery

The reasons

Examples

I miss prototype

Conclusion:

Sessions and cookies in Ruby on Rails

Table of Contents

Introduction

- Sessions -

Session in rails

Configure your sessions

Storage options

Session storage limitations

Session and Security

HowTo

Implement session expiration

Delete stale sessions

Find out active users

Access session data using session_id

Miscellaneous

- Cookies -

Cookie on rails

cookies vs. request.cookies

CookieJar

Miscellaneous

Ruby on Rails Security Guide

Table of Contents

Authentication

- Model -

SQL Injection

Activerecord Validation

Creating records directly from parameters

- Controller -

Exposing methods

Authorize parameters

Filter sensitive logs

Cross Site Reference(or Request) Forgery (CSRF)