Due to update-to-date JMeter 3.0 require Java 7, but for current offical Java support in Mac OSX is 6. So, before running new JMeter you might need to upgrade your Java. Please check my gist for installation Java 7.

This article just show you code example of how to use loop number and thread number in gem 'ruby-jmeter'.

Currently, we need to do stress test on your API service.

That’s see how to write codes in different scenarios.

Reference: http://jmeter.apache.org/usermanual/functions.html

[Scenario 1] You want to change request data based on thread number

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

test do
  defaults domain: 'YOUR.API.ENDPOINT', portocal: 'https' , port: '443'
  
  header [
    { name: 'Authorization', value: "Bearer 123456789" },
    { name: 'Content-Type', value: 'application/json' }
  ]

  post_body = {
    "order_number": "thread-${__threadNum}"
  }
 
  threads count: 1000, rampup: 1 ,loops: 10, scheduler: false do
      transaction "Shipments" do
        post name: 'Create',
             url: 'https://YOUR.API.ENDPOINT/v1/orders',
             raw_body: post_body.to_json do
               with_xhr
        end
      end
    view_results_tree
  end

  view_results_in_table
  graph_results
  aggregate_graph
  view_results_tree
  summary_report
end.run(
  gui: true,
  file: 'jmeter-order-api.jmx',
  log:  'jmeter-order-api.log',
  jtl:  'results-order-api.jtl'
)

[Scenario 2] You want to change request data based on thread number and iteration number

1
2
3
4
5
6
7
8

test do
  ...
  
  post_body = {
    "order_number": "thread-${__threadNum}-loop-${__BeanShell(vars.getIteration();,)}"
  }
  
  ...

[Scenario 3] You already have data array and you want to use the array element as request data and use iteration number as index to get data

use variables to set variable array

variables accept [{name: XXX, value: XXX}, {name: XXX, value: XXX}...]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

require 'pg'

test do
  ...
  
  conn = PG.connect(dbname:   'DATABASE_STAGING_DBNAME',
                    host:     'DATABASE_STAGING_HOST',
                    user:     'DATABASE_STAGING_USER',
                    password: 'DATABASE_STAGING_PWD'
                   )
  orders = []
  conn.exec( "SELECT * FROM orders where order_state = 'ready'" ) do |result|
    result.each do |row|
      orders << row
    end
  end

  order_index = 0
  post_body_array = orders.inject([]) do |arr, order|
    arr << {
      name: order_index.to_s,
      value: {
        orders: [
          {
            id: order['id'],
            order_state: 'shipped'
          }
        ]
      }.to_json
    }
    order_index += 1
    arr
  end

  variables post_body_array

  threads count: 1, rampup: 1, loops: order_index do
    transaction "Update Order" do
        post name: 'Update',
             url: 'https://YOUR.API.ENDPOINT/v1/orders/update_url',
             raw_body: '${__BeanShell(vars.get("${__BeanShell(vars.getIteration();,)}"))}' do
               with_xhr
             end
    end
    view_results_tree
    debug_sampler
  end
  
  ...

[Scenario 4] You already have data array and you want to use the array element as request data and use thread number as index to get data

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

...

  threads count: order_index, rampup: 1 ,loops: 1 do
    transaction "Update Order" do
        post name: 'Update',
             url: 'https://YOUR.API.ENDPOINT/v1/orders/update_url',
             raw_body: '${__BeanShell(vars.get("${__threadNum}"))}' do
               with_xhr
             end
    end
    view_results_tree
    debug_sampler
  end
  
  ...

In order to achieve this goal, we need to find a way recording the number of requests. Intuitively, we can creata a request_record table, before controller process the request, by comparing the number in database and client’s request limitation, we can either allow or deny the requests.
However, we don’t need to build like this way.

Thanks to Ruby gems, you are able to find some libraries can do the trick. In this article, I choose rack-throttle to help me count the requests. The reasons I prefer rack-throttle are:

1. To limit by defining a maximum number of allowed HTTP requests per a given time period
2. Compatible with any Rack application and any Rack-based framework.
3. Compatible with the memcached, memcache-client, memcache and redis gems.
4. Well documented and the most important thing is that codes are easy to read!

Different from recording, we still need to identify which client makes this request for deciding to allow or not. Here I use Doorkeeper to tell me which user is.

In this article, I will not show you how to setup Doorkeeper, but you are about to see how to customize your own limiter.

So, That’s go!

Requirements

My targets show as below:

1. Different clients has its limitation.
2. Should limit by second, minute, hour, day and month.
3. Support whitelist

Gemfile

At first, we need to add libraries, the most important gem is rack-throttle. Depend on which places you want to store your counting data, you might need to add other gems. For me, I prefer redis. So, I added as below.

1
2
3

# Api limit
gem 'rack-throttle'
gem 'redis'

Customize your throttler

Wait wait, before we start implementing, we should understand some mechnisms and terms.

First, rack!

you might notice that the gem we use called rack-throttle. Yes! this lib is built as a Rack middleware that provides logic for rate-limiting incoming HTTP requests to Rack applications.

Ok, so what’s Rack ?

according to rack official website,

Rack provides a minimal interface between webservers that support Ruby and Ruby frameworks.

To use Rack, provide an app: an object that responds to the call method, taking the environment hash as a parameter, and returning an Array with three elements:

The HTTP response code

A Hash of headers

The response body, which must respond to each

So, Rack can help you handle the HTTP request. And due to Rack take the environment hash and also can “return” fix format: array with code, headers and body. It’s very easy for you to add middleware which can help you do cacheing, authencation, …etc. In our case, gem rack-throttle will help you adding middleware easily.

After we briefly understand how Rack works, now let’s take a look how rack-throttle’s core class: limiter.

Why it work as Rack middilewate?

The key point is:

1
2
3
4
5
6
7
8

def initialize(app, options = {})
  @app, @options = app, options
end

def call(env)
  request = Rack::Request.new(env)
  allowed?(request) ? app.call(env) : rate_limit_exceeded(request)
end

So, after we add this middleware into our Rack, every time we receive the requests it will be invoked by call method. Then we check is allowed or not, if is allowed, we pass to next Rack middleware, otherwise we stop in this middleware.

Since I need to customize the throttler, I can inherit from Rack::Throttle::Limiter.

Let’s check the final code first.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88

# lib/api_limitation/application_throttler.rb

require 'rack/throttle'
module ApiLimitation
  class ApplicationThrottler < Rack::Throttle::Limiter
    attr_accessor :client, :ip

    def initialize(app, options = {})
      super
    end

    def client_identifier(request)
      if request.env['HTTP_AUTHORIZATION'] || request.params['access_token']
        token        = request.env['HTTP_AUTHORIZATION'].present? ? request.env['HTTP_AUTHORIZATION'].split(' ')[-1] : request.params['access_token']
        access_token = Doorkeeper::AccessToken.where(token: token).first
        @client = access_token ? User.find(access_token.resource_owner_id) : nil
      end
      @ip = request.ip.to_s
    end

    def max_per_second
      @client.try(:limitation).try(:second) || 1
    end

    def max_per_minute
      @client.try(:limitation).try(:minute) || 60
    end

    def max_per_hourly
      @client.try(:limitation).try(:hourly) || 3_600
    end

    def max_per_daily
      @client.try(:limitation).try(:daily) || 86_400
    end

    def max_per_monthly
      @client.try(:limitation).try(:monthly) || 2_592_000
    end

    def allowed?(request)
      client_identifier(request)
      return true if whitelisted?(request)

      ['second', 'minute', 'hourly', 'daily', 'monthly'].all? { |timeslot| send("#{timeslot}_check".to_sym) }
    end

    def whitelisted?(request)
      @client.try(:email).in? ['YOUR_WHITE_LIST']
    end

    protected

    ['second', 'minute', 'hourly', 'daily', 'monthly'].each do |timeslot|
      define_method("#{timeslot}_check".to_sym) do
        count = cache_get(key = send("#{timeslot}_cache_key".to_sym)).to_i + 1 rescue 1
        allowed = count <= send("max_per_#{timeslot}".to_sym).to_i
        begin
          cache_set(key, count)
          allowed
        rescue => e
          allowed = true
        end
      end
    end

    def second_cache_key
      [@client.try(:id) || @ip, Time.now.strftime('%Y-%m-%dT%H:%M:%S')].join(':')
    end

    def minute_cache_key
      [@client.try(:id) || @ip, Time.now.strftime('%Y-%m-%dT%H:%M')].join(':')
    end

    def hourly_cache_key
      [@client.try(:id) || @ip, Time.now.strftime('%Y-%m-%dT%H')].join(':')
    end

    def daily_cache_key
      [@client.try(:id) || @ip, Time.now.strftime('%Y-%m-%d')].join(':')
    end

    def monthly_cache_key
      [@client.try(:id) || @ip, Time.now.strftime('%Y-%m')].join(':')
    end

  end
end

The most important function is allowed?(request).

1
2
3
4
5
6

def allowed?(request)
    client_identifier(request)
    return true if whitelisted?(request)

    ['second', 'minute', 'hourly', 'daily', 'monthly'].all? { |timeslot| send("#{timeslot}_check".to_sym) }
end

Let’s check each functions.

First, client_identifier(request).

1
2
3
4
5
6
7
8

def client_identifier(request)
    if request.env['HTTP_AUTHORIZATION'] || request.params['access_token']
   token        = request.env['HTTP_AUTHORIZATION'].present? ? request.env['HTTP_AUTHORIZATION'].split(' ')[-1] : request.params['access_token']
   access_token = Doorkeeper::AccessToken.where(token: token).first
   @client = access_token ? User.find(access_token.resource_owner_id) : nil
      end
   @ip = request.ip.to_s
 end

We used doorkeeper for Oauth2 authentication.

So, each requests might have HTTP_AUTHORIZATION in header or access_token in url. Then we need to use the access_token to find who he is.

Second, whitelisted?(request)

1
2
3

def whitelisted?(request)
  @client.try(:email).in? ['YOUR_WHITE_LIST']
end

Simple! Right?, because we already have @client from client_identifier(request). I just try to find its email in list or not.

Finall part, ['second', 'minute', 'hourly', 'daily', 'monthly'].all? { |timeslot| send("#{timeslot}_check".to_sym) }

here I use metaprogramming to avoid writing too many duplicate codes which call these functions: second_check, minute_check, …etc.

And you can’t find these functions in codes, which is because I also use metaprogramming method to implement class’s instance methods.

1
2
3
4
5
6
7
8
9
10
11
12

['second', 'minute', 'hourly', 'daily', 'monthly'].each do |timeslot|
  define_method("#{timeslot}_check".to_sym) do
    count = cache_get(key = send("#{timeslot}_cache_key".to_sym)).to_i + 1 rescue 1
    allowed = count <= send("max_per_#{timeslot}".to_sym).to_i
    begin
      cache_set(key, count)
      allowed
    rescue => e
      allowed = true
    end
  end
end

Ok, so here I try to get count number from cache(for me, I used Redis) by cache_get which is in base class: Rack::Throttle::Limiter. And the cache keys are either client’s id or ip with timestamp. for example: 12345:2016-11-24 or 127.0.0.1:2016-11-24.

1
2
3
4

def second_cache_key
  [@client.try(:id) || @ip, Time.now.strftime('%Y-%m-%dT%H:%M:%S')].join(':')
end
...

After we get the number of count we can compare it to client’s limitation and add counting number as well.

And how do we get client’s limitation?

In our database design, user has one limitation which include these columns: second, minute…etc. So, I can easily get it by @client.limitation.XXX

1
2
3
4

def max_per_second
  @client.try(:limitation).try(:second) || 1
end
....

That’s it. it’s not too hard right?

Add your throttler into Rack.

This part is super easy, just follow rack-throttle’s README.

because I built it for Rails application.

I need to add this line into config/application.rb

1
2
3
4
5
6

module YOUR_AWESOME_APPLICATION
  class Application < Rails::Application
   ...
    config.middleware.use ::ApiLimitation::ApplicationThrottler, cache: Redis.new(host: YOUR_REDIS_HOST), key_prefix: :awesome_throttle, message: 'Rate Limit Exceeded, please upgrade your plan to get more requests'
  end
end

Do you remember that I mentioned I use Redis for caching count? Here is the magic.

I initialize @cache in here. So that it can be accessed in class.

So, I want to split this audio into seperate mp3. Spending few hours, I found this library!

I have to thank @ftrvxmtrx - the author of https://github.com/ftrvxmtrx/split2flac.

In this post, I just show to how to setup the environment to use ftrvxmtrx’s libarary in MacOSX.

1. Make sure you have homebrew

2. Install required/optional libs

1
2
3
4
5
6
7
8
9

brew install cuetools
brew install shntool
brew install id3lib
brew install flac
brew install homebrew/dupes/libiconv
brew install enca
# for install mac to split APE
brew tap fernandotcl/homebrew-fernandotcl
brew install monkeys-audio

3. Clone project

1

git clone https://github.com/ftrvxmtrx/split2flac.git

4. Convert files

1

cd split2flac

Check commands

1

./split2flac -h

Convert the file

1

./split2flac -f mp3 -o ~/Desktop/ -cue path/to/your/cue-file.cue path/to/your/ape-file.ape

-f: using mp3 as output format

-o: where to put the output files

cue: file location to your .cue

yay!

Preloader is a node server that uses phantomjs to create static HTML out of a javascript page.

you are able to use their solutions instead of installing in your own machine.

But, what if you just like me, want to try their service first and want to customize your own Preloader?

here are flows that show how I install Preloader in Ubuntu 14.04.

1. Create an instance in Amazon Web Service

hmmm, just go to AWS and create an instance using Ubuntu 14.04 image.

make sure you allow port 3000 can be access

2. Create a user

1
2
3

sudo adduser deploy
sudo adduser deploy sudo
su deploy

3. Make sure you can login using new user without pem key

1
2
3
4

cd ~
mkdir .ssh
cd .ssh
vim authorized_keys

then put the content in ~/.ssh/id_rsa.pub in your local machine into it.

4. Install all services

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

sudo locale-gen UTF-8
sudo apt-get update

# Install git
sudo apt-get install git

# Install phantomjs
sudo apt-get install phantomjs

# Install Node.js v6
curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -
sudo apt-get install -y nodejs

# Clone the project
git clone https://github.com/prerender/prerender.git
cd prerender
npm install

# Fix: Fontconfig warning: ignoring UTF-8: not a valid region tag
export LANGUAGE=en_US.UTF-8
export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8
locale-gen en_US.UTF-8

# Run Node server
node server.js

5. Now you can try in browser

http://your.ip:3000/https://www.google.com

6. Run your Preloader service after restart Machine

1
2

cd ~/prerender
vim startup.sh

Put these lines to the file

1
2
3
4
5

#!/usr/bin/env bash
export LANGUAGE=en_US.UTF-8
export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8
node /home/deploy/prerender/server.js

Allow the script can be excuted

1

chmod +x startup.sh

Setup /etc/init/prerender.conf

1

sudo vim /etc/init/prerender.conf

Put these lines to the file

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

#!upstart
description "A job that runs the prerender service"
author "Deploy"

start on filesystem or runlevel [2345]
script
    export HOME="/home/deploy/prerender"
    cd /home/deploy/prerender
    exec su -c ' /home/deploy/prerender/startup.sh'
end script

pre-start script
    echo "['date'] Prerender Service Starting" >> /var/log/prerender/prerender.log
end script

pre-stop script
    echo "['date'] Prerender Service Stopping" >> /var/log/prerender/prerender.log
end script

respawn
respawn limit 10 90

Before start you need to create this folder

1

sudo mkdir /var/log/prerender

Now you can

1

sudo service prerender start

References:

faye-websockets-part-1

faye-websockets-part-2

How to Use Faye as a Real-Time Push Server in Rails

====================================================

In your remote Ubuntu server

====================================================

Step1. Install Ruby environment in Ubuntu

Reference: https://gorails.com/deploy/ubuntu/14.04

Add user

1
2
3

sudo adduser deploy
sudo adduser deploy sudo
su deploy

Make sure you can login using new user

1
2
3
4

cd ~
mkdir .ssh
cd .ssh
vim authorized_keys

then put the content in ~/.ssh/id_rsa.pub in your local machine into it.

Install Ruby and RVM

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

sudo apt-get update

sudo apt-get install git-core curl zlib1g-dev build-essential libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev libcurl4-openssl-dev python-software-properties libffi-dev

sudo apt-get install libgdbm-dev libncurses5-dev automake libtool bison libffi-dev

gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3

curl -L https://get.rvm.io | bash -s stable

source ~/.rvm/scripts/rvm

rvm install 2.2.3

gem install bundler

Create fake SSL for thin server

1
2

mkdir -p /home/deploy/faye-server/shared/thin/ssl
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /home/deploy/faye-server/shared/thin/ssl/thin.key -out /home/deploy/faye-server/shared/thin/ssl/thin.crt

Create .env file

vim /home/deploy/faye-server/shared/.env

put like this

1

AUTH_TOKEN: 'your_secret'

Create thin.yml file

ps. copied from config/thin.yml.example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

port: 8080
user: deploy
group: deploy
pid: /home/deploy/faye-server/current/faye/pids/thin.pid
timeout: 30
wait: 30
log: /home/deploy/faye-server/current/log/thin.log
max_conns: 4096
require: []
environment: production
max_persistent_conns: 1024
servers: 1
threaded: true
no-epoll: true
daemonize: true
chdir: /home/deploy/faye-server/current
tag: faye
ssl: true
ssl-key-file: /home/deploy/faye-server/current/thin/ssl/thin.key
ssl-cert-file: /home/deploy/faye-server/current/thin/ssl/thin.crt
ssl-disable-verify: true

====================================================

In your local machine

====================================================

Step2. Clone this project

1

git clone https://github.com/alChaCC/faye-server.git

Update Some Configurations

config/deploy/staging.rb

1

server 'your server ip'

Step3. Deployment

1

cap staging deploy

How to test?

Before start you have to update the IP in faye-server/test/index.html and faye-server/test/test.rb

And also the secret in faye-server/test/test.rb

And open browser then type: https://your.server.ip:8080/faye/faye.js.

Since we make fake ssl, we need to tell browser to accept it anyway.

you have to make sure that you can see this.

Then, you can type in console

1
2

cd faye-server/test
python -m SimpleHTTPServer 8000

and also

1
2
3
4

irb
require './test'
t = SimpleTest.new
t.tester(1, 100)

Now you can open browser and type http://localhost:8000/

Map-reduce is a data processing paradigm for condensing large volumes of data into useful aggregated results.

MapReduce is a popular big data term in recent years proposed by Google. It is a method for manipulate large data sets parallelly and distributedly on many machines. In my words, I usually said that Map-Reduce, “Map” is to assign match function to many machines for cutting a huge data into small data sets(group matched data), and then use “Reduce” to aggregate these calculated data.

In this map-reduce operation, MongoDB applies the map phase to each input document (i.e. the documents in the collection that match the query condition). The map function emits key-value pairs. For those keys that have multiple values, MongoDB applies the reduce phase, which collects and condenses the aggregated data. MongoDB then stores the results in a collection. Optionally, the output of the reduce function may pass through a finalize function to further condense or process the results of the aggregation.

if you want more details, please check official documents

Let’s go check how to use map reduce in Mongoid.

I also write an article for aggregation method. Please check: Ruby on Rails Development Using Mongoid 5.0.0 - 2. How to Use Aggregation to Get Pageview Data

Again, Here are my models,

Calculate Unique Pageviews from 2016/1/1 to 2016/1/5

First, MongoDB use custom Javascript functions to do map and reduce operations. So, we need to write JS code as a string and pass to MongoDB.

In this example, I hope that the same page a user watched in the same day should be calculated just one time(eg. If I watch path ‘/’ 3 times a day, the pageview should be 1 not 3). So, I call it unique pageviews.

Let’s Check Map Function

I use “pathname” as an identifier(key) and use {“total”: 1} as a value. In my opinion, value is the result that you want to obtain.

In the map function, reference the current document as this within the function.

1
2
3
4
5
6
7
8
9
10

map = %q{ 
              function() {
                emit(
                  this.pathname,
                  {
                    total: 1
                  } 
                );
              } 
          }

How About Reduce Function?

Reduce function will get the key and value from map function.

1
2
3
4
5
6
7
8
9
10
11
12
13

reduce = %q{
              function(key, values){
                print("=====================");
                print("Key is:" + key);
                print("Value is:" + JSON.stringify(values));
                var r = { total: 0};
                for (var idx  = 0; idx < values.length; idx++) {
                  var tt = values[idx].total;
                  r.total += tt
                }
                return r;
              }
            }

Tips: we can use “print” for debugging.

Using MapReduce in Mongoid

Now, we can put map and reduce function together.

1
2
3
4
5

res = PageTracking.where(:created_at.gte => Time.parse('2016-01-01').beginning_of_day , 
:created_at.lte => Time.parse('2016-01-05').end_of_day)
.order_by(:created_at => 'desc')
.map_reduce(map,reduce)
.out(inline: true)

console will show like below

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

=> #<Mongoid::Contextual::MapReduce
  selector: {"created_at"=>{"$gte"=>2015-12-31 16:00:00 UTC, "$lte"=>2016-01-05 15:59:59 UTC}}
  class:    PageTracking
  map:
              function() {
                emit(
                  this.pathname,
                  {
                    total: 1
                  }
);
}

  reduce:
              function(key, values){
                print("=====================");
                print("Key is:" + key);
                print("Value is:" + JSON.stringify(values));
                var r = { total: 0};
                for (var idx  = 0; idx < values.length; idx++) {
                  var tt = values[idx].total;
                  r.total += tt
                }
return r;
}

  finalize:
  out:      {:inline=>true}>

Let’s get all results.

1

res.to_a

Here is my result.

1
2
3
4
5

[{"_id"=>"/", "value"=>{"total"=>3.0}},
 {"_id"=>"/activities", "value"=>{"total"=>1.0}},
 {"_id"=>"/activities/hadalabo01", "value"=>{"total"=>1.0}},
 {"_id"=>"/beautybuzz", "value"=>{"total"=>1.0}},
 {"_id"=>"/beautynews", "value"=>{"total"=>1.0}}]

ps. If you want to check the debugging message, you can

1
2
3
4
5

# check the location of MongoDB system log
cat /usr/local/etc/mongod.conf

# In my case, it will be located in /usr/local/var/log/mongodb/mongo.log
tail -1000f /usr/local/var/log/mongodb/mongo.log

And I see

1
2
3

2016-01-07T11:49:24.460+0800 I -        [conn3] =====================
2016-01-07T11:49:24.460+0800 I -        [conn3] Key is:/
2016-01-07T11:49:24.460+0800 I -        [conn3] Value is:[{"total":1},{"total":1},{"total":1}]

It’s weird, in result, I have many different “_id” such as ‘/’, ‘/activities’, ‘/beautybuzz’. But, in MongoDB system log, I just see ‘/’. Why?

According to Requirements for the reduce Function.

MongoDB will not call the reduce function for a key that has only a single value. The values argument is an array whose elements are the value objects that are “mapped” to the key.

Let’s check the data in my MongoDB,

1

PageTracking.where(:created_at.gte => Time.parse('2016-01-01').beginning_of_day , :created_at.lte => Time.parse('2016-01-05').end_of_day).order_by(:created_at => 'desc').to_a

As you can see, pathname except ‘/’ are just have one record. So, it will not be handled by reduce function.

1
2
3
4
5
6
7

[#<PageTracking _id: 568b78c07db99b9a30000001, created_at: 2016-01-04 16:00:00 UTC, updated_at: 2016-01-05 08:03:12 UTC, pathname: "/beautynews", daily_total_count: 1, user_id: BSON::ObjectId('568b28e67db99b1ae8000003')>,
 #<PageTracking _id: 568b7e5e7db99b9a30000017, created_at: 2016-01-04 16:00:00 UTC, updated_at: 2016-01-05 08:27:10 UTC, pathname: "/beautybuzz", daily_total_count: 1, user_id: BSON::ObjectId('568b28e67db99b1ae8000003')>,
 #<PageTracking _id: 568b7d8e7db99b9a30000013, created_at: 2016-01-04 16:00:00 UTC, updated_at: 2016-01-05 08:23:42 UTC, pathname: "/activities/hadalabo01", daily_total_count: 1, user_id: BSON::ObjectId('568b28e67db99b1ae8000003')>,
 #<PageTracking _id: 568b7d307db99b9a30000011, created_at: 2016-01-04 16:00:00 UTC, updated_at: 2016-01-05 08:22:08 UTC, pathname: "/activities", daily_total_count: 1, user_id: BSON::ObjectId('568b28e67db99b1ae8000003')>,
 #<PageTracking _id: 568b28e67db99b1ae8000005, created_at: 2016-01-04 16:00:00 UTC, updated_at: 2016-01-05 02:22:30 UTC, pathname: "/", daily_total_count: 3, user_id: BSON::ObjectId('568b28e67db99b1ae8000003')>,
 #<PageTracking _id: 568b28ad7db99b1ae8000002, created_at: 2016-01-04 16:00:00 UTC, updated_at: 2016-01-05 02:21:33 UTC, pathname: "/", daily_total_count: 1, user_id: BSON::ObjectId('568b28ad7db99b1ae8000000')>,
 #<PageTracking _id: 568a1ef77db99bc968000002, created_at: 2016-01-03 16:00:00 UTC, updated_at: 2016-01-04 07:27:51 UTC, pathname: "/", daily_total_count: 2, user_id: BSON::ObjectId('568a1ef77db99bc968000000')>]

If I watch ‘/activities’ again, and run MapReduce again.

yay, I can see /activities in system log.

1
2
3
4
5
6

2016-01-07T14:42:23.349+0800 I -        [conn3] =====================
2016-01-07T14:42:23.349+0800 I -        [conn3] Key is:/
2016-01-07T14:42:23.350+0800 I -        [conn3] Value is:[{"total":1},{"total":1},{"total":1}]
2016-01-07T14:42:23.350+0800 I -        [conn3] =====================
2016-01-07T14:42:23.350+0800 I -        [conn3] Key is:/activities
2016-01-07T14:42:23.350+0800 I -        [conn3] Value is:[{"total":1},{"total":1}]

Calculate Non-Unique Pageviews

In this example, I don’t care a user watched same page in the same day. I want all pageviews. (eg. If I watch path ‘/’ 3 times a day, the pageview should be 3). So, I call it non-unique pageviews.

The only thing I have to update is map function. Change total: 1 to total: this.daily_total_count

1
2
3
4
5
6
7
8
9
10

total_map = %q{ 
              function() {
                emit(
                  this.pathname,
                  {
                    total: this.daily_total_count
                  } 
                );
              } 
          }

yay!

1
2
3
4
5

[{"_id"=>"/", "value"=>{"total"=>6.0}},
 {"_id"=>"/activities", "value"=>{"total"=>2.0}},
 {"_id"=>"/activities/hadalabo01", "value"=>{"total"=>1.0}},
 {"_id"=>"/beautybuzz", "value"=>{"total"=>1.0}},
 {"_id"=>"/beautynews", "value"=>{"total"=>1.0}}]

Calculate Pageviews and Get Who Watch The Page

In this example, beside calculate pageviews, I want to get user who watch the page as well.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

map_with_user = %q{ 
              function() {
                var key = {
                  pathname: this.pathname
                };
                emit(
                  key,
                  {
                    total:  this.daily_total_count,
                    user_id:   this.user_id
                  } 
                );
              } 
          }

reduce_with_user = %q{
              function(key, values){
                print("=====================");
                print("Key is:" + JSON.stringify(key));
                print("Value is:" +  JSON.stringify(values));
                var r = { 
                    total: 0,
                    users: []
                };
                for (var idx  = 0; idx < values.length; idx++) {
                    r.total += values[idx].total;
                    r.users.push(values[idx].user_id);
                }
                return r;
              }
            }

PageTracking.where(:created_at.gte => Time.parse('2016-01-01').beginning_of_day , 
:created_at.lte => Time.parse('2016-01-07').end_of_day)
.order_by(:created_at => 'desc')
.map_reduce(map_with_user,reduce_with_user)
.out(inline: true).to_a

Tips: key can also be a hash.

Since we want to get all users so I add users: [], then I just push user_id in it.

The result is…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

[
{  
  "_id"  =>  {  
    "pathname"    =>"/"
  },
  "value"  =>  {  
    "total"    =>6.0,
    "users"    =>    [  
      BSON::ObjectId('568b28e67db99b1ae8000003'),
      BSON::ObjectId('568b28ad7db99b1ae8000000'),
      BSON::ObjectId('568a1ef77db99bc968000000')
    ]
  }
},
{  
  "_id"  =>  {  
    "pathname"    =>"/activities"
  },
  "value"  =>  {  
    "total"    =>3.0,
    "users"    =>    [  
      BSON::ObjectId('568e08937db99b12c1000000'),
      BSON::ObjectId('568b28e67db99b1ae8000003')
    ]
  }
},
{  
  "_id"  =>  {  
    "pathname"    =>"/activities/hadalabo01"
  },
  "value"  =>  {  
    "total"    =>1.0,
    "user_id"    =>BSON::ObjectId('568b28e67db99b1ae8000003')
  }
},
{  
  "_id"  =>  {  
    "pathname"    =>"/beautybuzz"
  },
  "value"  =>  {  
    "total"    =>1.0,
    "user_id"    =>BSON::ObjectId('568b28e67db99b1ae8000003')
  }
},
{  
  "_id"  =>  {  
    "pathname"    =>"/beautynews"
  },
  "value"  =>  {  
    "total"    =>1.0,
    "user_id"    =>BSON::ObjectId('568b28e67db99b1ae8000003')
  }
}
]

Take a look of system log, to see what’s key and values pass to reduce function.

1
2
3
4
5
6

2016-01-07T15:43:24.520+0800 I -        [conn3] =====================
2016-01-07T15:43:24.520+0800 I -        [conn3] Key is:{"pathname":"/"}
2016-01-07T15:43:24.521+0800 I -        [conn3] Value is:[{"total":3,"user_id":{"str":"568b28e67db99b1ae8000003"}},{"total":1,"user_id":{"str":"568b28ad7db99b1ae8000000"}},{"total":2,"user_id":{"str":"568a1ef77db99bc968000000"}}]
2016-01-07T15:43:24.521+0800 I -        [conn3] =====================
2016-01-07T15:43:24.521+0800 I -        [conn3] Key is:{"pathname":"/activities"}
2016-01-07T15:43:24.521+0800 I -        [conn3] Value is:[{"total":2,"user_id":{"str":"568e08937db99b12c1000000"}},{"total":1,"user_id":{"str":"568b28e67db99b1ae8000003"}}]

Here, you will notice that there are some values has user_id and some has users. Again, since some keys only has single value, it will not be processed by reduce function.

Use “finalize” function to beautify the result

It’s annoying, if data format is not consistent. So, I use finalize function to solve this problem.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

finalize = %q{
  function(key, reducedValue){
    print("=====================");
    print("Key is:" + JSON.stringify(key));
    print("reducedValue is:" + JSON.stringify(reducedValue));
    var result = {
      user_data: []
    }
    if(reducedValue.users){
      reducedValue.users.forEach(function(u){
        result.user_data.push(u);
       });
    } else {
      result.user_data.push(reducedValue.user_id);
    }
    return result
  }
}

PageTracking.where(:created_at.gte => Time.parse('2016-01-01').beginning_of_day , 
:created_at.lte => Time.parse('2016-01-07').end_of_day)
.order_by(:created_at => 'desc')
.map_reduce(map_with_user,reduce_with_user)
.finalize(finalize)
.out(inline: true).to_a

Yay, every results looks same~~

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

[
{  
  "_id"  =>  {  
    "pathname"    =>"/"
  },
  "value"  =>  {  
    "user_data"    =>    [  
      BSON::ObjectId('568b28e67db99b1ae8000003'),
      BSON::ObjectId('568b28ad7db99b1ae8000000'),
      BSON::ObjectId('568a1ef77db99bc968000000')
    ]
  }
},
{  
  "_id"  =>  {  
    "pathname"    =>"/activities"
  },
  "value"  =>  {  
    "user_data"    =>    [  
      BSON::ObjectId('568e08937db99b12c1000000'),
      BSON::ObjectId('568b28e67db99b1ae8000003')
    ]
  }
},
{  
  "_id"  =>  {  
    "pathname"    =>"/activities/hadalabo01"
  },
  "value"  =>  {  
    "user_data"    =>    [  
      BSON::ObjectId('568b28e67db99b1ae8000003')
    ]
  }
},
{  
  "_id"  =>  {  
    "pathname"    =>"/beautybuzz"
  },
  "value"  =>  {  
    "user_data"    =>    [  
      BSON::ObjectId('568b28e67db99b1ae8000003')
    ]
  }
},
{  
  "_id"  =>  {  
    "pathname"    =>"/beautynews"
  },
  "value"  =>  {  
    "user_data"    =>    [  
      BSON::ObjectId('568b28e67db99b1ae8000003')
    ]
  }
}]

Aggregations are operations that process data records and return computed results.

In my opinion, it just like queries but it can do some operations step by step when processing the query just like a pipeline.

if you want more details, please check official documents

Here are my models,

Get Data using “match”

If I want to find pageview from 2016/1/1 to 2016/1/5

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

sort_stage = {
        '$sort' => { 'created_at' => 1 }
    }
    
match_stage = {
          '$match' => {
            "created_at" => {
              "$gte" => Time.parse('2016-01-01').beginning_of_day, 
              "$lte" => Time.parse('2016-01-05').end_of_day
            }
          }
        }

result = PageTracking.collection.aggregate([sort_stage,match_stage])
result.each do |page|
  puts page
end

Here is my result:

1
2
3
4
5
6
7

{"_id"=>BSON::ObjectId('568a1ef77db99bc968000002'), "pathname"=>"/", "daily_total_count"=>2, "created_at"=>2016-01-03 16:00:00 UTC, "user_id"=>BSON::ObjectId('568a1ef77db99bc968000000'), "updated_at"=>2016-01-04 07:27:51 UTC }
{"_id"=>BSON::ObjectId('568b28ad7db99b1ae8000002'), "pathname"=>"/", "daily_total_count"=>1, "created_at"=>2016-01-04 16:00:00 UTC, "user_id"=>BSON::ObjectId('568b28ad7db99b1ae8000000'), "updated_at"=>2016-01-05 02:21:33 UTC}
{"_id"=>BSON::ObjectId('568b28e67db99b1ae8000005'), "pathname"=>"/", "daily_total_count"=>3, "created_at"=>2016-01-04 16:00:00 UTC, "user_id"=>BSON::ObjectId('568b28e67db99b1ae8000003'), "updated_at"=>2016-01-05 02:22:30 UTC}
{"_id"=>BSON::ObjectId('568b7d307db99b9a30000011'), "pathname"=>"/activities", "daily_total_count"=>1, "created_at"=>2016-01-04 16:00:00 UTC, "user_id"=>BSON::ObjectId('568b28e67db99b1ae8000003'), "updated_at"=>2016-01-05 08:22:08 UTC}
{"_id"=>BSON::ObjectId('568b7d8e7db99b9a30000013'), "pathname"=>"/activities/hadalabo01", "daily_total_count"=>1, "created_at"=>2016-01-04 16:00:00 UTC, "user_id"=>BSON::ObjectId('568b28e67db99b1ae8000003'), "updated_at"=>2016-01-05 08:23:42 UTC}
{"_id"=>BSON::ObjectId('568b7e5e7db99b9a30000017'), "pathname"=>"/beautybuzz", "daily_total_count"=>1, "created_at"=>2016-01-04 16:00:00 UTC, "user_id"=>BSON::ObjectId('568b28e67db99b1ae8000003'), "updated_at"=>2016-01-05 08:27:10 UTC}
{"_id"=>BSON::ObjectId('568b78c07db99b9a30000001'), "pathname"=>"/beautynews", "daily_total_count"=>1, "created_at"=>2016-01-04 16:00:00 UTC, "user_id"=>BSON::ObjectId('568b28e67db99b1ae8000003'), "updated_at"=>2016-01-05 08:03:12 UTC}

ps. I add a sort stage in front of match stage due to the official suggestion.Aggregation Pipeline Optimization

Group Data using “group”

Oops, in fact, I want all records with pathname => ‘/’ should be summed up to one record just like group_by in MySQL

So, we can add a new stage

1
2
3
4
5
6
7
8
9
10
11
12
13

group_stage = {
          "$group" => {
            "_id" => {
              "pathname" => "$pathname"
            },
            "page_count" => { "$sum" => "$daily_total_count" }
          }
         }
result = PageTracking.collection.aggregate([sort_stage,match_stage,group_stage])

result.each do |page|
  puts page
end

which mean we can group by “pathname” and when we do grouping we can sum all daily_total_count into “page_count”

Here is my result

1
2
3
4
5

{"_id"=>{"pathname"=>"/beautybuzz"}, "page_count"=>1}
{"_id"=>{"pathname"=>"/activities/hadalabo01"}, "page_count"=>1}
{"_id"=>{"pathname"=>"/activities"}, "page_count"=>1}
{"_id"=>{"pathname"=>"/beautynews"}, "page_count"=>1}
{"_id"=>{"pathname"=>"/"}, "page_count"=>6}

Add more information while grouping Data using “push”

Ok, great! But, I got an another requirement, Hi, Aloha, Can you tell me that who see each pages?

Easy, let’s add some codes in group_stage

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

group_stage = {
          "$group" => {
            "_id" => {
              "pathname" => "$pathname"
            },
            "page_count" => { "$sum" => "$daily_total_count" },
            "users" => {
              "$push" => {
                "user" => "$user_id"
              }
            }
          }
         }
result = PageTracking.collection.aggregate([sort_stage,match_stage,group_stage])

result.each do |page|
  puts page
end

Here is my result

1
2
3
4
5

{"_id"=>{"pathname"=>"/beautybuzz"}, "page_count"=>1, "users"=>[{"user"=>BSON::ObjectId('568b28e67db99b1ae8000003')}]}
{"_id"=>{"pathname"=>"/activities/hadalabo01"}, "page_count"=>1, "users"=>[{"user"=>BSON::ObjectId('568b28e67db99b1ae8000003')}]}
{"_id"=>{"pathname"=>"/activities"}, "page_count"=>1, "users"=>[{"user"=>BSON::ObjectId('568b28e67db99b1ae8000003')}]}
{"_id"=>{"pathname"=>"/beautynews"}, "page_count"=>1, "users"=>[{"user"=>BSON::ObjectId('568b28e67db99b1ae8000003')}]}
{"_id"=>{"pathname"=>"/"}, "page_count"=>6, "users"=>[{"user"=>BSON::ObjectId('568a1ef77db99bc968000000')}, {"user"=>BSON::ObjectId('568b28ad7db99b1ae8000000')}, {"user"=>BSON::ObjectId('568b28e67db99b1ae8000003')}]}

Cool, our aggregation run smoothly.

Aggregate Big Data

After 3 months, we found that some fetch pageview operations will fail.

Oops, since these fetch operations use big time range (ex: get pageview data from 2015/9/1 ~ 2016/1/1)

1
2
3
4
5
6
7
8
9
10
11
12
13
14

match_stage = {
          '$match' => {
            "created_at" => {
              "$gte" => Time.parse('2015-09-01').beginning_of_day, 
              "$lte" => Time.parse('2016-01-01').end_of_day
            }
          }
        }

result = PageTracking.collection.aggregate([sort_stage,match_stage,group_stage])

result.each do |page|
  puts page
end

I got this error message:

1
2

aggregate | FAILED | exception: Exceeded memory limit for $group, but didn't allow external sort. Pass allowDiskUse:true to opt in. (16945) | 9.955263s
Mongo::Error::OperationFailure: exception: Exceeded memory limit for $group, but didn't allow external sort. Pass allowDiskUse:true to opt in. (16945)

How can we solve this problem?

Just like message says, add allowDiskUse:true to it.

1
2
3
4
5
6
7

result = PageTracking.collection.aggregate(
            [sort_stage,match_stage,group_stage],
            :allow_disk_use => true
          )
result.each do |page|
  puts page
end

Yay, bug fixed.

1
2
3

...
you will got a lots of data ....
...

Limit data using “limit”

if you just want 1000 data pass to next stage

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

sort_stage = {
        '$sort' => { 'created_at' => -1 }
    }

limit_stage = {
      '$limit' => 1000
    }
    
result = UserPageTracking.collection.aggregate(
      [sort_stage, limit_stage , match_stage, group_stage],
      :allow_disk_use => true
    )
result.each do |page|
  puts page
end

Only want some specific data using “project”

If fact, we don’t need created_at, updated_at attributes in group stage, so we can …

1
2
3
4
5
6
7
8
9
10
11

project_stage = 
      { 
        "$project" => { user_id: 1, pathname: 1, daily_total_count: 1}
      }

result = PageTracking.collection.aggregate(
            [sort_stage,match_stage, project_stage, group_stage]
          )
result.each do |page|
  puts page
end

This tutorial series will help you start your Rails project with MongoDB.

And I use Mongoid 5.0.0 as an example.

In this tutorial, you will be able to see how to

1. Install MongoDB in Mac OSX

2. Create Some Database Users in MongoDB

3. Setup Rails Projects

(Tips)

1. Dump Data

2. Restore Data

Let’s go !

Install MongoDB in Mac OSX

1
2
3
4
5
6
7
8
9
10
11

# Intall Homebrew (Mac OSX package manager)

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

# Update the Hombrew packages first

brew update

# Install MongoDB

brew install MongoDB

If you want to see where MongoDB is installed

1

brew ls mongodb

Create Some Database Users in MongoDB

Since I want to enable access control on a MongoDB instance, just like Mysql, I have to use username and password to access MongoDB.

First, you must to know where you can setup MongoDB.

1
2
3
4
5
6
7

# In MacOSX

vim /usr/local/etc/mongod.conf

# In Ubuntu 

vim /etc/mongod.conf

Second, you have to make sure that authorization is disable.

In MacOSX

1
2
3
4
5
6
7
8
9
10
11
12

# /usr/local/etc/mongod.conf

systemLog:
  destination: file
  path: /usr/local/var/log/mongodb/mongo.log
  logAppend: true
storage:
  dbPath: /usr/local/var/mongodb
net:
  bindIp: 127.0.0.1
security:
  authorization: disabled 

In Ubuntu

1
2
3
4
5
6

# /etc/mongod.conf

...

noauth = true
...

Then (re)start MongoDB

1
2
3
4
5

# In Ubuntu
sudo service mongod restart

# In Mac OSX
mongod --config /usr/local/etc/mongod.conf

Ok, now you are a super user, you can access any database and perform any action. let’s create an administrator named ‘siteUserAdmin’ and he has “userAdminAnyDatabase” role first.

1
2
3
4
5
6
7
8
9

$ mongo
$ use admin
$ db.createUser(
 {
  user: 'siteUserAdmin',
  pwd: '1234567890',
  roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]
 }
)

Ok, now we can enable authorization mode.

In MacOSX

1
2
3
4
5
6

# vim /usr/local/etc/mongod.conf

...

security:
  authorization: enabled 

In Ubuntu

1
2
3
4
5
6

# /etc/mongod.conf

...

auth = true
...

And don’t forgot restart MongoDB.

Then we now have to use user name and password to access database.

1

mongo --host localhost --port 27017 --username siteUserAdmin --password  --authenticationDatabase admin

Now, we will create an user named ‘dbadmin’ who has ‘dbOwner’ role.

1
2
3
4
5
6
7
8
9
10
11

# assume your project will use database: 'your_awesome_project_development'

use your_awesome_project_development

db.createUser(
    {
        user: 'dbadmin',
        pwd: '1234567890',
        roles: [ { role: "dbOwner", db: "your_awesome_project_development" } ]
    }
)

if you want to update user’s setting.

1
2
3
4
5
6
7
8
9
10

db.updateUser(
    "dbadmin",
    {
        pwd: 'aloha',
        roles: 
          [
              {role: "read", db: "your_awesome_project_development"}
          ]
    }
)

if you want to know which roles can perform which actions, you can find anwser here:

http://docs.mongodb.org/master/reference/built-in-roles/#userAdmin

Setup Rails Projects

1. create a project without active-record

1

rails new your_awesome_project --skip-active-record

2. add Mongoid to Gemfile

1
2
3

# Gemfile

gem 'mongoid', '~> 5.0.0'

3. Create a mongoid.yml

1
2

bundle install
rails g mongoid:config

4. Update yout mongoid.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

development:
  clients:
    default:
      database: your_awesome_project_development
      hosts:
        - localhost:27017
      options:
        user: 'dbadmin'
        password: '1234567890'
        roles:
          - 'dbOwner'
test:
  clients:
    default:
      database: your_awesome_project_test
      hosts:
        - localhost:27017
      options:
        max_retries: 1
        retry_interval: 0
        user: 'dbadmin_tester'
        password: '1234567890'
        roles:
          - 'dbOwner'

Dump Data

Please make sure that you have backup role in production MongoDB.

I use ‘readWriteAnyDatabase’ role.

Assume that you need to dump data from production data and restore to local mongo server.

1

mongodump --host your.production.mongo.server.ip --port 37017 --username user --password --out /Users/AlohaCC/Desktop/production-mongodump-2015-10-04

Restore Data

Please make sure that you have backup role in local MongoDB.

1

mongorestore --host localhost --port 3017 --username user --password  /Users/AlohaCC/Desktop/production-mongodump-2015-10-04

let’s check it out~

If you have a class named: DumpDataFromOldServer and it located at folder ‘/app/development_only’. like this:

1
2
3
4
5

# app/development_only/dump_data_from_old_server.rb

class DumpDataFromOldServer < ActiveRecord::Base
    ....
end

Step1. Application.rb

you still need to let Rails know which folders should be loaded by using config.eager_load_paths.

ps. what’s difference between eager_load and auto_load?

please check this http://stackoverflow.com/questions/19773266/confusing-about-autoload-paths-vs-eager-load-paths-in-rails-4

1

 config.eager_load_paths += %W(#{Rails.root}/app/development_only)

It will be:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# config/application.rb

require File.expand_path('../boot', __FILE__)

Bundler.require(*Rails.groups)

module MyAwesomeProject
  class Application < Rails::Application
   ...
   
   config.eager_load_paths += %W(#{Rails.root}/app/development_only)
   
   ...
   end 
end

Step2. Update your environment file

ref: http://stackoverflow.com/questions/13756986/how-to-blacklist-directory-loading-in-rails

Add these lines

1
2
3
4
5
6
7

  path_rejector = lambda { |s| s.include?("app/development_only") }

  # Remove the path from being loaded when Rails starts:
  config.eager_load_paths = config.eager_load_paths.reject(&path_rejector)

  # Remove the path from being lazily loaded
  ActiveSupport::Dependencies.autoload_paths.reject!(&path_rejector)

It will be:

1
2
3
4
5
6
7
8
9
10
11

# config/environments/production.rb
Rails.application.configure do
  
  ...

  path_rejector = lambda { |s| s.include?("app/development_only") }

  config.eager_load_paths = config.eager_load_paths.reject(&path_rejector)

  ActiveSupport::Dependencies.autoload_paths.reject!(&path_rejector)
end

Step3. Test

1
2
3
4
5
6
7
8
9
10
11

# In Development console: rails c 

> DumpDataFromOldServer.all.first 

DumpDataFromOldServer Load (0.6ms)  SELECT  `dump_data_from_old_server`.* FROM `dump_data_from_old_server `  LIMIT 1
=> #<DumpDataFromOldServer id: 1, created_at: "2014-05-09 16:59:55">


# In Production console: rails c -e production
> DumpDataFromOldServer.all.first
NameError: uninitialized constant DumpDataFromOldServer

This implementation refer to Google guideline

Step1. Preparation

1. Create a Service Account

2. Download JSON file

And I put JSON file into config folder.

3. Copy a Email and Add to Google Analytics

Step2. Get Access Token

Add to Gemfile

1
2

gem "typhoeus"
gem 'jwt'

Create a file named lib/google_anallytics.rb

Again, this implementation refer to Google guideline. Please, check this guideline first. Then you will understand how I write fuction get_token.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

require 'typhoeus'
require "jwt"
require 'date'
module GoogleAnalytic
 
  def get_token
    json_file = File.join("#{Rails.root}/config", 'YOUR-JSON-FILE-NAME.json')
    file = File.read(json_file)
    key_hash = JSON.parse(file)
    aud     = "https://www.googleapis.com/oauth2/v3/token"
    now     = Time.new
    iat     = (now - 60).to_i
    @exp    = (now + 50.minutes).to_i
    scopes  = 'https://www.googleapis.com/auth/analytics.readonly'
    sub     = key_hash["client_email"]
    iss     = key_hash["client_email"]
    jwt_claim_set    = {
      'iss' => iss, 
      'sub' => sub, 
      'scope' => scopes,
      'aud' => aud, 
      'exp' => @exp,
      'iat' => iat
      }
    token            = JWT.encode(jwt_claim_set,  OpenSSL::PKey::RSA.new(key_hash["private_key"]), 'RS256')
    
    res = Typhoeus::Request.post('https://www.googleapis.com/oauth2/v3/token', 
      body: 
        { 
          grant_type: "urn:ietf:params:oauth:grant-type:jwt-bearer",
          assertion: token
        }
    )
    
    res_hash = MultiJson.load(res.body)

    token = res_hash["access_token"]
    return token
  end
end

Step3. Fetch Data

In my case, I create a model for saving Google Analytics data. So, I’ll show you how I fetch data.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71

class GoogleAnalyticEvent
  include GoogleAnalytic
  include Mongoid::Document
  include Mongoid::Timestamps
  field :category, type: String        
  field :label, type: String        
  field :action, type: String
  field :total_events, type: Integer       
  field :unique_events, type: Integer
  field :data_date, type: Date
  index({category: 1, label: 1, data_date: 1, action:1 })
  
  def get_event_from_ga_api(params={})
    @data_date = Date.today - 1
    @parameters = {
        'ids'         => "ga:12345678",
        'start-date'  => @data_date.strftime("%Y-%m-%d"),
        'end-date'    => @data_date.strftime("%Y-%m-%d"),
        'metrics'     => "ga:totalEvents,ga:uniqueEvents",
        'dimensions'  => "ga:eventCategory,ga:eventLabel,ga:eventAction",
        'max-results' => "10000"
      }
    query_paginate(@parameters)
  end
  
  def query_paginate(parameters)
    parameters['access_token'] = get_token
    count = 0
    loop do
        if Time.new.to_i > @exp
          parameters['access_token'] = get_token
        end
       res = Typhoeus::Request.get('https://www.googleapis.com/analytics/v3/data/ga', 
        params: parameters
       )
      result = MultiJson.load(res.body)
      if result["error"].nil?
        sync_to_db(result)  if result["totalResults"] > 0
        max_count = result["totalResults"] / parameters["max-results"].to_i
      end
      break if count == max_count || result["error"].present?
      count += 1
      parameters["start-index"] = parameters["max-results"].to_i * count + 1
    end
  end

  def sync_to_db(result)
    result["columnHeaders"].each_with_index do |h,i|
      case h["name"].to_s
      when "ga:eventCategory"
        @category= i 
      when "ga:eventLabel"
        @label   = i
      when "ga:eventAction"
        @action  = i
      when "ga:totalEvents" 
        @total_events  = i
      when "ga:uniqueEvents"
        @unique_events = i
      end
    end 
    result["rows"].each do |array_data|
        GoogleAnalyticEvent.create(category:      array_data[@category], 
                                    label:        array_data[@label], 
                                    action:       array_data[@action],
                                    total_events:       array_data[@total_events],
                                    unique_events:      array_data[@unique_events],
                                    data_date:          @data_date)
    end
  end
end

Let’s me explain each code snapshots.

3-1: First of all, I initialize query parameters such as ids, start-date, …etc in instance variable @parameters.

PS. you can easily understand which parameters you need by using Google Analytics Query Explorer. Once, you find out which parameters you need, just copy to here like I do.

1
2
3
4
5
6
7
8
9
10
11
12

def get_event_from_ga_api(params={})
    @data_date = Date.today - 1
    @parameters = {
        'ids'         => "ga:12345678",
        'start-date'  => @data_date.strftime("%Y-%m-%d"),
        'end-date'    => @data_date.strftime("%Y-%m-%d"),
        'metrics'     => "ga:totalEvents,ga:uniqueEvents",
        'dimensions'  => "ga:eventCategory,ga:eventLabel,ga:eventAction",
        'max-results' => "10000"
      }
    query_paginate(@parameters)
  end

3-2: Second, let’s see sync_to_db first.

After I got Google API’s response, I will save to database.

let’s see what do we get from google analytics first.

Google show you the return data’s column header name by “columnHeaders”. And “rows” data is given as an array. And each array’s elements refer to “columnHeaders” respectively. For exameple, in above picture, you can see “ga:eventCategory” is in first of columnHeaders. So, in rows, each array’s first element is value of “ga:eventCategory”. Then I save data to model GoogleAnalyticEvent.

That’s all fuction “sync_to_db” do.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

def sync_to_db(result)
    result["columnHeaders"].each_with_index do |h,i|
      case h["name"].to_s
      when "ga:eventCategory"
        @category= i 
      when "ga:eventLabel"
        @label   = i
      when "ga:eventAction"
        @action  = i
      when "ga:totalEvents" 
        @total_events  = i
      when "ga:uniqueEvents"
        @unique_events = i
      end
    end 
    result["rows"].each do |array_data|
        GoogleAnalyticEvent.create(category:      array_data[@category], 
                                    label:        array_data[@label], 
                                    action:       array_data[@action],
                                    total_events:       array_data[@total_events],
                                    unique_events:      array_data[@unique_events],
                                    data_date:          @data_date)
    end
  end

3-3: Finally, let’s see function query_paginate.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

def query_paginate(parameters)
    parameters['access_token'] = get_token
    count = 0
    loop do
        if Time.new.to_i > @exp
          parameters['access_token'] = get_token
        end
       res = Typhoeus::Request.get('https://www.googleapis.com/analytics/v3/data/ga', 
        params: parameters
       )
      result = MultiJson.load(res.body)
      if result["error"].nil?
        sync_to_db(result)  if result["totalResults"] > 0
        max_count = result["totalResults"] / parameters["max-results"].to_i
      end
      break if count == max_count || result["error"].present?
      count += 1
      parameters["start-index"] = parameters["max-results"].to_i * count + 1
    end
  end

I have to get access token by “get_token”, And this function also assign expire time to @exp.

1

parameters['access_token'] = get_token

Then I write a loop for pagination. At first, I check token is expired or not, if is expired, I have to get token again.

1
2
3

if Time.new.to_i > @exp
  parameters['access_token'] = get_token
end

Now, I use Typhoeus::Request.get method with params fetching data from Google Analytics.

1
2
3
4

res = Typhoeus::Request.get('https://www.googleapis.com/analytics/v3/data/ga', 
        params: parameters
       )
result = MultiJson.load(res.body)

And if there are no any errors, I will save data to database.

1
2
3
4

if result["error"].nil?
  sync_to_db(result)  if result["totalResults"] > 0
  max_count = result["totalResults"] / parameters["max-results"].to_i
end

Finally, break or continue to get next batch data.

1
2
3

break if count == max_count || result["error"].present?
count += 1
parameters["start-index"] = parameters["max-results"].to_i * count + 1

That’s it.

Inspired by

1. http://www.jetthoughts.com/blog/tricks/2014/08/05/stress-testing-your-rails-application-using-jmeter.html
2. https://github.com/flood-io/ruby-jmeter
3. http://jmeter.apache.org/usermanual/get-started.html

In your local machine (Mac OSX)

1
2

brew update
brew install jmeter --with-plugins

In your remote server

1
2

# if your want test in remote server
sudo apt-get install jmeter 

Gemfile

write test plans for JMeter easily

1

gem ruby-jmeter

寫Test Plan

[Rereference] 語法支援哪些：

1. https://github.com/flood-io/ruby-jmeter/blob/5ae25cc32c8d05fcbe32bf143bdfbbd2d657517c/lib/ruby-jmeter/DSL.md

2. 可查看 alies_methods https://github.com/flood-io/ruby-jmeter/blob/5ae25cc32c8d05fcbe32bf143bdfbbd2d657517c/lib/ruby-jmeter/dsl.rb

Write A Stress Test Helper

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

# test/stress/stress_helper.rb

require 'ruby-jmeter'

def default_setting(params={})
  domain    = params[:domain]        || 'localhost' 
  protocol  = params[:protocol]      || 'https'
  port      = params[:port]          || '3000'
  cookie    = params[:cookie].to_s   || 'true'
  # defaults 會建立jmeter的 "HTTP Request Default"，讓你設定 domain或是 protocol...等數值
  defaults domain: domain, protocol: protocol, port: port

  # cookies 會建立jmeter的 "HTTP Cookie Manager"
  # 如果希望每個threads都有自己的cookie，要將此行code，寫到thread內
  # cookies policy 之前卡了我很久，見註一
  cookies policy: 'compatibility',clear_each_iteration: true unless params[:cookie] == 'false'
end

# Login Helper 
# 因為我們登入後會倒回首頁，所以，我有檢查 使用者的名稱是否有出現
def login(email,pwd,nickname)
  visit name: 'Sign In', url: '/users/sign_in' do
    # 會抓變數，並且將值assign給 'csrf-token' 與 'csrf-param'
    extract name: 'csrf-token', xpath: "//meta[@name='csrf-token']/@content", tolerant: true
    extract name: 'csrf-param', xpath: "//meta[@name='csrf-param']/@content", tolerant: true
  end

  # For devise issue
  http_header_manager name: 'X-CSRF-Token', value: '${csrf-token}'

  # 送出登入
  submit name: 'Submit Sign In form', url: '/users/sign_in',
    always_encode: true,
    fill_in: {
      'utf8'          => '✓',
      '${csrf-param}' => '${csrf-token}',
      'user[email]'        => email,
      'user[password]'     => pwd,
      'commit'             => 'Log in'
    } do
    # 登入後會倒回首頁，檢查是否包含你的nickname
    assert contains: nickname, scope: 'main'
  end
end

註一： cookies policy 之前卡了我很久，因為我發現我登入都會錯誤！都是csrf-token 不相符，我的認知，那是因為Devise使用cookie作為判斷機制，所以cookie policy不能選用rfc2109，要選用compatibility 他會符合目前幾乎所有線上網站的cookie需求！

Ruby-Jmeter <-> Jmeter

Non Login User Stress Test Plan

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63

# test/stress/non_login.rb

require './stress_helper' 
 
test do
  default_setting domain: 'your.awesome.website', portocal: 'https' , port: '443'
  #default_setting() # 本機測試使用
  
  # 壓測，登入後的行為
  # * 使用threads是因為 thread 代表jmeter裡面的thread_group，thread_group就是告訴jmeter
  #   有多少使用者要模擬，多久送一次requests，多少requests要被送出
  # * counts 代表 多少個使用者
  # * rampup 每個使用者的執行間隔
  # * loops 代表多重複做多少次測試
  
  for i in 1..10 
    threads count: 20, rampup: 1 ,loops: 1 do
      think_time 5000, 2000
        transaction "活動專區#{i}" do
          # 這邊會去爬 class: 'items' 底下每個 <a> 的href
        # 並且把他assign給 "act-urls"
          visit name: '活動專區首頁', url: "/activities?sort=1&page=#{Random.rand(1..4)}" do 
            extract name: 'act-urls', xpath: "//div[contains(@class,'items')]//a//@href", tolerant: true 
          end
          for i in 1..3
            visit name: "隨便點活動#{i}", url: "${act-urls_#{Random.rand(1..12)}}" do
              extract name: 'product-urls', xpath: "//div[contains(@class,'item-info')]//a//@href", tolerant: true 
            end
          end
        end
      view_results_tree
      debug_sampler
    end
  end 

  for i in 1..5
    threads count: 20, rampup: 1 ,loops: 1 do
      think_time 5000, 2000
        transaction "活動專區-標籤#{i}" do
          visit name: '活動專區首頁', url: "/activities" do 
            extract name: 'act-tag-urls', xpath: "//div[contains(@class,'keywords')]//a//@href", tolerant: true 
          end
          for i in 1..4
            visit name: "隨便點活動標籤#{i}", url: "${act-tag-urls_#{Random.rand(1..4)}}"
          end
        end
      view_results_tree
      debug_sampler
    end
  end 

  view_results_in_table
  graph_results
  aggregate_graph
  view_results_tree
  summary_report
 
end.run(
  gui: true,
  file: 'jmeter-visitor.jmx',
  log:  'jmeter-visitor.log',
  jtl:  'results-visit.jtl'
)

Ruby-Jmeter <-> Jmeter

Login User Stress Test

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

# test/stress/login.rb 

require './stress_helper' 

test do
  default_setting domain: 'your.awesome.website', portocal: 'https' , port: '443'
  # default_setting() # 本機測試使用

  # 壓測，登入後的行為
  threads count: 10, rampup: 1, loops: 1 do
    think_time 5000, 2000
 
    transaction 'Log In and View page' do
 
     login('your@login.account','your-password','your-nickname')

    # 這邊會去爬 class: 'ucnh-fs-ls-uctrls' 底下每個 <a> 的href
    # 並且把他assign給 "users-urls"
      visit name: 'activity', url: '/activities' do 
        extract name: 'users-urls', xpath: "//div[contains(@class,'ucnh-fs-ls-uctrls')]//a//@href", tolerant: true 
      end
  
  # 因為users-urls是個 array，用法就是 ${users-url_X}, X代表array第X個element
      visit name: 'user list1', url: '${users-urls_1}'
      visit name: 'user lists2', url: '${users-urls_3}'
      visit name: 'user lists3', url: '${users-urls_4}'
      visit name: 'user profile', url: '${users-urls_5}'
      visit name: 'user account edit', url: '${users-urls_6}'
    end
    debug_sampler # 給Debug用
    view_results_tree
  end
  view_results_in_table
  graph_results
  aggregate_graph
  view_results_tree
  summary_report
 
end.run(
  gui: true,
  file: 'jmeter-after-sign-in.jmx',
  log:  'jmeter-after-sign-in.log',
  jtl:  'results-after-sign-in.jtl'
)

Ruby-Jmeter <-> Jmeter

開始測試

1
2
3
4

cd your_awesome_projects/test/stress
# ruby 你要執行的.rb檔，ex: 
ruby non_login.rb
# 此時會跳出一個apache jmeter的畫面，點選綠色執行按鈕即可

Others

Remove tracking jmeter generated files

1
2

echo "/test/stress/*.log" >> .gitignore
echo "/test/stress/*.jmx" >> .gitignore

1. PERFORMANCE TESTING RAILS AGAINST REAL DATA

2. Performance Testing

3. http://railscasts.com/episodes/368-miniprofiler

First thing you should know

In Rails 4, performance test functionality was extracted out ! So, you need to add

# Gemfile
gem 'rails-perftest' ,group: :benchmark

And if you want your rails-perftest show memory usage. You also need to add

# Gemfile
gem 'ruby-prof' ,group: :benchmark

[! Important !] In my environment,

rails-4.1.6, ruby-2.2.2.133, x86_64-darwin14

After doing lots of efforts, I finally can get memory usage info.

But I fount this issue. please check: https://github.com/ruby-prof/ruby-prof/issues/165.

And I’m not sure that my memory usage is correct or not.

But I’ll still show you how I do these jobs. Maybe after fixing the issue, we can get correct informantion.

Create a new environment for performance test

create a new file: config/environments/benchmark.rb

In fact, all contents are copied from config/environments/production.rb since we need to know performance in real world.

Rails.application.configure do
  # Settings specified here will take precedence over those in config/application.rb.

  # Code is not reloaded between requests.
  config.cache_classes = true

  # Eager load code on boot. This eager loads most of Rails and
  # your application in memory, allowing both threaded web servers
  # and those relying on copy on write to perform better.
  # Rake tasks automatically ignore this option for performance.
  config.eager_load = true

  # Full error reports are disabled and caching is turned on.
  config.consider_all_requests_local       = false
  config.action_controller.perform_caching = true

  # Enable Rack::Cache to put a simple HTTP cache in front of your application
  # Add `rack-cache` to your Gemfile before enabling this.
  # For large-scale production use, consider using a caching reverse proxy like nginx, varnish or squid.
  # config.action_dispatch.rack_cache = true

  # Disable Rails's static asset server (Apache or nginx will already do this).
  config.serve_static_assets = false

  # Compress JavaScripts and CSS.
  config.assets.js_compressor = :uglifier
  # config.assets.css_compressor = :sass

  # Do not fallback to assets pipeline if a precompiled asset is missed.
  config.assets.compile = false

  # Generate digests for assets URLs.
  config.assets.digest = true

  # `config.assets.precompile` and `config.assets.version` have moved to config/initializers/assets.rb

  # Specifies the header that your server uses for sending files.
  # config.action_dispatch.x_sendfile_header = "X-Sendfile" # for apache
  # config.action_dispatch.x_sendfile_header = 'X-Accel-Redirect' # for nginx

  # Force all access to the app over SSL, use Strict-Transport-Security, and use secure cookies.
  config.force_ssl = true

  # Set to :debug to see everything in the log.
  config.log_level = :info

  # Prepend all log lines with the following tags.
  # config.log_tags = [ :subdomain, :uuid ]

  # Use a different logger for distributed setups.
  # config.logger = ActiveSupport::TaggedLogging.new(SyslogLogger.new)

  # Use a different cache store in production.
  # config.cache_store = :mem_cache_store
  config.cache_store = :dalli_store

  # Enable serving of images, stylesheets, and JavaScripts from an asset server.
  # config.action_controller.asset_host = "http://assets.example.com"

  # Ignore bad email addresses and do not raise email delivery errors.
  # Set this to true and configure the email server for immediate delivery to raise delivery errors.
  # config.action_mailer.raise_delivery_errors = false

  # Enable locale fallbacks for I18n (makes lookups for any locale fall back to
  # the I18n.default_locale when a translation cannot be found).
  config.i18n.fallbacks = true

  # Send deprecation notices to registered listeners.
  config.active_support.deprecation = :notify

  # Disable automatic flushing of the log to improve performance.
  # config.autoflush_log = false

  # Use default logging formatter so that PID and timestamp are not suppressed.
  config.log_formatter = ::Logger::Formatter.new

  # Do not dump schema after migrations.
  config.active_record.dump_schema_after_migration = false

end

add configuration: config/database.yml

default: &default
  adapter: mysql2
  encoding: utf8
  host: 127.0.0.1
  username: root
  password: your_password

staging:
  <<: *default
  database: your_project_development

development:
  <<: *default
  database: your_project_development

test:
  <<: *default
  database: your_project_test

production:
  <<: *default
  database: your_project_production

benchmark:
  <<: *default
  database: your_project_production 

Create a test helper file to setup your benchmark tests:

create a new file: test/benchmark_helper.rb

ENV["RAILS_ENV"] = "benchmark"
require File.expand_path('../../config/environment', __FILE__)
require 'rails/test_help'
require 'rails/performance_test_help'

class ActionDispatch::PerformanceTest
   self.profile_options = { runs: 5, metrics: [:wall_time, :memory,:objects, :gc_runs, :gc_time],
                            output: 'tmp/performance', formats: [:flat, :graph_html, :call_tree, :call_stack] }
end

Create a test file

rails generate performance_test page

and it will create a file named page.rb under test/performance/

require 'benchmark_helper'

class PageTest < ActionDispatch::PerformanceTest
  test "homepage" do
    get '/'
  end
end

Add a rake task for running your real-world benchmarks (optional)

lib/tasks/test_benchmark.rake

Rake::TestTask.new(:real_world_benchmark => ['test:benchmark_mode']) do |t|
    t.libs << 'test'
    t.pattern = 'test/performance/**/*_test.rb'
  end
end

Update your Rakefile

Rails 4 no longer defines db:test:prepare, however, rails-perftest still use test:prepare. So, we need to workaround.

require File.expand_path('../config/application', __FILE__)

Rails.application.load_tasks

Rake.application.instance_eval do
  # Remove test:prepare
  @tasks['test:benchmark'].prerequisites.shift if @tasks['test:benchmark']
  @tasks['test:profile'].prerequisites.shift if @tasks['test:profile']
end

For memory measurement, we need the railsexpress patches.

https://github.com/skaes/rvm-patchsets

rvm get stable

But my rvm isn’t update to date correctly, So I use

cd /tmp
git clone https://github.com/skaes/rvm-patchsets.git
cd rvm-patchsets
./install.sh

Then I run

rvm reinstall 2.2.2 --patch railsexpress

After reinstall Ruby, we need to recreate our gemset

cd your_project
rvm gemset create your_project
gem install bundler
bundle install

Finally

These command should work~

RAILS_ENV=benchmark bundle exec rake test:real_world_benchmark
RAILS_ENV=benchmark bundle exec rake test:benchmark
RAILS_ENV=benchmark bundle exec rake test:profile

Then these command will create some files under your_project/tmp/performance

PS. Please notice that before performance test, you should make sure that there are no any errors while visit the page.

Without using “rails-perftest”, you might use..

if You need detailed information, you can use “request_profiler”

inspired by https://www.coffeepowered.net/2013/08/02/ruby-prof-for-rails/

we can use request_profiler, which allows you to profile rack requests using ruby-prof.

gem 'request_profiler'

And you might need to add this line into config/enviornments/benchmark.rb

config.middleware.use "Rack::RequestProfiler"

On every page you want to profile, just add ?profile_request=true

for example, we want to profile “/” , just key

https://localhost:3000/?profile_request=true

then you wiil see file named 2015-07-14-16-49-19—profile_request=true.html under tmp/performance/

if You need to quickly and easily find out what is bottleneck, “rack-mini-profiler” is best choice.

Gemfile

gem 'rack-mini-profiler', require: false  

Create a new file config/initializers/rack_profiler.rb

if Rails.env == 'development' || Rails.env == 'benchmark'
  require 'rack-mini-profiler'

  # initialization is skipped so trigger it
  Rack::MiniProfilerRails.initialize!(Rails.application)
end

Add to app/controllers/application_controller.rb

before_action :authorize_for_miniprofiler
def authorize_for_miniprofiler
  if Rails.env == 'development' || Rails.env == 'benchmark'
    Rack::MiniProfiler.authorize_request
  end
end

now you can see this under development and benchmark environments

You might want to read

1. http://code.oneapm.com/ruby/2015/04/08/ruby-profilers/

1. http://railscasts.com/episodes/182-cropping-images-revised
2. http://stackoverflow.com/questions/12762728/how-to-crop-image-on-upload-with-rails-carrierwave-and-minimagick
3. https://coderwall.com/p/e9d_ja/using-carrierwave-uploader-for-tableless-model-in-rails
4. http://stackoverflow.com/questions/24262388/carrierwave-processing-only-after-the-model-has-been-saved-model-id-is-nil

What I done in this article

1. Click Upload Image

2. Choose Image And Preview

3. Edit Avatar And Preview Image Synchronously

4. Upload to AWS S3

5. Image Cropped And Uploaded

Steps

0. Description

I use Devise for sign up, Carrierwave for upload image and Jcrop for image selection. There is an User model and user has one UserImage model. Carrierwave uploader is mounted on UserImage. In this tutorial, I’ll not go through upload things. You can find lots of resources such as carrierwave-file-uploads. Also, I’ll not show all devise registration and popup(modal) things. Still, there are many resources on the Internet. Ok, let’s go.

1. Gemfile

I use jcrop for cropping image. So, add jcrop library to Gemfile.

gem 'jcrop-rails-v2'

ps. don’t forgot bundle install

2. Model

In User model, I only need to tell that User has a main image and it refer to UserImage model, and user signup form contains UserImage’s attributes, therefore, User model should accept these nested attributes. And I create a method for building image used in upload form(you can see usage in view)

app/models/user.rb

class User < ActiveRecord::Base
  ...
  has_one :main_image, :class_name => "::UserImage"
    accepts_nested_attributes_for :main_image, reject_if: :all_blank, allow_destroy: true


    def input_main_image
      self.main_image ||= self.build_main_image
      end
  ...
  end

UserImage model has “file” attribute which is used for upload, and we use carrierwave as our uploader, we need to mount this uploader to “file” attribute. [!! Important !!] I guess due to Rails work flow, when UserImageUploader upload image, the UserImage custom attributes: crop_x, crop_y, crop_w and crop_h will not available to UserImageUploader that we can’t crop image as we want. So we need to skip upload callback first, and we will upload after user is saved.

Ps. if your attribute used for mount uploader named “hello”,

mount_uploader :file, UserImagesUploader
skip_callback :save, :after, :store_file!

will be

mount_uploader :hello, UserImagesUploader
skip_callback :save, :after, :store_hello!

app/models/user_image.rb

# == Schema Information
#
# Table name: user_images
#
#  id                 :integer          not null, primary key
#  user_id            :integer
#  file               :string(255)
#  created_at         :datetime
#  updated_at         :datetime
#

class UserImage < ActiveRecord::Base
  belongs_to :user

    # Used for user avatar image edit
    attr_accessor :crop_x, :crop_y, :crop_w, :crop_h

    mount_uploader :file, UserImagesUploader

    # We need to upload after all attributes assigned，so we skip upload callback first
    skip_callback :save, :after, :store_file!  
end

3. View

In first picture, we have a form for sign up. Due to I have to upload an user image, we need to build it first. Then, UserImage’s attributes need to be filled. And I will use some javascript methods for editing so I write some div elements such as “fake-button-upload”, “fake-button-edit”. Also, I need to popup a image for user editing. Below “.uploaded” codes are used for popup. Finally, you can see I setup some ids, yap~ for javascript.

app/views/users/registrations/new.html.slim

.upload_personal_image
  = f.fields_for :main_image, resource.input_main_image do |build|
    = build.file_field :file, class: "personal-image-upload-file"
    .preview-outer
      = image_tag(resource.main_image.file, id: "preview") if resource.main_image
    .personal-image-upload.icon-default-photo-150x150
    .fake-button-upload 上傳個人照
    .fake-button-edit 編輯大頭照

    .uploaded
      .edit-title 編輯個人照
      = image_tag(resource.main_image.file, id: "preview-edit") if resource.main_image
      .uploaded-image-edit-attr
        - for attribute in [:crop_x, :crop_y, :crop_w, :crop_h]
          = build.text_field attribute, :id => attribute
      .edit-desc 請將想要顯示的範圍拖曳至中心圓圈處！
      .fake-button-edit-ok 設為個人照

4. CSS

app/assets/stylesheets/front/users/registrations.css.coffee

First we need to require jquery.Jcrop. And, I hope that our crop selector is a circle so I add….

.jcrop-holder div
{
    -webkit-border-radius: 50% !important;
    -moz-border-radius: 50% !important;
    border-radius: 50% !important;
    margin: 0 auto;
    // opacity: 0.6 !important;
}

You can ignore them but you should notice that some classes are display none such as .personal-image-upload-file, .preview-outer and .uploaded. These classed will show by javascript control.

//=require jquery.Jcrop

...

.jcrop-holder div
{
    -webkit-border-radius: 50% !important;
    -moz-border-radius: 50% !important;
    border-radius: 50% !important;
    margin: 0 auto;
}

.personal-image-upload-file {
    display: none;
  }

  .preview-outer {
    overflow: hidden;
      display: none;
      width: 150px;
      height: 150px;
      position: absolute;
      left: 0px;
      top: 0px;
      #preview{
        width: 150px;
        height: 150px;
      }
   }
   .uploaded {
  background-color: #FFFFFF;
  text-align: center;
  display: none;
  .edit-title {
    font-size: 20px;
    margin: 0px auto 20px auto;
    padding-top: 30px;
  }
  img {
    border: 1px dashed #dd406f;
    z-index: 2;
  }
  .edit-desc{
    margin-top: 10px;
    margin-bottom: 10px;
    font-size: 14px;
  }

  .uploaded-image-edit-attr{
    display: none;
  }
  .fake-button-edit-ok{
    width: 100px;
    height: 30px;
    font-size: 16px;
    display: inline-block;
    line-height: 30px;
    cursor: pointer;
    background-color: #DD406F;
    border-radius: 3px;
    color: #FFFFFF;
    text-decoration: none;
    margin-bottom: 30px;
  }
}

5. Javascript -> most important part for cropping image and previewing image.

First, we need to require jquery.Jcrop

#= require jquery.Jcrop

If user click .personal-image-upload icon or .fake-button-upload button, I’ll trigger real upload file click.

$('.personal-image-upload').click -> 
  $('.personal-image-upload-file').trigger('click')

$('.fake-button-upload').click ->
  $('.personal-image-upload-file').trigger('click')

How to preview image?

First, I have to know original image size since if user don’t resize image, I still need to send crop_x, crop_y, crop_w and crop_h to carrierwave crop function. Second, after FileReader read an image, I’ll set file path to element #preview and #preview-edit src attribute. Then, I’ll hide upload icon and show edit button.

initPreviewImage = ->    
    readURL = (input) ->
      if input.files and input.files[0]
        reader = new FileReader

        reader.onload = (e) ->
          img = new Image
          img.onload = ->
            $('#crop_x').val(0);
            $('#crop_y').val(0);
            $('#crop_w').val(img.width);
            $('#crop_h').val(img.height);
          img.src = e.target.result

          $('.preview-outer').css("display": "inline-block")
          $('#preview').attr 'src', e.target.result
          $('#preview-edit').attr 'src', e.target.result
          $('.personal-image-upload').hide()
          $('.fake-button-edit').css("display": "block")
          $('.fake-button-edit').css("right": "0px")
          $('.fake-button-upload').css("left": "0px")

        reader.readAsDataURL input.files[0]

    $('#user_main_image_attributes_file').change ->
      readURL this
 initPreviewImage();

At last, the cropping part. I setup three global variables for jcrop_api and image size(boundx, boundy). When the element .fake-button-edit button is clicked, I will call showUserImageEdit() and call my modal method for popup function. In showUserImageEdit function, if jcrop_api already existed(user choose a file already), I have to use jcrop_api to reset new image. And #preview-edit is used for binding jcrop, when user select new area or move selection, it will call update_crop function to update crop_x, crop_y, crop_w and crop_h(don’t forgot these attributes will send to carrierwave uploaded for cropping) and I will update #preview image as well.

Ps. you might ask why rx = 150 / coords.w and ry = 150 / coords.h ? The answer is, I give preview size 150x150px in CSS.

jcrop_api = undefined
  boundx = undefined
  boundy = undefined

  showUserImageEdit = ->
    if jcrop_api
      jcrop_api.setImage($('#preview').attr('src'));

    update_crop = (coords) ->
      rx = 150 / coords.w
      ry = 150 / coords.h
      $('#preview').css
        width: Math.round(rx * boundx) + 'px'
        height: Math.round(ry * boundy) + 'px'
        marginLeft: '-' + Math.round(rx * coords.x) + 'px'
        marginTop: '-' + Math.round(ry * coords.y) + 'px'
      $('#crop_x').val(Math.floor(coords.x));
      $('#crop_y').val(Math.floor(coords.y));
      $('#crop_w').val(Math.floor(coords.w));
      $('#crop_h').val(Math.floor(coords.h));


    $('#preview-edit').Jcrop({
        bgOpacity: 0.4, 
        bgColor: 'black',
        onChange: update_crop,
        onSelect: update_crop,
        allowSelect: true,
        allowResize: true,
        setSelect: [50, 0, 150, 150],
        aspectRatio: 1
      }, -> 
        bounds = this.getBounds();
        boundx = bounds[0];
        boundy = bounds[1];
        jcrop_api = this);
    $('.uploaded').show()
    $('.fake-button-edit-ok').click -> 
      myModal().setTarget($('.uploaded')).close()
      jcrop_api.release()

  initPreviewImageEdit = ->
    $('.fake-button-edit').click -> 
      showUserImageEdit();
      myModal().setTarget($('.uploaded')).open()
  initPreviewImageEdit();

whole code app/assets/javascripts/users/registrations.js.coffee

#= require jquery.Jcrop
jQuery ->
  $(document).on 'ready page:load', -> 

    $('.personal-image-upload').click -> 
      $('.personal-image-upload-file').trigger('click')

    $('.fake-button-upload').click ->
      $('.personal-image-upload-file').trigger('click')

    initPreviewImage = ->    
      readURL = (input) ->
        if input.files and input.files[0]
          reader = new FileReader

          reader.onload = (e) ->
            img = new Image
            img.onload = ->
              $('#crop_x').val(0);
              $('#crop_y').val(0);
              $('#crop_w').val(img.width);
              $('#crop_h').val(img.height);
            img.src = e.target.result

            $('.preview-outer').css("display": "inline-block")
            $('#preview').attr 'src', e.target.result
            $('#preview-edit').attr 'src', e.target.result
            $('.personal-image-upload').hide()
            $('.fake-button-edit').css("display": "block")
            $('.fake-button-edit').css("right": "0px")
            $('.fake-button-upload').css("left": "0px")

          reader.readAsDataURL input.files[0]

      $('#user_main_image_attributes_file').change ->
        readURL this
    initPreviewImage();

    jcrop_api = undefined
    boundx = undefined
    boundy = undefined

    showUserImageEdit = ->
      if jcrop_api
        jcrop_api.setImage($('#preview').attr('src'));

      update_crop = (coords) ->
        rx = 150 / coords.w
        ry = 150 / coords.h
        $('#preview').css
          width: Math.round(rx * boundx) + 'px'
          height: Math.round(ry * boundy) + 'px'
          marginLeft: '-' + Math.round(rx * coords.x) + 'px'
          marginTop: '-' + Math.round(ry * coords.y) + 'px'

        $('#crop_x').val(Math.floor(coords.x));
        $('#crop_y').val(Math.floor(coords.y));
        $('#crop_w').val(Math.floor(coords.w));
        $('#crop_h').val(Math.floor(coords.h));


      $('#preview-edit').Jcrop({
          bgOpacity: 0.4, 
          bgColor: 'black',
          onChange: update_crop,
          onSelect: update_crop,
          allowSelect: true,
          allowResize: true,
          setSelect: [50, 0, 150, 150],
          aspectRatio: 1
        }, -> 
          bounds = this.getBounds();
          boundx = bounds[0];
          boundy = bounds[1];
          jcrop_api = this);

      $('.uploaded').show();

      $('.fake-button-edit-ok').click -> 
        myModal().setTarget($('.uploaded')).close()
        jcrop_api.release()

    initPreviewImageEdit = ->
      $('.fake-button-edit').click -> 
        showUserImageEdit();
        myModal().setTarget($('.uploaded')).open()
    initPreviewImageEdit();

6. Carrierwave uploader

As you can see I create a new version named _customize and it only do its job when model(UserImage model) has crop_x, crop_y, crop_w and crop_h attributes. And I use minimagick crop method to crop image before upload.

class UserImagesUploader < CarrierWave::Uploader::Base

  include CarrierWave::MiniMagick

  storage :fog

  ...

  version :_customize, :if => :customize? do 
    process :crop_img
    resize_to_fill(150, 150)
  end

  def customize? picture
    !model.crop_x.blank? && !model.crop_y.blank? && !model.crop_w.blank? && !model.crop_h.blank?
  end

  def crop_img
    if !model.crop_x.blank? && !model.crop_y.blank? && !model.crop_w.blank? && !model.crop_h.blank?
      manipulate! do |img|
        x = model.crop_x.to_i
        y = model.crop_y.to_i
        w = model.crop_w.to_i
        h = model.crop_h.to_i
        img.crop("#{w}x#{h}+#{x}+#{y}")
        img
      end
    end
  end

  ...

end

7. Final step: Upload Image

Do you remember that I skip upload callback in app/models/user_image.rb ?

skip_callback :save, :after, :store_file!  

So, when should we do upload job?

Answer is after user save !

class Users::RegistrationsController < Devise::RegistrationsController
  after_filter :upload_avatar, :only => :create

  ... 

  protected

  def upload_avatar
    if resource.persisted? && resource.main_image # user is created successfuly
      resource.main_image.store_file! 
      resource.main_image.file.recreate_versions!
    end
  end

  ... 
end

8. Don’t forgot …

Since we use Devise for sign up, and we use nested attributes, we need to tell Devise permitted parameters.

app/controllers/applicatio.rb

class ApplicationController < ActionController::Base
  ...
  before_action :configure_permitted_parameters, if: :devise_controller?

  private

  def configure_permitted_parameters
    devise_parameter_sanitizer.for(:sign_up) { |u| u.permit(:email, :password, :password_confirmation, main_image_attributes: ['id', '_destroy', 'file', 'crop_x', 'crop_y', 'crop_w', 'crop_h'])}
  end 
  ... 
end

舉例來說：

在browser console下

escape("台中市")
> "%u53F0%u4E2D%u5E02"

但是在rails c 下

CGI.escape("台中市")
> "%E5%8F%B0%E4%B8%AD%E5%B8%82"

Why?

查了一下Google，發現是因為在 js 在escape中文時，是將他編成 unicode

所以，

javascript 如果要送中文字給Ruby 請使用

encodeURIComponent("台中市")
> "%E5%8F%B0%E4%B8%AD%E5%B8%82"

但是，往往沒辦法你不能去改人家的javascript，你只好…. javascripts escape string to UTF-8

查到的作法如下：

在rails c 下

unicode_str = "%u53F0%u4E2D%u5E02"

unicode_str.gsub(/\%u([\da-fA-F]{4})/) {|m|  [$1].pack('H*').unpack('n*').pack('U*')}

=> "台中市"

參數解釋：

因為 unicode 的字串都是由 %u 開頭，外加上 4個可能是數字可能是 小寫a~f 或是 大寫A~F，所以我們先透過 gsub 這個function將符合的字抓出來處理

抓出來字之後，我們要使用 pack 和 unpack 方法，來將字進行解碼、編碼

我們先抓一個字來看，就以 “台” 為例， (ps. pack 只能用在array, unpack可以用在string)

gsub 會將 “53F0” 丟進去處理

["53F0"].pack('H*')

  => "S\xF0"

# H: 代表了將他pack組成16進位字(hex string (high nibble first))

"S\xF0".unpack('n*')

=> [21488]

# n: 他會return一個 Integer，他代表了16-bit unsigned, network (big-endian) byte order

[21488].pack('U*')

=> "台"

# U: 將16位元NBO組成UTF-8

REF:
UNESCAPE編碼錯誤

那反過來呢，要如何將UTF-8的字，做成像 javascript escape 後的結果

在rails c下

return_str = ""
"台中市".each_char { |c| return_str += "%u#{c.unpack("U*").pack("n*").unpack("H*").first}" }


return_str 
> "%u53f0%u4e2d%u5e02"

假設，你現在要串接的使用Big5寫的API，你發現他接收的parameters 居然是%A5%78%A4%A4%A5%AB

因為他default是接受，用 javascript escape Big5編碼的字

所以你要將你UTF-8的字，轉成符合他格式的字

這時候你要這樣做：

  return_str = ""
  Iconv.conv("BIG5", "UTF8", str).unpack("H*").first.scan(/../).each do |s|
    return_str += "%#{s}"
  end

一樣來看說明：

Iconv.conv("BIG5", "UTF8", "台中市")

=> "\x{A578}\x{A4A4}\x{A5AB}"


Iconv.conv("BIG5", "UTF8", "台中市").unpack("H*")

=> ["a578a4a4a5ab"]

Iconv.conv("BIG5", "UTF8", "台中市").unpack("H*").first.scan(/../)

=> ["a5", "78", "a4", "a4", "a5", "ab"]

打完收工！

還是，你是個後端工程師，無時無刻，要去看一下mysql是否還活著，apache2是否快掛了

還有，你是個全端工程師，雖然你已經強的跟神一樣，但是你就是不想要寫的script自動去重啟的service

這時候，你需要的是Monit

他可以幫你監控你的daemon processes、檔案系統、Network、Space

如果想要看完整的設定文件請點 Monit Documentation

整理一下，我的使用的感覺

1. 使用DSL，所以用起來很直覺
2. 使用 # 當作註解
3. 不同的監控，可以有不同的動作(ex: alert, restart, stop, UNMONITOR)
4. 彈性設定時間來監控
5. 彈性設定幾次重啟後，就不要再重啟了
6. 可以用if, else …等來做動作判斷！
7. 提供網頁GUI，讓你很好的操作或是看伺服器狀態！
8. 簡單來說，超多客製化應用！網路上也滿多資源可以參考！

但是，如果你跟我一樣是個懶人，這時候，你就需要看這篇文章

ps. 這是我剛設定好的樣子，等我有發現哪裡有問題，會再上來更新XD

Step 1. 安裝Monit

sudo apt-get install monit

Step 2. 設定他

ps. 我一開始看文件，傻傻的以為設定在 ~/.monitrc，一切都會work，但是…經過我約莫半小時的測試…我放棄了～ 我還是把它寫在

sudo vim /etc/monit/monitrc

我改了哪些東西

# 設定Monit多久監控一次
set daemon 60

#設定log放在哪裡
set logfile /var/log/monit.log

#設定 mail server 
set mailserver 你的.postfix.伺服器 port 25

# ps. 如果你的mail server是在其他地方，譬如AWS SES
# set mailserver email-smtp.us-east-1.amazonaws.com port 587
#   username "amazon_username" password "amazon_password"
#   using TLSV1
#   with timeout 30 seconds

# Email格式
set mail-format {
      from: monit@alohacc.cc
  reply-to: y.alohac@gmail.com
      subject: [Aloha系統通知] $SERVICE $EVENT at $DATE
      message: Monit $ACTION $SERVICE at $DATE on $HOST: $DESCRIPTION.
         Hi, system now is waiting your attention~~~ come on baby~~
         By Aloooooooooooooooooooooha
    }

    #設定收到所有異常通知的人，貌似目前不支援一次設定很多....所以要key很多遍
set alert aloha.chen@itrue.com.tw
set alert y.alohac@gmail.com

# 監控主機CPU、Memory、Swap以及Uptime的數值
check system ccaloha.cc
    if loadavg (1min) > 4 then alert
    if loadavg (5min) > 2 then alert
    if memory usage > 75% then alert
    if swap usage > 25% then alert
    if cpu usage (user) > 70% then alert
    if cpu usage (system) > 30% then alert
    if cpu usage (wait) > 20% then alert


    # 起Web服務
 set httpd port 2812 and
    use address 0.0.0.0      # 我要讓他可以對外連線
    allow localhost           # 開放localhost可以連
    #allow 0.0.0.0/0.0.0.0   # 開放所有IP都可以連得到
    allow 你的.ip.位置        # 只開放你的ip可以連  
    allow 可登入帳號:他的密碼 

    # 監控Apache2
check process apache with pidfile /var/run/apache2/apache2.pid
    start program = "/etc/init.d/apache2 start" with timeout 60 seconds
    stop program  = "/etc/init.d/apache2 stop"
    if cpu > 50% for 2 cycles then alert
    #if TOTAL CPU is greater than 80% for 5 cycles then restart
    if mem > 100 MB for 5 cycles then stop # 網路上參考下來的寫法，但是，monit -t 會報錯
    if failed port 80 for 2 cycles then restart
    #if failed port 443 for 2 cycles with timeout 15 seconds then restart # 網路上參考下來的寫法，但是，monit -t 會報錯
    if failed port 443 for 2 cycles then restart

# 監控mysql
check process mysqld with pidfile /var/run/mysqld/mysqld.pid
    start program = "/etc/init.d/mysql start"
    stop program = "/etc/init.d/mysql stop"
    if failed host 127.0.0.1 port 3306 protocol mysql then alert
    if failed host 127.0.0.1 port 3306 protocol mysql then restart
    if 7 restarts within 10 cycles then unmonitor

# 監控硬碟空間使用
check filesystem root-filesystem with path /dev/xvda1
    if space usage > 80% for 5 times within 15 cycles then alert

啟動它

1. 檢查config 檔是否正確

    sudo monit -t 

2. 啟動

    sudo monit 

3. 如果你改了設定檔，你可以這樣重啟

    sudo monit reload

Web Gui

設定完成後，你就可以看到

點擊一個監控的process，你可以發現，你可以直接透過網頁重啟歐！

另外，你也可以收到email歐

Plus, we usually use the copy of production database in stage server.

In some applications, we will send email to users for some purposes such as confirmation mail after user

register, reset password mail…etc.

And, in staging environment, we have to avoid our operators accidentally send email to real users.

Therefore, we have to implement some codes for filtering.

Let’s go

According to Action Mailer Basics, we can register an interceptor to avoid some unexpected actions.

And my requirement is I want staging server can send mail to our employees. In our case, our company mail address is xxx@itrue.com.tw. So, we have to implement code that only allow email server send to users like: XXX@itrue.com.tw.

Step1. Gemfile

gem 'settingslogic'

Step2. Add a lib on app/lib/settings.rb

class Settings < Settingslogic
    source "#{Rails.root}/config/application.yml"
    namespace Rails.env
end

Step3. Add your white list on application.yml

defaults: &defaults
  allowed_send_mail_domain: '@itrue.com.tw'

development:
  <<: *defaults

test:
  <<: *defaults

production:
  <<: *defaults

Step4. Write a interceptor on app/interceptor/staging_mail_interceptor.rb

ref: Filtering emails on staging

class StagingMailInterceptor

  def self.delivering_email(message)
    message.to = extract_allowed_recepients(message)
    message.perform_deliveries = false if message.to.empty?
  end

  private

  def self.extract_allowed_recepients(message)
    message.to.select { |address| allowed_address?(address) }
  end

  def self.allowed_address?(address)
    allowed_domains = Settings.allowed_send_mail_domain.split(',')

    matches_allowed = allowed_domains.count { |domain| address.include?(domain) }

    matches_allowed != 0
  end

end

Step5. Register interceptor to ActionMailer on app/config/initializers/sandbox_email_interceptor.rb

require 'staging_mail_interceptor'

if Rails.env.staging?
  ActionMailer::Base.register_interceptor(StagingMailInterceptor)
end

Done

i-TRUE 的目標是成為全世界最棒的專業型Database Media Company

我每週一早上負責跟同事們介紹Google Analytics，這是其中一個介紹投影片！

陸續更新中…

And you are able to learn

1. How I do Google API pagination
2. How I use module to build flexible function
3. How I use Google Query Explorer to speed up development
4. How to apply for google API access

Let’s check it out.

Step 1. Apply Google Analytics API access authorization

1-1 go to Google API console

1-2 Find Google Analytics API

1-3 Enable Google Analytics API

1-4 Create New Client ID

1-5 Choose client type

Since this is for server-to-server usage, I choose this

1-6 Download key

ps. Since I use Ruby on Rails, so I put it in config/ga_api_key_20150408.p12

1-7 Add API user into your GA user group

2. [TIPS] Use Google Query Explorer to speed up development

1. Go to Google Query Explorer

2. Get your view id

3. Try your query parameter first and know how your data look like

And it provides a easy tool that you don’t need to remember any parameters.

3. Let’s write a sample code

3-1 Add gem to Gemfile

gem 'google-api-client'

3-2 create a rb file

I update some codes copied from this page.https://gist.github.com/joost/5344705

  require 'google/api_client'
  require 'date'

  client = Google::APIClient.new(:application_name => 'Urcosme-GA-sync', :application_version => '1')

  key_file = File.join("#{Rails.root}/config", 'ga_api_key_20150408.p12')
  key = Google::APIClient::PKCS12.load_key(key_file, 'notasecret')
  service_account = Google::APIClient::JWTAsserter.new(
      'hello@developer.gserviceaccount.com',
      ['https://www.googleapis.com/auth/analytics.readonly', 'https://www.googleapis.com/auth/prediction'],
      key)
  client.authorization = service_account.authorize

  analytics = client.discovered_api('analytics', 'v3')

  parameters = {
        'ids'         => "ga:12345",
        'start-date'  => (Date.today - 30).strftime("%Y-%m-%d"),
        'end-date'    => Time.now.strftime("%Y-%m-%d"),
        'metrics'     => "ga:avgTimeOnPage",
        'filters'     => "ga:pagePath=~/"
      }
  result = client.execute(:api_method => analytics.data.ga.get, :parameters => parameters)

This sample code demonstrates how google-api-client work. However, I need to fetch different kinds of data such as specific pageviews, event, and campaign. I have to refactor it.

4. Let’s build a flexible code

4-1 create a module file: “ga.rb” in /lib/

In above code, it can be seperated to 2 parts. One is uniform part, the other is variable part.

Unchange part

@client = Google::APIClient.new(:application_name => 'Urcosme-GA-sync', :application_version => '1')
key_file = File.join("#{Rails.root}/config", 'ga_api_key_20150408.p12')
key = Google::APIClient::PKCS12.load_key(key_file, 'notasecret')
service_account = Google::APIClient::JWTAsserter.new(
          'hello@developer.gserviceaccount.com',
          ['https://www.googleapis.com/auth/analytics.readonly', 'https://www.googleapis.com/auth/prediction'],
          key)
@client.authorization = service_account.authorize
@analytics = @client.discovered_api('analytics', 'v3')

Change part. In this code, I already write some default parameters. But, it can be rewrite in other place.

@data_date = Date.today - 1
@parameters = {
    'ids'         => "ga:12345",
    'start-date'  => @data_date.strftime("%Y-%m-%d"),
    'end-date'    => @data_date.strftime("%Y-%m-%d"),
    'metrics'     => "",
    'dimensions'  => "",
    'max-results' => "10000"
}

And the whole code show as below.

lib/ga.rb

require 'google/api_client'
require 'date'
module GoogleAnalytic

  # 每一個GA API執行前，需要初始化
  def initialize_ga
    @client = Google::APIClient.new(:application_name => 'Urcosme-GA-sync', :application_version => '1')
    key_file = File.join("#{Rails.root}/config", 'ga_api_key_20150408.p12')
    key = Google::APIClient::PKCS12.load_key(key_file, 'notasecret')
    service_account = Google::APIClient::JWTAsserter.new(
      'hello@developer.gserviceaccount.com',
      ['https://www.googleapis.com/auth/analytics.readonly', 'https://www.googleapis.com/auth/prediction'],
      key)
    @client.authorization = service_account.authorize
    @analytics = @client.discovered_api('analytics', 'v3')
    @data_date = Date.today - 1
    @parameters = {
        'ids'         => "ga:12345",
        'start-date'  => @data_date.strftime("%Y-%m-%d"),
        'end-date'    => @data_date.strftime("%Y-%m-%d"),
        'metrics'     => "",
        'dimensions'  => "",
        'max-results' => "10000"
      }
  end

  # 將資料儲存到MongoDB，請複寫我
  def sync_to_db(result)
  end
end

4-2 let’s Rails application know our lib

config/application.rb

module MyAwesomeApp
  class Application < Rails::Application
    config.autoload_paths += %W(#{config.root}/lib)   end
end

4-3 create a model to save API data

Now I want to get all our campaign data. So I create …

ps. I use MongoDB as database.

app/models/ga_campaign.rb

class GACampaign
  include GoogleAnalytic
  include Mongoid::Document
  include Mongoid::Timestamps
  field :campaign_name, type: String      # 廣告活動名稱
  field :campaign_source, type: String    # 廣告活動來源    
  field :campaign_medium, type: String    # 廣告活動媒介
  field :campaign_content, type: String   # 廣告活動內容
  field :session, type: Integer
  field :pageview, type: Integer
  field :data_date, type: Date            # 原始資料的時間

  def get_campaign_from_ga_api
    initialize_ga
    @parameters['metrics'] = "ga:pageviews,ga:sessions"
    @parameters['dimensions'] = "ga:campaign,ga:source,ga:medium,ga:adContent"
    @parameters['filters'] = "ga:campaign!=(not set)"
    sync_to_db(@parameters)
  end

  def sync_to_db(parameters)
     request = {
        :api_method => @analytics.data.ga.get,
        :parameters => parameters
     }
    result = @client.execute(request)
    result.data.rows.each do |array_data|
        GACampaign.create(campaign_name:    array_data[0], 
                          campaign_source:  array_data[1], 
                          campaign_medium:  array_data[2],
                          campaign_content: array_data[3],
                          session:          array_data[4],
                          pageview:         array_data[5],
                          data_date:        @data_date)
    end
  end
end

But…. if result is more than max-result (according to google, maximun is 10000 see google doc)

How can I do paginate?

4-4 Google API pagination

Since, this method will be used in every “GA model” so I put it in lib/ga.rb

according to GA document

If not supplied, the starting index is 1. (Result indexes are 1-based. That is, the first row is row 1, not row 0.) Use this parameter as a pagination mechanism along with the max-results parameter for situations when totalResults exceeds 10,000 and you want to retrieve rows indexed at 10,001 and beyond.

here is my method. Not quiet good, but it’s working.

def query_paginate(parameters)
     request = {
      :api_method => @analytics.data.ga.get,
      :parameters => parameters
    }
    count = 0
    loop do
      result = @client.execute(request)
      sync_to_db(result)
      max_count = result.data.total_results / parameters["max-results"].to_i
      break if count == max_count
      count += 1
      request[:parameters]["start-index"] = parameters["max-results"].to_i * count + 1
    end
  end

let’s see full code.

require 'google/api_client'
require 'date'
module GoogleAnalytic

def initialize_ga
    @client = Google::APIClient.new(:application_name => 'Urcosme-GA-sync', :application_version => '1')
    key_file = File.join("#{Rails.root}/config", 'ga_api_key_20150408.p12')
    key = Google::APIClient::PKCS12.load_key(key_file, 'notasecret')
    service_account = Google::APIClient::JWTAsserter.new(
      'hello@developer.gserviceaccount.com',
      ['https://www.googleapis.com/auth/analytics.readonly', 'https://www.googleapis.com/auth/prediction'],
      key)
    @client.authorization = service_account.authorize
    @analytics = @client.discovered_api('analytics', 'v3')
    @data_date = Date.today - 1
    @parameters = {
        'ids'         => "ga:12345",
        'start-date'  => @data_date.strftime("%Y-%m-%d"),
        'end-date'    => @data_date.strftime("%Y-%m-%d"),
        'metrics'     => "",
        'dimensions'  => "",
        'max-results' => "10000"
      }
  end

  # 將資料儲存到MongoDB，請複寫我
  def sync_to_db(result)
  end

  # 如果API存取，外加換頁功能，因為GA一次request最多一萬筆資料
  def query_paginate(parameters)
     request = {
      :api_method => @analytics.data.ga.get,
      :parameters => parameters
    }
    count = 0
    loop do
      result = @client.execute(request)
      sync_to_db(result)
      max_count = result.data.total_results / parameters["max-results"].to_i
      break if count == max_count
      count += 1
      request[:parameters]["start-index"] = parameters["max-results"].to_i * count + 1
    end
  end
end

4-5 refactor app/models/ga_campaign.rb

class GACampaign
  include GoogleAnalytic
  include Mongoid::Document
  include Mongoid::Timestamps
  field :campaign_name, type: String      # 廣告活動名稱
  field :campaign_source, type: String    # 廣告活動來源
  field :campaign_medium, type: String    # 廣告活動媒介
  field :campaign_content, type: String   # 廣告活動內容
  field :session, type: Integer
  field :pageview, type: Integer
  field :data_date, type: Date            # 原始資料的時間

  def get_campaign_from_ga_api
    initialize_ga
    @parameters['metrics'] = "ga:pageviews,ga:sessions"
    @parameters['dimensions'] = "ga:campaign,ga:source,ga:medium,ga:adContent"
    @parameters['filters'] = "ga:campaign!=(not set)"
    query_paginate(@parameters)
  end

  def sync_to_db(result)
        result.data.rows.each do |array_data|
        GACampaign.create(campaign_name:    array_data[0], 
                          campaign_source:  array_data[1], 
                          campaign_medium:  array_data[2],
                          campaign_content: array_data[3],
                          session:          array_data[4],
                          pageview:         array_data[5],
                          data_date:        @data_date)
    end
  end
end

4-6 how to use

ga_campaign = GACampaign.new
ga_campaign.get_campaign_from_ga_api

4-7 other example: fetch GA Event data

The things I have to change are @parameters and sync_to_db

It’s easy, right~

app/models/ga_event.rb

class GAEvent
  include GoogleAnalytic
  include Mongoid::Document
  include Mongoid::Timestamps
  field :event_category, type: String      # 事件類別
  field :event_label, type: String         # 事件標籤
  field :event_action, type: String        # 事件動作
  field :total_events, type: Integer       
  field :unique_events, type: Integer
  field :data_date, type: Date            # 原始資料的時間

  def get_event_from_ga_api
    initialize_ga
    @parameters['metrics'] = "ga:totalEvents,ga:uniqueEvents"
    @parameters['dimensions'] = "ga:eventCategory,ga:eventLabel,ga:eventAction"
    @parameters['filters'] = "ga:eventCategory==test"
    query_paginate(@parameters)
  end

  def sync_to_db(result)
    result.data.rows.each do |array_data|
      GAEvent.create(event_category:      array_data[0], 
                     event_label:        array_data[1], 
                     event_action:       array_data[2],
                     total_events:       array_data[3],
                     unique_events:      array_data[4],
                     data_date:        @data_date)

    end
  end

end

1. 每個連結的點擊數

2. 開信率

Google Analytics

這是一個電子報，基本上屬於一種廣告！

那GA怎麼那麼厲害知道誰點了什麼？

該不會Google 大神，連Email也滲透？！

ps. 我猜他應該也差不多都知道我們在Email上的一舉一動，只要你用Gmail…

那GA要怎麼紀錄？

基本上就是將連結網址帶上一些”參數”，當使用者點了這個連結之後，就你就會被帶到那個連結網址，這時候那個網站一定有埋GA的javascript ，這就是他的Key~ GA透過js爬到你的網址，當他發現你有帶某些參數，他就知道你是從哪裡來的！

我剛剛提到了好幾次參數

沒錯！這就是關鍵！

讓我們來看一下 有哪些參數

GA在有文件說明網址要如何使用

而且，GA還很貼心的幫大家準備了 網址產生器

來看一下，實際上會產生什麼樣的連結：

假設我們追蹤

透過Email點擊，來到首頁的人

Url 原本是長這樣

www.urcosme.com

ps. urcosme.com小弟目前待的公司 XDDDDD廣告一下！

經過我的GA的巧手，它會長這樣

www.urcosme.com/?utm_source=%E9%9B%BB%E5%AD%90%E5%A0%B1&utm_medium=%E9%9B%BB%E5%AD%90%E9%83%B5%E4%BB%B6&utm_content=%E6%B8%AC%E8%A9%A6%E6%B8%AC%E8%A9%A6&utm_campaign=%E6%88%91%E5%A5%BD%E5%B8%A5

案….這是….!？

讓我幫你翻譯一下

www.urcosme.com/?utm_source=電子報&utm_medium=電子郵件&utm_content=測試測試&utm_campaign=我好帥

當我把連結改成這樣後，基本上使用者也是會點到你的首頁，但是對於GA而言，他就知道你是從電子郵件過來

但! 身為一個有責任感又假掰的IT，

我們當然希望讓系統自動幫行銷的同仁帶入這些參數，讓正妹同事覺得沒有你不行～

1. Rails 要如何實作“點擊追蹤”

其實很簡單，我們要寫一個 小小小爬蟲，把內容爬過一次

把有 <a> 的找出來，然後把資訊加進去就好了！

那…爬蟲要怎麼寫？

各位，既然是來到 Rails ，Rails什麼不多，鐵路最多，歐不～ 是輪子最多！

跟大家介紹個 Gem : Nokogiri

我們就直接用實作，來說明他可以幹麻

直接來看的程式碼：

這是我寫在 app/models/edm.rb 的code

這個model資訊是這樣

# == Schema Information
#
# Table name: edms
#
#  id                 :integer          not null, primary key
#  name               :string
#  send_at            :datetime
#  title              :string(255)
#  state              :string(255)
#  content            :text
#  created_at         :datetime
#  updated_at         :datetime
#


def parse_link_in_email(user_id)
    # 讀進email html
    html =  Nokogiri::HTML(self.content)
    # 找出所有 a 
    a_nodes = html.css('a')
    a_nodes.each do |a|
      # 抓出href 並且加上GA 追蹤
      tracking_a = a['href'] +  "?utm_source=#{self.name}-#{CGI.escape(a['href'])}" + 
                                "&utm_medium=email" +
                                "&utm_content=#{self.id}-#{user_id}" + 
                                "&utm_campaign=#{self.title}"
      a['href'] = tracking_a
    end
    return html.to_html
 end

幾個重點：

1. 使用Nokogiri::HTML(self.content) => 讀HTML進來
2. html.css(‘a’) => 抓出所有<a>
3. a[‘href’] => 抓出這個<a>，裡頭的屬性href
4. html.to_html => 轉回HTML

Done !

所以當使用者點擊信件的link時，GA就會看到…..

攬客 >> 廣告活動 >> 所有廣告活動

2. Rails 要如何實作“開信率”

開信率我是參考這篇Blog來實作

簡單來說，我們需要埋一個image tag，然而那個tag

會帶上一些參數，讓GA知道這幹嘛的！

為什麼要使用這種標籤的方式紀錄呢？

假設使用者打開信之後，並且點了一個連結，

因為這個點了連結的動作，我們知道他一定有開信，歐噎～ 做完了～

But………..

那如果他只有打開信，沒有點擊勒？ 如果他打開信，點了N個連結，那開信不就也被多紀錄了很多次？(ps.點擊紀錄是每點一次就紀錄一次)

所以我們必須透過從 Google Analytics “GET” 一個標記(就是圖片啦)，當我們跟GA要圖的時候，可以告訴GA一些參數，嘿嘿，這樣就有辦法紀錄了！

各位，又看到參數兩個字

那又有哪些可以用勒？ (這邊我只列我目前有在用的，詳細可以參考GA collections parameters)

那我們就不看sample，直接來看實作code

def parse_link_in_email(user_id)

    ... 

    body = html.at_css("body")
    img_node = Nokogiri::XML::Node.new("img",body)
    img_node['src'] = "https://www.google-analytics.com/collect?v=1" + 
                      "&tid=#{Settings.google_analytics_key}" + 
                      "&uid=#{user_id}"+
                      "&t=event" +
                      "&ec=email-#{self.name}-#{self.title}" + 
                      "&ea=open" + 
                      "&el=user_id-#{user_id}" + 
                      "&cs=#{self.name}" + 
                      "&cm=email" + 
                      "&cn=#{self.title}"
    body << img_node

    ...
end

幾個重點：

1. 使用Nokogiri::XML::Node.new(“img”,body) => 新增一個<img>標籤
2. img_node[‘src’] => 設定這個<img>，裡頭的屬性src
3. body << img_node => 將code埋到body裡面

那在GA你會看到什麼勒？

3. Rails Mailer、its View and model

剩下我沒提到的部份，不過這些就是基本寄信的功能，我就不贅述了

app/models/edm.rb

has_many :edm_user_ships  # 假設你有要寄信的清單
has_many :users, :through => :edm_user_ships
def send_mail
    self.users.each do |user|
        EdmMailer.delay.send_edm(self, user.email, user.id)
    end
end

app/mailers/edm_mailer.rb

class EdmMailer < ActionMailer::Base
  default from: "service@urcosme.com"

  def send_edm(e_notify, email, user_id = nil)
    @e_notify = e_notify
    @user_id = user_id

    mail to: email, subject: e_notify.title
  end
end

app/views/edm_mailer/send_edm.html.slim

= raw @e_notify.parse_link_in_email(@user_id)

Done