I am still testing this setup so please be aware to have adequate testing before trying to put it in production and share your valuable thoughts in comments if you have better idea for implementing same thing. As we know, Redis is an open source, BSD licensed, advanced key-value store with lots of features which makes it preferred choice. Persistency (disk dump) is one of them. In a busy, loaded Redis server, persistency can have its own noticeable overhead and impact performance.

How can we have data reliability/recoverability without having overhead of persistency in working redis server?

We are talking about non-transient data here which is important for application. There can be lots of approaches. You can tune the disk dump frequency settings in redis.conf to have a fine balance of things, this will do the trick for Most of times. You can also have a master slave setup where master do the work and only slave have disk dump enabled, this looks acceptable until one fine day where master server reboots or redis service restarted, and find nothing in its datadir. Consequently slave also get informed that no data exists so data will be wiped out from slave as well. Though you can recover data from old disk dumps from slave but this can be disastrous. One way I can think of to avoid such incident is to rsync data (dump) from slave to master datadir. There would be some latency and you can loss some data but in our case this looks like fine balance between performance and reliability. Here are the steps to implement this along with script to do the sync:

Step 1. Turn off persistency in Redis master server
You can do this using config set commands so no need to restart redis service. Do not forget to update redis.conf also:

$ redis-cli config set save ""
OK
$ vim /etc/redis/redis.conf
 save ""
 
#save 900 1
#save 300 10
#save 60 10000

Step 2. Make sure persistency (disk dump) enabled in slave server (by default they are enabled). Set up password less ssh access from master to slave server so rsync can work smoothly. You can follow this guide to have that in ubuntu.

Step 3. In Master server, put this script and update its variables to point to correct datadir etc.

$ cat redis-sync.sh 
#!/bin/bash
 
#j#
# redis-sync.sh : sync redis db from slave to master since persistency in master is disabled.
#j#
 
LogFile="/var/log/redis/redis-sync.log"
LogRotate=3     # keep log for these number of days
LocalDataDir="/data/db/redis/"
RemoteDataFile="/data/db/redis/dump.rdb"
 
if [ $# -ne 1 ]; then
  echo 'Usage: redis-sync.sh <serverip> or logtrunc'
  exit 0
fi
 
# it's a good idea to truncate self log file to avoid disk space alert due to it. 
if [ $1 == "logtrunc" ]; then
  LinesToKeep=`echo $LogRotate*24*60 | bc`
  tail -n $LinesToKeep $LogFile > $LogFile.new
  rm -f $LogFile
  mv $LogFile.new $LogFile
  echo "$(date +"%b %d %R") Log file $LogFile truncated to have last $LogRotate days of logs only." >> $LogFile
  exit
fi
 
Instances=`pgrep redis-sync | wc -l`
if [ $Instances -gt 2 ]; then
  echo "$(date +"%b %d %R") $Instances instance(s) of this script already running, exiting to let them complete."  
 exit 1
fi
 
# Generally redis db dump file is not readable by other user/group, hence give read permission to it
ssh ubuntu@$1 "sudo chmod +r $RemoteDataFile"
if [ $? -ne 0 ]; then
 echo "$(date +"%b %d %R") Error: Communication failure or unable to make dump readable at $1." >> $LogFile
 exit 1
fi
 
# sync dump from slave, redirect output to $LogFile for debugging else trash it
rsync -avz ubuntu@$1:$RemoteDataFile $LocalDataDir > /dev/null 2>&1
if [ $? -ne 0 ]; then
 echo "$(date +"%b %d %R") Error: Unable to rsync $RemoteDataFile from $1 server!" >> $LogFile
 exit 1
fi
 
touch $LocalDataDir
echo "$(date +"%b %d %R") OK: redis dump $RemoteDataFile syncd from $1." >> $LogFile

Supply slave server ip/hostname to script for sync, let’s do a test run, check log file for messages:

$ ./redis-sync.sh 10.0.0.xxx
The authenticity of host '10.0.0.xxx (10.0.0.xxx)' can't be established.
ECDSA key fingerprint is d0:e7:0a:a1:56:32:ca:ec:f1:e3:90:7d:1b:56:0b:5b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.0.xxx' (ECDSA) to the list of known hosts.
 
$ tailf /var/log/redis/redis-sync.log 
Apr 22 05:44 OK: redis dump /data/db/redis/dump.rdb syncd from 10.0.0.xxx.

Looks fine. You will get error message in log file in case script failed to work properly.

Step 4. Put script in cron, also to avoid getting lots of disk space consumed by script log file, call it with logtrunc option to truncate that. You can set no. of days in script for which you want to have logs or can have automatic rotation.

$ crontab -e 
* * * * * /root/custom-scripts/redis-sync.sh 10.0.0.xxx
 
# truncate log file
0 4 * * * /root/custom-scripts/redis-sync.sh logtrunc

Step 5. You can disable auto starting of Redis service upon machine reboot, first call sync script to sync dump from slave and then start the service. Although this doesn’t have specific benefit since if master goes down, slave won’t do anything but in rare cases where data dir/dump in master got corrupted or get missed, it a good idea to take fresh from slave.

$ update-rc.d redis-server disable
$ vim /etc/rc.local
/root/custom-scripts/redis-sync.sh 10.0.0.147
/etc/init.d/redis-server start

As mentioned earlier, this is just quick fix. Please share your thoughts on shortcomings/better approach, if any.

Redis is an open source, advanced key-value store. This guide provides you instructions/commands you can use to install and configure redis with master/slave config in two ec2 Ubuntu instances. Our intention is to use redis to store php sessions which needs to be shared among other instances.

1. Open ports in aws firewall:
Redis works with port 6379 which you need to open in aws firewall. You can do this by command line tools or Amazon Consle. IN case of Console, navigate to Security Groups, select security group in which instances are running and then select “inbound” tab to see/update rules.

Put port 6379 in port range text box and select the same security group (in which you are working right now) so that all instances in this security group can community using this port. Make sure to not open the port for world (by selecting 0.0.0.0/0 in source), this would be really bad practice from security point of view.

If you need to monitor redis from some other host/nagios etc. You need to open port for them as well either by specifying their IP address or their security group.

2. Install using apt-get
You can setup Redis by installing using apt-get/aptitute or downloading manually. Both are easy ways but this installation through apt-get will be quick though it won’t fetch you latest/greatest stable build of Redis.

If you want to install latest build, then you should compile it and for that you can jump to step 3 right now.

A quick search for redis-server shows that the default Ubuntu repositories having very old version of Redis:

$ apt-cache show redis-server
Package: redis-server
Priority: optional
Section: universe/misc
Installed-Size: 511
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Chris Lamb <lamby@debian.org>
Architecture: amd64
Source: redis
Version: 2:2.2.12-1build1
Depends: libc6 (>= 2.7), adduser
Filename: pool/universe/r/redis/redis-server_2.2.12-1build1_amd64.deb
...
...

Here you can see redis-server_2.2.12 while on Redis site, latest stable version is 2.4.17 as of writing this article. Obviously we should not install the old 2.2.12 release.

We can pick another repository which should have latest or at least near to latest version. One of such repository from where you can get the package is dotdeb. Here in this repositories I can see version 2.4.16 which is very near to latest release.

Add following in /etc/apt/sources.list file:

$ sudo vim  /etc/apt/sources.list
	deb http://packages.dotdeb.org squeeze all
	deb-src http://packages.dotdeb.org squeeze all

Get and add dotdeb gnupg key after which run apt-get update and install redis-server:

$ wget http://www.dotdeb.org/dotdeb.gpg; cat dotdeb.gpg | sudo apt-key add -
$ sudo apt-get update
$ sudo apt-get install redis-server

3. Install using manual downloading it from redis site
This will enable you to get latest stable binaries of redis server. This process is neatly explained on Redis site itself here.

I’m here quickly putting commands with minor changes:

$ sudo apt-get install build-essential
$ cd /usr/src/
$ wget http://download.redis.io/redis-stable.tar.gz
$ tar xvzf redis-stable.tar.gz
$ cd redis-stable
$ sudo make
$ cd src 
$ sudo cp redis-server /usr/local/bin/
$ sudo cp redis-cli /usr/local/bin/
 
$ sudo mkdir /etc/redis /var/redis /var/log/redis
$ cd ../utils; sudo cp redis.conf.tpl /etc/redis/redis.conf; sudo cp redis_init_script /etc/init.d/redis
$ sudo update-rc.d redis defaults

4. Update config to make master/slave
After following above commands in both of the instances, you will have Redis installed and running in them. Making Redis work in master/slave config is relatively very easy, all you need to do is to tell one of server to get updates from other. On the instance, where you want to make Redis slave, login in it and update redis.conf file. Essential parameters you need to check/update are shown below:

$ sudo vim /etc/redis/redis.conf
daemonize yes
pidfile /var/run/redis/redis-server.pid
port 6379
#bind 127.0.0.1
slaveof 10.x.x.x 6379

You can see the last line slaveof which specify from which server it should get updates using which port. It’s a good idea to specify internal IP of other instance for such internal communication for relatively better performance as well as save some costs.

You need to check/update config in both of the servers except on master, do not put that “slaveof” configuration line.

Start redis and check:

$ sudo /etc/init.d/redis start
$ redis-cli ping
PONG

Redis is installed, configured and running.

5. Install PHP client and update PHP config to use it
Our intention is to use Redis with PHP and for that you need to install Redis client for PHP. You can get more info about Redis clients here.

$ sudo apt-get install php5-redis
$ cat /etc/php5/conf.d/redis.ini
; configuration for php redis module
extension=redis.so
$ sudo service apache2 reload
$ sudo service apache2 reload
 * Reloading web server config apache2
   ...done.

You can see that PHP config is updated to use Redis extension. We reloaded Apache to apply updated PHP configs. You are good to go. Let me know if you face any issues or there’s any suggestions to improve the process.

While you can download Ruby directly from here or from repositories (though you may not find latest version in it). I preferred using the awesome Ruby version manager tool (rvm) for this purpose. Our setup was in requirement of latest Ruby, Gems and Passenger with Apache2 module and that setup seems to be straightforward but I faced lots of smaller issues and hence thought of collecting and putting final commands here so that next time it would be a quick setup without all that digging.

These commands are tested successfully in Ubuntu 12.04 ec2 instance.

Step 1. Install essential packages
I am using latest Ubuntu 12.04 AMI in a new small instance. We need to install required packages/libraries first otherwise you will be bumped up with hiccups and needs to install them one by one when got stuck so let’s install them at first place itself:

$ sudo apt-get install build-essential apache2 zlib1g-dev libcurl4-openssl-dev libssl-dev apache2-prefork-dev libapr1-dev libaprutil1-dev

Step 2. Install RVM and latest stable Ruby
We will install RVM and specify that it should install latest stable Ruby as well.

1
2

$ sudo curl -L https://get.rvm.io | sudo bash -s stable --ruby
$ source /usr/local/rvm/scripts/rvm

Please note that you need to run the last ‘source /usr/local/rvm/scripts/rvm’ command once for all users where Ruby availability is needed otherwise other users won’t find Ruby like below example:

$ su exampleuser
$ ruby --version
The program 'ruby' can be found in the following packages:
 * ruby1.8
 * ruby1.9.1
Try: sudo apt-get install <selected package>
 
$ source /usr/local/rvm/scripts/rvm
 
$ ruby --version
ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux]

Ruby is installed. Let’s proceed to setup Apache2 with Passenger now.

Step3. Install Passenger and its apache module
We need to install OpenSSL extension/library for Ruby, I tried installing it with apt-get but that’s also fetching Ruby1.8 libraries which I don’t want to install, so I got it from Ruby source code, compiled and installed:

$ cd /usr/local/rvm/src/ruby-1.9.3-p194/ext/openssl/
$ sudo ruby extconf.rb
$ sudo make && sudo make install

Time to install passenger, module for apache and configure apache to use that module:

$ sudo gem install passenger
$ sudo passenger-install-apache2-module
$ sudo vim /etc/apache2/httpd.conf
   LoadModule passenger_module /usr/local/rvm/gems/ruby-1.9.3-p194/gems/passenger-3.0.17/ext/apache2/mod_passenger.so
   PassengerRoot /usr/local/rvm/gems/ruby-1.9.3-p194/gems/passenger-3.0.17
   PassengerRuby /usr/local/rvm/wrappers/ruby-1.9.3-p194/ruby
$ sudo service apache2 restart

Setup is done.

If you enjoyed this article, please share it with your friends. If there’s any issue/correction/suggestion, I’d be glad to hear that, please put it in comments below.

Other helpful articles in this blow on AWS ec2:

• Quickly setup git server with gitolite, gitweb, ssh and http auth
• Create secure mongodb replica-set in aws ec2 instance
• create and manage raid0, raid10 using ebs volumes in aws ec2 instance
• Simple upstart job in ubuntu to ensure your processes keep running
• Install and configure ftp server in aws ec2

Installation and configuration of MongoDB is relatively very easy. There are different ways you can run MongoDB, be it running single server, master-slave, replica-set or running in sharded/cluster way. By default, MongoDB installation doesn’t provide secure environment where clients should authenticate themselves before accesssing database.

My setup consists 2 small instances running as normal MongoDB servers and 1 micro instance running as arbiter. An arbiter is special member in replica-set which takes part in voting/selection of primary but doesn’t store any data in it. Its perfectly fine to host your arbiter server within an existing server running for some other service like load balancer (haproxy) etc because of its lighweight resource utilization. To make setup secure, clients must use username/password to connect to MongoDB and also members of replica-set must have shared key file.

I have tested instructions written below in instances running Ubuntu 12.04 but I strongly reccommend to have adequate testing/checking before putting such setup in production.

Step 1: Create instances and install MongoDB

Create 3 instances of your choice (micro/small/medium etc). I have used 64-bit publicly available Ubuntu AMIs from here for my instances. Let’s assume we got 3 instances with following internal IPs:

Instance1: 10.150.150.150
Instance2: 10.150.150.151
Instance3: 10.150.150.152

Time to configure them now, execute following commands in all 3 instances:

$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
$ echo "deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen" | sudo tee /etc/apt/sources.list.d/10gen.list
$ sudo apt-get update
$ sudo apt-get install mongodb-10gen

Here first we added 10gen key in our keys database and then added mongodb repository path. After that apt-get install mongodb-10gen will install latest stable release of mongodb in our server.

You can also store your mongodb database in separate EBS volume or with RAIDed EBS volumes, If interested, please refer this detailed article about it: create and manage RAID0/RAID10 using EBS volumes in AWS EC2 Ubuntu instance

Step 2: Configure replica-set name and initialize it
Open the mongodb config file /etc/mongodb.conf and jump to last line. Uncomment the line which starts with replSet and put a name for your replica-set, your file should look like this in all 3 instances:

$ tail /etc/mongodb.conf
#slave = true
#source = master.example.com
# Slave only: specify a single database to replicate
#only = master.example.com
# or
#master = true
#source = slave.example.com
 
# in replica set configuration, specify the name of the replica set
replSet = myreplica

Make sure to restart the mongodb service after any change in config file:

$ sudo service mongodb restart
mongodb stop/waiting
mongodb start/running, process 28129

Now in instance1 or instance2, start mongo shell by typing ‘mongo’. Initialize the replica-set and other 2 instances. Please remember that our 3rd instance is arbiter one so there’s different command to add that instance in replica-set:

ubuntu@instance1:~$ mongo
> rs.initiate();
> rs.add("10.150.150.151"); 
> rs.addArb("10.150.150.152");  ## its rs.addArb() to add arbiter

The output of these commands are not shown here because I already have working replica-set and just replaying the command for the purpose of this article. Anyhow, mongo should reply with “ok” upon a correct command and throws error otherwise.

Once you add servers in replica-set, the members of replica-set have voting among themselves where arbiter will vote and the server which have highest uptime will become primary (you can override that with priorty setting though).

After a while of initializing replica-set, the instance1 should become primary:

$ mongo
myreplica:PRIMARY>  rs.conf();
{
        "_id" : "myreplica",
        "version" : 6,
        "members" : [
                {
                        "_id" : 0,
                        "host" : "10.150.150.150",
                        "priority" : 2
                },
                {
                        "_id" : 1,
                        "host" : "10.150.150.151",
                        "priority" : 2
                },
                {
                        "_id" : 2,
                        "host" : "10.150.150.152",
                        "arbiterOnly" : true
                }
        ]
}
myreplica:PRIMARY>

You can use “rs.status()” for more detailed information about replica-set:

myreplica:PRIMARY> rs.status();
{
        "set" : "myreplica",
        "date" : ISODate("2012-09-10T09:48:25Z"),
        "myState" : 1,
        "members" : [
                {
                        "_id" : 0,
                        "name" : "10.150.150.150",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 269445,
                        "optime" : Timestamp(1347270449000, 1),
                        "optimeDate" : ISODate("2012-09-10T09:47:29Z"),
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "10.150.150.151",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 269303,
                        "optime" : Timestamp(1347270449000, 1),
                        "optimeDate" : ISODate("2012-09-10T09:47:29Z"),
                        "lastHeartbeat" : ISODate("2012-09-10T09:48:23Z"),
                        "pingMs" : 0
                },
                {
                        "_id" : 2,
                        "name" : "10.150.150.152",
                        "health" : 1,
                        "state" : 7,
                        "stateStr" : "ARBITER",
                        "uptime" : 269261,
                        "lastHeartbeat" : ISODate("2012-09-10T09:48:24Z"),
                        "pingMs" : 0
                }
        ],
        "ok" : 1
}

You can jump to other hosts instance2/instance3 and check status from there as well. Now the replica-set is operational, let’s make it secure.

Step 3: Enable authentication and secure replica-set
Authentication is not enabled by default in MongoDB but its very easy to use. Open config file /etc/mongodb.conf and uncomment the auth=true line and comment out noauth line. These two lines should look like below:

$ grep auth /etc/mongodb.conf
#noauth = true
auth = true

Make this change in all instances and restart mongodb service.

$ sudo service mongodb restart
mongodb stop/waiting
mongodb start/running, process 28329

Go to mongo shell again and configure the administrative user:

$ mongo
> use admin; 
switched to db admin
> db.addUser("mongoadmin","mongopass");

After this, you need to authenticate yourself to access details. Let’s try to get in mongo shell and show databases:

$ mongo
> show dbs; 
Mon Sep 10 10:02:09 uncaught exception: listDatabases failed:{ "errmsg" : "need to login", "ok" : 0 }

So you need to supply your credentials:

> use admin;
> db.auth("mongoadmin","mongopass"); 
1
myreplica:PRIMARY>

If you have some database, you can create username/password for that and instruct teams to use that user/pass to get connect. Let’s say you have a database name proddb, create a user/pass for it:

myreplica:PRIMARY> use proddb; 
myreplica:PRIMARY> db.addUser("produ","mypass");
myreplica:PRIMARY> db.addUser("produread","otherpass", true);

Here, produ user will have read/write access to database proddb while produread will have only read-only access.

Step 4: Secure replica-set members
To make your replica-set more secure, you can enable keyFile option where each server needs to have an identical key file which works like a password for their internal communication. Essentially its just a text file.

Repeat following commands in all instances.

Create a key file:

$ echo "myrandomtextforkeyfiletosecurereplicaset" | sudo tee /etc/mongodb.key
$ sudo vim /etc/mongodb.conf
keyFile=/etc/mongodb.key  ## uncomment and set path of key file
$ sudo service mongodb restart

Now each member should have this identical file to remain a member of replica-set. Get more information about securing MongoDB from here.

Have you tried securing your MongoDB installation? Is there anything you would like to recommend in this context or if there’s any comment/suggestion, please put it in comments below.

Helpful related articles:
* create and manage RAID0/RAID10 using EBS volumes in AWS EC2 Ubuntu instance
* Simple and efficient MongoDB Backup using script

This is small post explaining a simple concept in Ubuntu. One of my friend requested about a way to ensure his processes keep running. Of course there are multiple ways to do and I have documented this earlier with details my earlier post.

Let’s review quickly:

1. Run processes through cron but it’s only helpful if check frequency is per minute or higher.
2. Use some process monitoring solution like Monit.
3. Create custom daemon wrapper script which keep checking processes all the time.

Due to some path/env issue (I have not investigated that), my friend said his scripts are not running as expected under Monit or throug cron. Also cron is not an option here because he needs checking processes very frequently, let’s say every 5 seconds. This left us third option where we can create a custom wrapper script which should keep running like a daemon and keep an eye on processes. Now the question is how to ensure that our daemon keeps running all the time?

As described in my earlier blog post on Redhat based Linux, we can put entry in /etc/inittab which will ensure that our daemon will always be there in processlist but that sysV type has been discontinued in Ubuntu (12.04 I am talking about) and hence we need to create a simple process using upstart.

Let’s say our daemon script path is /root/daemon/mymonitor.sh

Create a config file for upstart job:

 #!upstart
description     "MyMon: monitoring my custom processes."
 
start on startup
stop on shutdown
 
respawn
 
exec start-stop-daemon --start --make-pidfile --pidfile /var/run/mymon.pid --exec /root/daemon/mymon.sh >> /var/log/mymon.log 2>&1

Here, “start on startup” and “stop on shutdown” implies the process should get started when system boots up and should be gracefully terminated upon shutdown.

respawn keyword ensures that this process should be running all the time. If its crashed/stopped, it will be respawned automatically.

exec is used here to execute our script, it gives PATH and optional arguments to our script. You can use ‘script’ keyword instead to write down multiple script commands to be executed by /bin/sh. You can check full documention here.

After saving the upstart config, time to start our job, check its pid and try stopping it:

$ sudo start mymon 
mymon start/running, process 20890
 
$ cat /var/run/mymon.pid 
20890
 
$ sudo stop mymon
mymon stop/waiting

Do you also think its simple and efficient method for process monitoring? or if you have any other suggestion? would appreciate your comments.

Other helpful related articles:
* Run scripts as daemon or through cron continuously
* Bash script to backup essential log files in Linux
* Top 5 most useful tools for Linux Admin