Linux Admin Zone

Simple and efficient MongoDB Backup using script

jagbir — Wed, 09 May 2012 10:17:54 +0000

MongoDB Backup types and strategies are neatly explained in its documentation, which you can check here. In case you are not familiar with MongoDB backup types and strategies, please have a look at its documentation.

What I am describing here is a simple script which we are using since months to take MongoDB backup and transfer it over to our Backup server. Here are few things its doing:

As we have multiple MongoDB Replica Sets, the script identify current replica set and check whether current server is Master or Slave, exit if its Master. We take backup only from Slave host.
Take Backup using mongodump command.
Upon successful completion of dump, transfer that to our Backup server. Ensure that ssh key based authentication is setup between both servers to implement seamless and secure transfer. It creates new directory based on current timestamp under replicaset directory in specified path at Backup server and transfer dump there.
Log each steps described above, send alert mail if any step fail with description or send confirmation mail upon successful execution with essential details.
Sample confirmation mail is below:

Subject: Backup for  done on  at 09/May/12 08:00:01 AM
Body: Mongo Backup Status for  on 09/May/12 08:00:01 AM. 
 
08:00:01 AM:  is slave and looks OK.
08:00:01 AM: Starting Dump, executing /usr/local/mongodb/mongo/bin/mongodump --out /databases/dump --host ...
08:34:06 AM: Mongodump command completed. Backup size is 25G. 
08:34:10 AM: Directory created on backup server, copying data using scp...
08:57:00 AM: Copied dump to backup server in directory /home/backup/Mongodump/replicaset/datestamp/.
08:57:00 AM: Mongo Backup process completed successfully.

Here is the full bash script, please note that you need to update variables properly and have to check and update it to run in your environment which can be entirely different from mine. Its just an idea to automate MongoDB backup:

##
# Script to take mongo backup using mongodump and store it in Backup Server
##
 
#!/bin/bash
 
## Set variables
TodayDate=`date +"%d/%b/%g %r"`
DateStamp=`date +%d%m%y%H%M%S`
CurrentTime=`date +"%r"`
 
MongoBinPath="/usr/local/mongodb/mongo/bin"
ReplicaSet=`echo 'rs.status()' | $MongoBinPath/mongo | egrep "set" | awk -F \" '{print $4}'`
MongoHost=`hostname`
LocalBackupPath="/databases/dump"
 
LogFile="/var/log/mongo-backup.log"
IsOK=0
CmdStatus=""
 
BackupHost=xx.xx.xx.xx
BackupHostPath="/home/backup/mongobackup"
BackupHostPort=22
 
MailNotification="admin@domain.com anotheradmin@domain.com"
 
echo -e "Mongo Backup Status for $ReplicaSet on $TodayDate. \n" &gt; $LogFile
 
## Check whether host is slave and in good state for backup
for i in `echo "rs.status()" | $MongoBinPath/mongo | egrep "name" | awk -F \" '{print $4}'| cut -f 1 -d :`; do
 IsMaster=`echo "db.isMaster()"| $MongoBinPath/mongo --host $i | grep ismaster|awk -F ":" '{print $2}' | cut -f 1 -d ,`;
 TheState=`echo "rs.status()"| $MongoBinPath/mongo --host $i | grep -i mystate | awk -F ":" '{print $2}' | cut -f 1 -d ,`;
 if [ $IsMaster == "false" -a $TheState -eq 2  ]; then
  MongoHost=$i
  IsOK=1
  echo "$CurrentTime: $MongoHost is slave and looks OK." &gt;&gt; $LogFile
  break
 fi
done
 
CurrentTime=`date +"%r"`
 
## Exit if not good
if [ $IsOK -eq 0 ]; then
   echo "$CurrentTime: Error: Either $MongoHost is not slave or not in good state. Aborting Backup, Please check!" &gt;&gt; $LogFile
   mail -s "Backup error for $ReplicaSet from $MongoHost on $TodayDate" $MailNotification &lt; $LogFile    exit 1; fi ## Remove earlier backup CmdStatus=$(rm -rf $LocalBackupPath/*) ## Start backup process echo "$CurrentTime: Starting Dump, executing $MongoBinPath/mongodump --out $LocalBackupPath --host $MongoHost..." >> $LogFile
 
CmdStatus=`$MongoBinPath/mongodump --out $LocalBackupPath --host $MongoHost`
if [ $? -ne 0 ]; then
  echo "$CurrentTime: There is an issue while trying to take dump in $MongoHost. Aborting dump process, please check! " &gt;&gt; $LogFile
  cat $CmdStatus &gt;&gt; $LogFile
  mail -s "Backup error for $ReplicaSet from $MongoHost on $TodayDate" $MailNotification &lt; $LogFile   exit fi CurrentTime=`date +"%r"` BackupSize=$(du -sh $LocalBackupPath/. | awk '{ print $1 }') echo "$CurrentTime: Mongodump command completed. Backup size is $BackupSize. " &gt;&gt; $LogFile
 
## dump is fine then scp it to backup server
## create directory first
CmdStatus=$(ssh -p $BackupHostPort $BackupHost "mkdir -p  $BackupHostPath/$ReplicaSet/$DateStamp")
 
if [ $? -ne 0 ]; then
  echo "$CurrentTime: Either failing to connect Backup server using ssh or destination directory already exist!" &gt;&gt; $LogFile
  cat $CmdStatus &gt;&gt; $LogFile
  mail -s "Backup error for $ReplicaSet from $MongoHost on $TodayDate" $MailNotification &lt; $LogFile   exit fi CurrentTime=`date +"%r"` echo "$CurrentTime: Directory created on backup server, copying data using scp..." &gt;&gt; $LogFile
 
CmdStatus=`scp -P $BackupHostPort -r $LocalBackupPath/* $BackupHost:/$BackupHostPath/$ReplicaSet/$DateStamp/`
if [ $? -ne 0 ]; then
  echo "$CurrentTime: Unable to scp dump to $BackupHost:/$BackupHostPath/$ReplicaSet/$DateStamp using port $BackupHostPort. " &gt;&gt; $LogFile
  cat $CmdStatus &gt;&gt; $LogFile
  mail -s "Backup error for $ReplicaSet from $MongoHost on $TodayDate" $MailNotification &lt; $LogFile   exit fi CurrentTime=`date +"%r"` echo "$CurrentTime: Copied dump to backup server in directory $BackupHost$BackupHostPath/$ReplicaSet/$DateStamp/." &gt;&gt; $LogFile
echo "$CurrentTime: Mongo Backup process completed successfully." &gt;&gt; $LogFile
mail -s "Backup for $ReplicaSet done on $MongoHost at $TodayDate" $MailRecipients $MailNotification &lt; $LogFile

Its just a basic script and may needs further enhancements. In case you have suggestion/queries, please put it below in comments.

A very small bash script challenge

jagbir — Wed, 25 Apr 2012 10:35:52 +0000

*Kind Note*: This was written with a sense of humor to allow visitors quickly discover code anomaly and suggests fixes but if it is not up to your mark, please close your browser tab instead of making unnecessary noise. Thanks You!
I am putting a damn small thing here regarding bash script for fun.

Here it goes: Need to create a bash script which asks for a word from user, say either “one”, “two” or “three” and then check in single if statement (no else if section) that if its not “one” or “two” or “three” then print Not OK otherwise print OK.

Further, here is a snippet, check and figure out why its not working as expected:

#!/bin/bash
echo -n Enter a word:
read myword
 
if [[ $myword != "one" || $myword != "two" || $myword != "three" ]]; then
 echo Not OK
else
 echo OK
fi

Please put your suggestion/solution script in comments.

How to setup Git http authentication using LDAP in Apache

jagbir — Wed, 18 Apr 2012 13:05:37 +0000

In earlier article, I have described setting up git server with gitolite, gitweb, ssh and http auth using passwd file. Here as an extension of that article, I am describing how to do authentication using LDAP so that authentication become more seamless and avoid any sort of manual work for managing access when you have LDAP for authenticating users.

Before proceeding for change in config, you should confirm that ldap and authnz_ldap modules are there in Apache. You can check that using httpd -M command, following should be there in output:

$ httpd -M 
..
 ldap_module (shared)
 authnz_ldap_module (shared)
..

If this is not the case, then please install these modules and make sure you load them in your Apache config (usually /etc/httpd/conf/httpd.conf) like this:

LoadModule ldap_module modules/mod_ldap.so
LoadModule authnz_ldap_module modules/mod_authnz_ldap.so

After having these modules to facilitate authentication, we need to remove or comment out following lines in our git config file /etc/httpd/conf.d/git.conf:

<Location />
    AuthType Basic
    AuthName "Private Git Access"
    Require valid-user
    AuthUserFile /var/www/gitweb/passfile
Location>

After removing or commenting out above lines, put these lines in the file:

<Location "/">
    AuthType Basic
    AuthName "Git Authentication"
    AuthBasicProvider ldap
    AuthzLDAPAuthoritative off
    AuthLDAPURL "ldap://:389/ou=xx,dc=xx,dc=xx,dc=com?sAMAccountName?sub?(objectClass=user)"
    AuthLDAPBindDN <user>@<mydomain>
    AuthLDAPBindPassword <user password>
    Require valid-user
Location>

Here make sure to supply correct LDAP url and provide info of one user and its password so that Apache can contact LDAP to retrieve authentication information. You also needs to update gitolite.conf to manage authorization for git repositories for LDAP user.

Reload Apache to apply new settings and you should be able to access Git repository over http using LDAP user.

Common issues:
If authentication not working, put “Loglevel Debug” option in your Apache VirtualHost and check Apache error logs. In case you notice following error:

[Wed Apr 18 15:02:13 2012] [debug] mod_authnz_ldap.c(454): [client xx.xx.xx.xx] [25749] auth_ldap authenticate: accepting user.name
[Wed Apr 18 15:02:13 2012] [debug] mod_authnz_ldap.c(821): [client xx.xx.xx.xx] [25749] auth_ldap authorise: declining to authorise

Then make sure AuthzLDAPAuthoritative off entry is there in Apache git config file, I have already mentioned it above just in case if you missed it.

In case you notice “[User Not Found]” in error log, then check your user name again and make sure the user exist in correct OU/group specified in ldap url.

Quickly setup git server with gitolite, gitweb, ssh and http auth

jagbir — Fri, 30 Mar 2012 08:26:17 +0000

As per the official definition, Git is a free & open source, distributed version control system designed to handle everything from small to very large projects with speed and efficiency. I am describing here steps which I followed to setup a Git server along with Gitolite, Gitweb, ssh and http auth in RHEL5 machine. I have done the installations using RPMs (lazy men’s method) which I got from here: http://pkgs.repoforge.org/git/

Step 1: Download the required RPMs or install using source

Here are the RPMs I downloaded from source mentioned above (of course, download the latest version of these RPMs when you wants to do installation):

git-1.7.8.2-2.el5.rf.x86_64.rpm
gitolite-1.5.9.1-2.el5.noarch.rpm
gitweb-1.7.8.2-2.el5.rf.x86_64.rpm

You may also need to have some perl dependencies which you can install through CPAN or can also download the RPMs for them, I needed below ones:

perl-DBI-1.617-1.el5.rfx.x86_64.rpm
perl-Git-1.7.8.2-2.el5.rf.x86_64.rpm
perl-TermReadKey-2.30-3.el5.rf.x86_64.rpm (Optional)
perl-Error-0.17017-1.el5.rf.noarch.rpm (Optional)

Step 2: Install the RPMs:

$ rpm -ivh perl-DBI-1.617-1.el5.rfx.x86_64.rpm perl-TermReadKey-2.30-3.el5.rf.x86_64.rpm perl-Error-0.17017-1.el5.rf.noarch.rpm git-1.7.8.2-2.el5.rf.x86_64.rpm perl-Git-1.7.8.2-2.el5.rf.x86_64.rpm  gitolite-1.5.9.1-2.el5.noarch.rpm gitweb-1.7.8.2-2.el5.rf.x86_64.rpm

We have Git, Gitolite and Gitweb installed now.

Step 3: Configure Gitolite for authentication/authorization:

We need to configure Gitolite and the information for that is already described here so I am skipping that part.

Step 4: (Optional) Test Git with Gitolite:

Its worth a try to quickly test Git with Gitolite you just installed/configured. Jump to your pc and if you have Linux, generate public/private keys using ssh-keygen utility in case you already don’t have, for testing purposes, you can use following command:

/usr/bin/ssh-keygen -N '' -t rsa -f /root/.ssh/id_rsa

In case you are using Windows (which unfortunately I am using as of now), you can use puttygen utility and can refer a good tutorial here for exact process.

Copy your public key file to Git server, rename it to yourname.pub and put it in this directory so that Gitolite can refer/read them when needed: /var/lib/gitolite/.gitolite/keydir/

Time to clone gitolite-admin repository now, for Linux, just use:

$ git clone git@serverip:gitolite-admin

For Windows, you can install msysgit and optionally you can install a cool Git client like TortoiseGit from here. To Clone the gitolite-admin repository now, browse any directory, right click, choose Git Clone… and put required information. A sample screenshot is below:

Clone should get successful and you will get gitolite-admin repository in your pc. Go inside and update gitolite.conf to add new repositories/users. This process is described here if you want to continue testing.

Step 5: Configure Gitweb, http access of Git

This process is also documented by original author here but that is for OpenSuSE and while following that, I ran in some issues, so here posting information to setup this in RHEL machine which is working for me. You may want to refer that documentation in case things are not very clear reading my instructions because I am not diving in details and focus is more on practical execution.

Add following line in /var/lib/gitolite/.gitolite.rc file:

$GL_GITCONFIG_KEYS = "gitweb.url receive.denyNonFastforwards receive.denyDeletes";

Add some config entries in gitolite.conf file along with entry for daemon user. My gitolite.conf looks like below:

$ cat /var/lib/gitolite/.gitolite/conf/gitolite.conf
repo    gitolite-admin
RW+     =   git daemon
 
repo    tproject
RW      = git jagbir daemon
R       = @all
config  gitweb.url = git@serverip:tproject
config  receive.denyNonFastforwards = true
config  receive.denyDeletes         = true
 
repo    @all
R       =   daemon gitweb

Don’t forget to add daemon in all repositories, whether for Read write or just read to enabling browsing through http.

Step 6: Configure Apache under SuExec:
Apache runs under Apache user while our Git repositories are under Gitolite user. We have to use SuExec module in Apache so that it will also run under Gitolite user and be able to update information in repositories. Confirm that SuExec module is there in you Apache by running: $ httpd –M command and you should have suexec_module (shared) line in output.

Update permissions of suexec program. We also needs to have a wrapper script and to know where to put it check options of suexec, here are commands:

$ chgrp apache /usr/sbin/suexec
$ chmod 4750 /usr/sbin/suexec
$ /usr/sbin/suexec -V
-D AP_DOC_ROOT="/var/www"
-D AP_GID_MIN=100
-D AP_HTTPD_USER="apache"
-D AP_LOG_EXEC="/var/log/httpd/suexec.log"
-D AP_SAFE_PATH="/usr/local/bin:/usr/bin:/bin"
-D AP_UID_MIN=500
-D AP_USERDIR_SUFFIX="public_html"

So path for our wrapper script and Gitweb is /var/www as shown above in AP_DOC_ROOT value. Create a wrapper script in /var/www/bin/ directory (create bin directory first). My script looks like below which you can copy as is:

$ cat /var/www/bin/gitolite-suexec-wrapper.sh
#!/bin/bash
 
USER=$1
 
export GIT_PROJECT_ROOT="/var/lib/gitolite/repositories"
export GITOLITE_HTTP_HOME="/var/lib/gitolite"
 
exec /usr/bin/gl-auth-command $USER

Because Gitweb will also runs under gitolite user, copy all of its files to /var/www directory and make sure the owner of /var/www directory (along with all subdirectories/files should be gitolite user), here are commands:

$ cp -r /usr/share/gitweb /var/www
$ chown –R gitolite.gitolite /var/www

Update gitweb.conf file to point to gitolite directory where all repositories are there, below line should be there in /etc/gitweb.conf file:

our $projectroot = "/var/lib/gitolite";

Step 7: Configure Virtualhost in Apache:
Here is my apache virtual host file, which you can copy as is (of course, change ServerName, Alias etc as per your values):

$  cat /etc/httpd/conf.d/git.conf
 
<VirtualHost *:80>
 
ServerName  git.mydomain.com
ServerAlias git
 
DocumentRoot /var/www/gitweb
 
SuexecUserGroup gitolite gitolite
 
SetEnv GIT_PROJECT_ROOT /var/lib/gitolite/projects
SetEnv GIT_HTTP_EXPORT_ALL
 
SetEnv GITOLITE_HTTP_HOME /var/lib/gitolite
 
ScriptAliasMatch \
"(?x)^/(.*/(HEAD | \
info/refs | \
objects/(info/[^/]+ | \
[0-9a-f]{2}/[0-9a-f]{38} | \
pack/pack-[0-9a-f]{40}\.(pack|idx)) | \
git-(upload|receive)-pack))$" \
/var/www/bin/gitolite-suexec-wrapper.sh/$1
 
<Directory "/var/www/gitweb">
Options ExecCGI
AllowOverride None
AddHandler cgi-script .cgi
DirectoryIndex gitweb.cgi
Order allow,deny
Allow from all
Directory>
 
<Directory "/var/www/bin">
<Files "gitolite-suexec-wrapper.sh">
Order allow,deny
Allow from all
Files>
Directory>
 
<Location />
AuthType Basic
AuthName "Git Access"
Require valid-user
AuthUserFile /var/www/gitweb/authfile
Location>
 
VirtualHost>

As you can see we are using basic authentication here and for that, you need to create file which will have auth information, create file and sample user (gitolite) to test it:

$ htpasswd -cmd /var/lib/gitolite/authfile gitolite
New password:
Re-type new password:
Adding password for user gitolite
 
$ cat /var/lib/gitolite/authfile
gitolite:wG7/EAcl9kdvU

Make sure you have initialize the repository to enable its access via http, let’s prepare testing repository for this purpose:

$ cd /var/lib/gitolite/repositories/testing.git
$ sudo -u gitolite git --bare init
$ sudo -u gitolite git update-server-info
$ mv hooks/post-update.sample hooks/post-update
$ chmod +x hooks/post-update

The above steps are needed for http access otherwise you will get error like below in your apache error logs when trying to clone:

[Tue Apr 10 15:34:16 2012] [error] [client 10.100.xx.xx] Repository not exported: '/var/lib/gitolite/repositories/testing'

All files under /var/www should have gitolite as owner, let’s update permissions once more:

$  chown -R gitolite:gitolite /var/www

Step 7: Test it out:
Restart apache and try to browse your server now: http://serverip. It should ask username/password and after supply correct, you should be able to see gitweb interface showing your repositories where you can traverse in them.

In case you see a blank page, then it might be issue with SuExec. Check suexec log file: /var/log/httpd/suexec.log. You may see a message like:

[2012-03-30 04:14:26]: cannot run as forbidden uid (100/gitweb.cgi)

This means suexec won’t execute under user/group have userid/groupid less than 500 (system). In this case you can change this id for our gitolite user as per below:

$ usermod -u 650 gitolite
$ groupmod -g 650 gitolite

650 is just an example here, you can use any value above 500 in case 650 is already used by existing user/group. As user/group id get changed, you need to set permissions again for your directories:

$ chown –R gitolite:gitolite /var/www

Try now and you should be able to browse smoothly. Please put a comment below if you are still facing any issues, I would try to help you out.

Update: If you want to perform authentication using LDAP for git which I have described in next article, you can access it using below link:

* Setup Git auth using LDAP

Fix mysql memory table error: The table xtable is full

jagbir — Mon, 20 Feb 2012 08:34:55 +0000

Replication just stopped in one Slave server with error: The table xtable is full which means no more records are permitted to insert in this table by MySQL and hence this has broken the replication.

I checked that xtable is having storage engine as Memory. In such tables, the max. no. of records you can insert is controlled by variable max_heap_table_size. When checking the size of this variable, I found that this is having default value:

mysql> show variables like 'max_heap_table_size';
+---------------------+----------+
| Variable_name       | Value    |
+---------------------+----------+
| max_heap_table_size | 16777216 |
+---------------------+----------+
1 row in set (0.00 sec)

So we need to increase the value of this variable and then issue Alter Table command to make it effective. Also do not forget to add variable with new value in your my.cnf.

Change the variable value:

mysql> set  max_heap_table_size=268435456;
Query OK, 0 rows affected (0.00 sec)
 
mysql> show variables like 'max_heap_table_size';
+---------------------+-----------+
| Variable_name       | Value     |
+---------------------+-----------+
| max_heap_table_size | 268435456 |
+---------------------+-----------+
1 row in set (0.00 sec)

Now let’s truncate the table and issue Alter table on it to make the value effective for this table:

mysql> truncate table xtable;
Query OK, 0 rows affected (0.00 sec)
 
mysql> alter table xtable ENGINE=MEMORY;
Query OK, 88 rows affected (0.06 sec)
Records: 88  Duplicates: 0  Warnings: 0

After this fix, the issue has been resolved. Make sure that you do not run the truncate command on table which have important data. I have issued truncate because it contains temporary/transactional data so we are fine with removing all records here.