If you haven’t heard of Duplicity or Duply before, Duply is basically a wrapper for Duplicity which makes it easier to manage. Duplicity itself is similar to rsnapshot except, it uses tar to efficiently store differences between backups (instead of hardlinks). Here’s the description from the man page:

Duplicity incrementally backs up files and directory by 
encrypting tar-format volumes with GnuPG and uploading 
them to a remote (or local) file server. Currently local, 
ftp, ssh/scp, rsync, WebDAV, WebDAVs, HSi and Amazon S3 backends 
are available. Because duplicity uses librsync, the incremental 
archives are space efficient and only record the parts of files 
that have changed since the last backup. Currently duplicity 
supports deleted files, full Unix permissions, directories, 
symbolic links, fifos, etc., but not hard links.

I wrote this mainly as a reference for myself when I need to set duply up on another server, but it might be useful for others as well.

CentOS 5 / 6 Instructions

Install the EPEL repo:

#Cent 6:
rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpm
#Cent 5:
rpm -Uvh http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm

Install duplicity:

yum --enablerepo=epel install duplicity

Install duply:
Get the URL for the latest version here: http://duply.net/?title=Duply-downloads
download it to your server, extract it, and copy duply to /usr/local/bin/duply then chmod +x /usr/local/bin/duply

wget http://dev.justynshull.com/duply_1.5.5.1.tgz
tar xvzf duply_1.5.5.1.tgz
cp duply_1.5.5.1/duply /usr/local/bin/duply
chmod +x /usr/local/bin/duply

Note: Bug 675234 is a request to have duply put into the Fedora repo, but there is also a .spec and source rpm if you wish to build an rpm yourself.

Setup a basic Duply profile

mkdir /etc/duply && chmod 700 /etc/duply
duply testvm1 create

Note: If you don’t create /etc/duply, then it will use $HOME/.duply by default.

Take a look at the options in /etc/duply/testvm1/conf and configure it to your liking. There are many different TARGET formats you can use, including ssh, rsync over ssh, ftp, and even amazon’s S3. You can view them all here: URL Formats
This is what mine usually look like(without encryption) using rsync over ssh:

# egrep -v '^#|^$' /etc/duply/testvm1/conf
GPG_KEY='disabled'
TARGET='rsync://backups.justynshull.com//home/testvm1/backups'
TARGET_USER='testvm1'
SOURCE='/'
MAX_AGE=3M
MAX_FULL_BACKUPS=3
MAX_FULLBKP_AGE=30D
VOLSIZE=3500
DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "
DUPL_PARAMS="$DUPL_PARAMS --volsize $VOLSIZE "
DUPL_PARAMS="$DUPL_PARAMS --include=/etc \
        --include=/home \
        --include=/root \
        --include=/var/www \
        --include=/var/lib/mysql \
        --include=/var/log \
        --exclude=/** "

I prefer to use multiple –include= options rather than fill /etc/duply/testvm1/exclude with every directory I *don’t* want backed up. Either way will work though. Also, if you’re going to use rsync or ssh/sftp, I’d recommend setting up the backup server so that you can log in with ssh keys and generate a separate key for each server you’re backing up from. You’ll also have to have ssh’d into the backup server as that user at least once to avoid errors about the target host key.

Encryption
Duply/Duplicity supports using GPG to encrypt volumes before uploading them to the remote server, and the easiest way to enable encryption is by putting this in your conf:

#comment out #GPG_KEY from earlier
#GPG_KEY='disabled'
GPG_PW='secret_password'

This will encrypt the volumes using the passphrase you put in GPG_PW, but you can refer to the documentation for how to set it up to use actual gpg keys.

Including MySQL Backups
If you’re running mysql on the server, you should consider adding something similar to this to /etc/testvm1/pre, which gets run automatically by duply, to dump all databases before backing up the server.

 
#!/bin/sh
mkdir -pv /root/db_backups
for db in $(mysql -uroot -e 'show databases' -s --skip-column-names | grep -v 'information_schema');
do
        mysqldump -uroot $db > /root/db_backups/$db.sql;
        sleep 10;
done

Run your first backup:

duply testvm1 backup

Automate it
If all goes well(no errors), then you should be okay to set up a cronjob to run duply backup.

# crontab -l
30      3       *       *       *       /usr/local/bin/duply testvm1 backup
30      5       *       *       sun       /usr/local/bin/duply testvm1 backup_verify_purge --force

Purge Old Backups
If you use the above crontab, the 2nd line will run once a week, purging old backups from the remote server. If you’re worried about keeping too many backups, you might want to increase how often this runs and also decrease the options in the duply profile configuration.

Verify Backups
Duply makes it easy to see what you’re currently backing up.
To see a list of backups stored on the remote server:

duply testvm1 status

To see a list of files that have changed since the last backup:

duply testvm1 verify

List all files in a backup yesterday(leave out to show latest):

duply testvm1 list 1D

Restore Backups
It’s just as easy to restore complete or partial backups.
Restore the entire latest backup to /tmp/restore:

duply testvm1 restore /tmp/restore

Restore backup from 7 days ago to /tmp/restore:

duply testvm1 restore /tmp/restore 1W

Restore single file or directory to /tmp/restore:

duply testvm1 fetch home/justyns /tmp/restore
#When using 'fetch', make sure you leave off the leading slash.

Restore a file from a month ago:

duply testvm1 fetch home/justyns/plans_for_world_dom.txt /home/justyns/plans_for_world_dom.txt 1M

Dom-0 is using four 1.5tb 7200rpm seagate drives in a software Raid-10, /dev/md2.  Both Dom0 and DomU have 1gb of ram and are using ext3. Xen is using mainly default settings with the default scheduler.

From DomU with the DomU image being file-based on ext3 Dom-0 fs
DomU has 1gb ram

[root@DomU]# dd if=/dev/zero of=tmpfile.bin bs=1024k count=10k
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 192.556 seconds, 55.8 MB/s

From Dom0 on the same filesystem where above DomU’s image is stored.
Dom0 has 1gb ram

[root@Dom0]# dd if=/dev/zero of=tmpfile.bin bs=1024k count=10k
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 72.0743 seconds, 149 MB/s

I was really surprised to see there was this much of a difference between Dom0 and DomU disk access. Several tests were run, and the results didn’t vary much at all.

I also ran a bonnie++ benchmark on both Dom0 and DomU.
Dom0:

 Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
razer.justynshul 2G   419  98 144092  21 68102   2   882  97 183977   2 699.3   0
Latency             60846us    1136ms     381ms   53632us   98757us     371ms
Version  1.96       ------Sequential Create------ --------Random Create--------
razer.justynshull.c -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency                87us     447us     486us     694us      34us      41us
1.96,1.96,razer.justynshull.com,1,1304536826,2G,,419,98,144092,21,68102,2,882,97,183977,2,699.3,0,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,60846us,1136ms,381ms,53632us,98757us,371ms,87us,447us,486us,694us,34us,41us

DomU:

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
razer2.justynshu 2G   446  99 94601  12 62259   1  1006  99 130897   1 182.1   0
Latency             64057us    1771ms    2969ms   19222us    1887ms     348ms
Version  1.96       ------Sequential Create------ --------Random Create--------
razer2.justynshull. -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency             44083us     395us     410us      59us      10us      66us
1.96,1.96,razer2.justynshull.com,1,1304538856,2G,,446,99,94601,12,62259,1,1006,99,130897,1,182.1,0,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,64057us,1771ms,2969ms,19222us,1887ms,348ms,44083us,395us,410us,59us,10us,66us

Round 2 – LVM-based disk on DomU

Here are the results of the same tests, except this time DomU is using LVM as the root disk instead of a file-based image. It is still on the same exact hardware and drives/raid set-up.

[root@DomU]# dd if=/dev/zero of=tmpfile.bin bs=1024k count=10k
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 123.34 seconds, 87.1 MB/s

Bonnie++ test:

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
CentOS           2G   444  99 78784   6 48588   1   957  97 190338   1 612.1   0
Latency             18285us     433ms     306ms   33926us   48079us     456ms
Version  1.96       ------Sequential Create------ --------Random Create--------
CentOS              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency              6908us     386us     402us     433us      23us      33us
1.96,1.96,CentOS,1,1304828315,2G,,444,99,78784,6,48588,1,957,97,190338,1,612.1,0,16,,,,,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,18285us,433ms,306ms,33926us,48079us,456ms,6908us,386us,402us,433us,23us,33us

I should re-run the bonnie tests with a large test size since these may be inaccurate with only 2g. Going purely off of the dd results, using LVM is the way to go.
The Sequential Output from bonnie++ is higher for the DomU using a file-based disk image, but that doesn’t match up to any of the dd tests on the same DomU. When I have time, I’ll re-run the bonnie tests with larger file sizes to see if the results stay the same or not.

I ended up buying iamj.us/tyn, and that’s what I’m going to set this up on. If I wanted to make things shorter, I could take off the /tyn but then I dont think it’d make as much sense. http://iamj.us/tyn11l redirects back to this page, for example. If you need help picking out a short domain name, try out domai.nr.

To create my own shortener, I decided just to use php’s base_convert function which will convert to and from bases 2-36. For a personal url shortener, you shouldn’t need more than base 36. I did end up having to write a base 62 converter class for sh0tz so that I can keep urls short, but that’s another post another time.

Create the Database

mysql> create database iamjurl;
mysql> grant all privileges on iamjurl.* to 'dbuser'@'localhost' identified by 'password';
mysql> flush privileges;
mysql> CREATE TABLE `urls` (
    ->   `id` int(10) NOT NULL AUTO_INCREMENT,
    ->   `url` varchar(1024) NOT NULL,
    ->   PRIMARY KEY (`id`)
    -> ) ENGINE=MyISAM DEFAULT CHARSET=utf8;

Super simple. One column for the url id, and one column for the url. You could add more fields for tracking views and that sort of thing if you wanted.

Write the PHP Guts
First, we need the ability to insert a new url into the database and echo out the short url.

if (isset($_GET['new'])) {
        //Somebody called /url.php?new to create a new short url
        if (isset($_GET['url'])) {
                //If url= isnt set to anything, then we don't really care since theres no link to add
                $dbcon = mysql_connect($db['host'],$db['user'],$db['pass']);
                if (!$dbcon) die ('Error connecting to db: ' . mysql_error());
                mysql_select_db($db['name'],$dbcon);
                $query = 'INSERT INTO urls (url) VALUES ("' . mysql_real_escape_string($_GET['url']) . '");';
                $result = mysql_query($query, $dbcon);
                if (!$result) die ('Invalid query: ' . mysql_error());
                //the new url's added to the database, now we get the auto_increment id and base_convert it to base36
                $shorturl = base_convert(mysql_insert_id($dbcon), 10, 36);
                //and echo it out to the browser
                echo "{$baseurl}{$shorturl} \n";
                mysql_close($dbcon);
        }
}

That should be fairly self-explanatory with the comments in there. So now you can go to http://iamj.us/url.php?new&url=http://google.com and it will add a new row to the urls table with http://google.com in it. It’ll also echo https://iamj.us/tyn1o or something similar to the screen. Whatever the baseurl is plus the base36 id of that row.

Now, if we actually want to be able to go to that link using a short url?

if (isset($_GET['go'])) {
        //somebody got redirected either with .htaccess or went directly to url.php?go=
        $dbcon = mysql_connect($db['host'],$db['user'],$db['pass']);
        if (!$dbcon) die ('Error connecting to db: ' . mysql_error());
        mysql_select_db($db['name'],$dbcon);
        //take go= and convert it back to base10 to match the mysql row id
        $shortid = base_convert(mysql_real_escape_string($_GET['go']),36,10);
        $query = 'SELECT url FROM urls WHERE id="' . $shortid . '";';
        $result = mysql_query($query, $dbcon);
        if (!$result) die ('Invalid query: ' . mysql_error());
        //While testing this, it's easier to echo $query out and make sure you're getting the right url
        //returned from mysql
        //      echo $query;
        //      echo mysql_result($result,0);
        //if we're not testing, then just redirect the user to the url we received from mysql
        header("Location: " . mysql_result($result,0));
}

Again, this should be self-explanatory. Go to /url.php?go=o and it converts the letter o to base10 which turns into 24, and then redirects the browser to the url stored in mysql with the id of 24. If you wanted to track views or other statistics, this would be the spot to do so(before redirecting.)

We don’t actually want to use /url.php?go=. We want to use(in my case) /tynXXXX. We do this with a simple mod_rewrite rule. This is what my .htaccess looks like:

RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(tyn)(.*)$ /url.php?go=$2 [NC,L]

The RewriteRule line would need changed if you’re not going to use a prefix to the shorturl(/xxx instead of /tynXXX). ^(.*) should be sufficient, and change $2 to $1.

Security
As it is right now, there’s not much security to this script. You can rename url.php to something like ASDF089234fasdf.php if you want to use security through obscurity. With an .htaccess, noone will know the actual filename of this script, and if you use the bookmarklet below you won’t have to remember it either.
If you plan on letting other people use it, adding checks for duplicate urls and reusing the same id would be a good idea. I’m already using mysql_real_escape_string but it wouldn’t hurt to sanitize the input even further.

Each time a row gets added, the id is only incremented by 1. Meaning it’d be easy for someone to get one url and then start going forward or backward down your list of urls just by subtracting/adding 1(in base36) to the url. To prevent this, try multiplying $shorturl by 303 or some other large odd number. Run this code to see what I mean:

<?php
/**
* basetest.php - Outputs a table showing how multiplying a number before base_converting it helps with obscurity.
* Author: Justyn Shull <justyn@justynshull.com>
*/
$magicnum = 303;
$i=1;
echo "<table><tr><td>id</td><td>Normal</td><td>x303</td></tr>";
while ($i<=50)
{
        echo "<tr><td>$i</td>";
        echo "<td>" . base_convert($i,10,36) . "</td>";
        echo "<td>" . base_convert($i*$magicnum,10,36) . "</td></tr>\n";
        $i++;
}
echo "</table>";
?>

See how the ‘normal’ column is easy to decipher, but the ‘x303′ column is a little more random?

Bonus!
Adding an actual gui to this script would make things awesome, right? Because you don’t want to urlencode URLs on your own and type in a long url everytime you want to make a short url. I’d recommend displaying an input box if url isn’t set and having the form method set to get.

Or you can do what I did, and just create a simple bookmarklet like this:

javascript:void(location.href='http://iamj.us/url.php?new&url='+escape(location.href))

Result
Go to http://iamj.us/url.php to see what my end result looks like for now. It’s essentially the same code in this post plus a few extras.

One solution, and the one I decided on for now, is to use Subversion.   Ever since 1.5, WordPress has been using subversion for its version control so of course they allow public read-only access to the subversion repository.   Whether you are already familiar with subversion or not, it is relatively easy to install WordPress using svn and then keep it up to date as well.

Here’s an example on my server:

$cd /var/www/vhosts/testdomain.com/httpdocs/
$svn co http://core.svn.wordpress.org/tags/3.0 .
.....svn will download the 3.0 branch of wordpress to the current directory ...

Be sure to include the period at the end of that svn checkout command so that it downloads wordpress to the directory you’re in. Otherwise it will create a new directory named 3.0. Once Subversion finishes checking out the branch, and assuming your permissions are all good then you can just go to http://testdomain.com/ and run the WordPress install script to put in your database info and whatnot. You could also modify wp-config.php manually if you’re into that.

And to upgrade wordpress?    Easy!   You just need to switch the branch subversion has checked out currently.

$cd /var/www/vhosts/testdomain.com/httpdocs/
$svn sw http://core.svn.wordpress.org/tags/3.0.1/ .

It will update all the files that need to be updated. Depending on the release, you’ll probably need to go to http://<wordpress install>/wp-admin/upgrade.php once the files are updated.

Now with this, you could create a bash script to loop through all of your wordpress installation directories and run the svn switch command to update to the latest stable wordpress branch.    You can find a list of the current available branches here: http://core.svn.wordpress.org/tags/

Just testing the CDN to see if this image gets uploaded automatically like it should.

It uses php’s shell_exec function to call the du utility on the directories it’s in. So if your host doesn’t allow you to use shell_exec, this isn’t going to work for you. If they do allow shell_exec, but do not allow ssh access then this is perfect.

You can view it here as well.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68

<HTML><HEAD><TITLE>Disk Usage Report</TITLE></HEAD><BODY>
<?php
///////////////////////////////////////////////////////////
//Disk Usage script v2                       ///
//du2.php                          ///
/////////////                         ///
//Author:  Justyn Shull                    ///
//E-mail:  justyn@justynshull.com           ///
/////////////                      ///
//based off my old du.php script:         ///
//This lists the directories in $dir with the   ///
//space they take up.  Not sorted, but allows  ///
//you to click on a directory to navigate     ///
////////////////////////////////////////////////
 
/////TODO: Fix it so you don't end up with the full path you've taken
//        eg:   du2.php?dir=../mail/../html/
 
$time1 = microtime(True);
chdir(getenv('DOCUMENT_ROOT'));
chdir("..");
$homedir = shell_exec('pwd');
echo "Home: $homedir";
if ($_GET["dir"] == "")
    //$dir = getenv('DOCUMENT_ROOT')."/.."; // *should* be the users's home
    $dir = ".";
else
    $dir = $_GET["dir"];
 
echo escapeshellarg($dir)."<br />";
chdir($dir);
echo shell_exec("pwd")."<br />";
$homeindir = strpos(shell_exec("pwd"),$homedir);
//if ($homeindir === false) die ("Invalid dir");  //hopefully this will keep us from leaving $home
 
exec('ls -la',$dirs,$rc);
if ($rc != 0) die("Error listing files");
$numdir = count($dirs) - 1;
for ($x = 1;$x <= $numdir; $x++){
    $type = substr($dirs[$x],0,1);
    if ($type == "d") {
    $lsdirs[] = substr(strrchr($dirs[$x]," "),1);
    }
}
$numdir = count($lsdirs) - 1;
 
// Print usage now
echo "<br />";
for ($x = 0; $x <= $numdir; $x++) {
    if ($x == 1) {
        echo "<a href=du2.php?dir=$dir/$lsdirs[$x]>";
        echo "<img src=\"/icons/folder.gif\" border=0></a>    ..<br />";
    } else {
    unset($du);
    exec("du -hs ".escapeshellarg($lsdirs[$x]),$du,$rc);
    if ($rc != 0) die("Error on ".$x.": ".escapeshellarg($lsdirs[$x]));
    echo "<a href=du2.php?dir=$dir/$lsdirs[$x]>";
    echo "<img src=\"/icons/folder.gif\" border=0></a>".$du[0]."<br />";
    }
}
 
 
$time2 = microtime(True); 
$dutime = ($time2 - $time1); 
echo '<p>Page generated in <b>'.$dutime.'</b> seconds</p>';
?>
 
</BODY></HTML>

I still need to install/configure subversion, and move some of my old stuff over to wordpress.   There isn’t much to move, so it won’t be hard.    The old version of the site can be found here:

http://justynshull.com/oldsite/