Much better would be simply to backup to S3. This is easily done through a little script that sits in your /etc/cron.daily so that it is automatically called once every day. But I’d like to have more than just one backup – I’d like some kind of automatic rotation of backups. In the end I came up with the a reasonably cute idea, and that is to keep between 28 and 31 backups: ie, all the data that I’m backup up is pushed into a folder named after the day-of-the-month in an S3 bucket, so today (7th June 2013) all my backups are going into a folder named something like s3://mybackups/07

In a month’s time (7th July 2013) this backup will be overwritten by the July 7th backup. That’s not a bad solution really. If you want longer backups you can hack the below script and have two scripts – one for a one-backup-per-day strategy and also another copy of the script that stores by month name which essentially rotates by month. That would give you daily backups for the past month, and monthly backups for the past year. Useful.

In terms of your standard webserver you want to backup your /var/www (or wherever you keep your htdocs), along with any config info, so I also backup /etc/apache2 /etc/php5 /etc/mysql /etc/cron.daily. Of course on top of that you’ll need a backup of your database which you can get by calling mysqldump. Then compress the lot and chuck it up to s3. You’ll want to use s3cmd for this. Thankfully I found a script which I modified to do the rotations and use s3cmd to upload the resulting backups (original source: http://stnor.wordpress.com/2011/08/01/backing-up-ec2-mysql-and-configuration-files-to-s3/)

So first you’ll need a copy of s3cmd installing. On ubuntu/debian that is a fairly straight-forward:

sudo apt-get install s3cmd
s3cmd --configure

This is important: Do NOT configure s3cmd with your root AWS credentials – yes it will work, but would you store your root server password in a plaintext file? No, and your AWS credentials give the holder access to unlimited resources, your billing details, your machine images, everything. Just watch this 2-minute you-tube video on creating AWS users & groups with restricted access, create a new user/group that only has access to S3 and use those credentials to configure s3. It’s not hard, it’ll take you just a few minutes to do. Then wait a couple more minutes for these new credentials to propagate through amazon’s systems and you’re ready to carry on. Ok, safety first rant over! Let’s continue.

Then modify the following script to suit your purposes:

1. 1. Specify the names of your mysql databases in that you need backing up in DATABASES
2. 2. Add mysql login details for each DB in the format: databasename_USER and databasename_PW
3. 3. Specify which directories to backup in DIRECTORIES – for me that is config stuff and my /var/www
4. 4. Specify the name of the s3 bucket you’re going to backup into in the S3_BUCKET_URL

The script also assumes you have tar and gzip installed, but I’ll assume you can figure that bit out for yourself.

## Specify data base schemas to backup and credentials
DATABASES="wp myotherdb"

## Syntax databasename as per above _USER and _PW
wp_USER=username
wp_PW=password
myotherdb_USER=username
myotherdb_PW=password

## Specify directories to backup (it's clever to use relaive paths)
DIRECTORIES="/var/www root etc/cron.daily etc/cron.monthly etc/apache2 etc/mysql etc/php5" 

## Initialize some variables
DATE=$(date +%d)
BACKUP_DIRECTORY=/tmp/backups
S3_CMD="s3cmd"

## Specify where the backups should be placed
S3_BUCKET_URL=s3://mybackupbucket/$DATE/

## The script
cd /
mkdir -p $BACKUP_DIRECTORY
rm -rf $BACKUP_DIRECTORY/*

## Backup MySQL:s
for DB in $DATABASES
do
BACKUP_FILE=$BACKUP_DIRECTORY/${DB}.sql
USER=$(eval echo \$${DB}_USER)
PASSWORD=$(eval echo \$${DB}_PW)
/usr/bin/mysqldump -v -u $USER --password=$PASSWORD -h localhost -r $BACKUP_FILE $DB 2>&1
gzip $BACKUP_FILE 2>&1
$S3_CMD put ${BACKUP_FILE}.gz $S3_BUCKET_URL 2>&1
done

## Backup of config directories
for DIR in $DIRECTORIES
do
BACKUP_FILE=$BACKUP_DIRECTORY/$(echo $DIR | sed 's/\//-/g').tgz
tar zcvf ${BACKUP_FILE} $DIR 2>&1
$S3_CMD put ${BACKUP_FILE} $S3_BUCKET_URL 2>&1
done

Then, assuming you’ve called it something like backupToS3.sh, make it executable and test it:

chmod +x backupToS3.sh
sudo ./backupToS3.sh

Once you’ve ironed out any issues simply copy it over to /etc/cron.daily so that it runs daily:

sudo cp backupToS3.sh /etc/cron.daily

Now, the above script does daily backups, but if you want to do monthly backups you simply need to make a copy of the file (since you’ll likely want a daily and monthly backup rotation) and edit the DATE variable to use months rather than day-of-the-month. If you use the month number you’ll probably want to either prefix the month number with the word “month”, or pop them into a subdirectory called “monthly”, alternatively you could use the month name, for instance:

DATE=$(date +%m)        // month number
DATE=$(date +%b)        // 3-letter month name
DATE=$(date +%B)        // full month name
DATE=$(date +%m-%B)     // month number, dash, full month name

Then make it executable and test it as you did the previous script, and then copy it into cron.monthly:

sudo cp monthlyBackupToS3.sh /etc/cron.monthly

Presumably this will then fire on the first of the month (I haven’t checked), but you could always put it in cron.daily so that monthly backup is from the last day of its month (for previous months, the present month would be up to date).

The fixes that people are fudging in aren’t very safe at all: the suggestion is to put set a transparent 1×1 pixel .gif file as the background of the relevant tags, and one solution I saw applied to to every tag using the * selector.

The problem with using this technique is that:

1. It’s another file to download.
2. The performance hit on redraw times in IE8 and lower essentially breaks your website if you’re using it for every p tag, (or worse for them all)

The problem is that CSS doesn’t allow detection of auto-fit, or Android (rightly, and Android Chrome doesn’t have auto-fit either). So here was my solution:

/* work around mobile device auto-fitting */
@media only screen and (max-device-width: 800px) {
   #content p {
        background-image:url(data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw==); 
        background-repeat:repeat;
    }
}

On my website, ‘#content’ is where the ‘p’ tags reside that are being auto-fitted, naturally you’d need to change this. ‘body’ will work, yes, but the more specific the lower the impact on redraw time.

What this does is :

1. Embed the 1×1 pixel directly into the style-sheet as a data-insert rather than having the browser do another file load, and
2. Only target smaller devices (since auto-fit only applies to smaller devices), hence ruling out the IE8 which is desktop only.

Alternatively you could apply this to everything and then override it in your IE8 stylesheet, but why inflict this fudge on desktop users at all?

sudo apt-get install php5-tidy

So first fix the malformed HTML:

$html = file_get_contents("myfile.html");
$config = array(
    'indent' => true,
    'output-xhtml' => true,
    'wrap' => 0);
$tidy = tidy_parse_string($html, $config, 'UTF8');
$tidy->cleanRepair();

And then load it into DOMDocument:

$doc = new DOMDocument();
$doc->loadHTML($tidy)

Then it’s just a matter of ripping out the tags you don’t want. Note how we’re iterating through the $nodes variable – it MUST be done this way if you’re planning on removing the nodes (as I am) because as they’re removed they also disappear from the collection. A foreach will do some odd stuff – probably terminate after the first node, and a for-loop will have you missing every other node. Instead, just remove the first child until there are no children:

$nodes = $doc->getElementsByTagName("script");
while ($nodes->length > 0) {
    $node = $nodes->item(0);
    remove_node($node);
}
function remove_node(&$node) {
    $pnode = $node->parentNode;
    remove_children($node);
    $pnode->removeChild($node);
}
function remove_children(&$node) {
    while ($node->firstChild) {
        while ($node->firstChild->firstChild) {
            remove_children($node->firstChild);
        }
        $node->removeChild($node->firstChild);
    }
}

It should be simple enough to undo the last change you made to a system file when booting into recovery mode, but alas it is not straightforward because when booting into recovery mode the filesystem is mounted as read-only! Oh dear!

1. Boot into “Ubuntu (Recovery Mode)”
2. From the recovery menu drop to a root shell.
3. Enter the following command to remount your filesystem as read-write:

mount -o remount,rw /

4. Make the required changes to your operating system and then exit/reboot/whatever.

Hope that helps someone out there!

apache2ctl status

via a cronjob and this gives me various stats on how many servers are used/free. Useful. But upon moving some old websites off the server the above call began to fail – I’d get the error message

Forbidden
You don't have permission to access /server-status on this server.

There are lots of solutions around the web (some about SELinux, some about setting allow/deny in an apache config file under the location /server-status namespace), none of which actually solved my problem. They did give enough hints for me to work out what was going on! You see when you make the above call to apache2ctl it then (through mod_status) requests /server-status on the default website on your webserver. Now if you only have one site then that should be pretty easy to work out which one it is going to, but if you’re hosting multiple sites then its not quite as obvious. For me it was the first site in the /etc/apache2/sites-enabled directory, (or you can find which one it is by visiting the webserver via its IP address rather than through a domain name as it is the domain name that is used to direct the browser call to a specific site that you host – if you visit your webserver via its IP address it doesn’t really know what to serve you so you’re sent to the default site).

So my default site was refusing to serve /server-status. I couldn’t think why this was until I visited my .htaccess file and realised that I was using mod_rewrite to rewrite any www.mysite.com calls to mysite.com like so:

RewriteEngine On
RewriteCond %{HTTP_HOST} !^mysite\.com$
RewriteRule (.*) http://mysite.com/$1 [R=301,L]

The answer then was to pop in an exclusion for /server-status so that it would be unaffected by mod_rewrite:

RewriteEngine On
RewriteCond %{HTTP_HOST} !^mysite\.com$
RewriteCond %{REQUEST_URI} !^/server-status
RewriteRule (.*) http://mysite.com/$1 [R=301,L]

And hey presto it works! Hurrah!

UPDATE: I’ve found a better version. I host the site on a virtual server (amazon’s ec2 stuff) and that means I often need to test a new instance of the server using either the IP address or a funny amazon made up name like ec2-99-98-97-96.compute-7.amazonaws.com, but if I stick that in the address bar of a browser the .htaccess reroutes it to mysite.com (the one that is live), so really I just need to slice off the www from any HTTP_HOST that is found and keep an exception in there for /server-status so that apache2ctl status doesn’t complain. Here’s what it looks like now:

RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_HOST} ^www\.(.*)$ [NC]
RewriteCond %{REQUEST_URI} !^/server-status
RewriteRule ^(.*)$ http://%1/$1 [R=301,L]

If you wanted to be ultra portable you could put an if around it:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_HOST} ^www\.(.*)$ [NC]
RewriteCond %{REQUEST_URI} !^/server-status
RewriteRule ^(.*)$ http://%1/$1 [R=301,L]
</IfModule>

curl: (6) name lookup timed out
Ouch, curl --progress-bar -I --max-time 5 --url https://route53.amazonaws.com/date --insecure failed with exit status 6

I couldn’t work out whether it mean that some sort of insecurity had made it fail (please use quotes to show where the command ends in future!). So I ran the command above (from curl onwards) on its own to see what would happen:

curl --progress-bar -I --max-time 5 --url https://route53.amazonaws.com/date --insecure

…and sure enough, back came the error:

curl: (6) name lookup timed out

Ok, so its time out or the site isn’t up at all, not a security related issue. I then browsed to https://route53.amazonaws.com/2011-05-05 (correct at the time of publishing) and found that it took more than 5 seconds, but that it did eventually connect. It came back with:

UnknownOperationException/

So the site is working and up, but that it was taking quite a while. In the above curl command we’re just giving it 5 seconds to respond which isn’t too long in the world of requests really, so I altered the above command to set the time-out to be 15 seconds rather than 5 and it worked fine (http code 200):

curl --progress-bar -I --max-time 15 --url https://route53.amazonaws.com/date --insecure

So somewhere in dnscurl.pl there’s a line similar to the above one but it only allows 5 seconds for a response, absolutely not long enough, so we need to change it to something more sensible, perhaps 20 or 30 seconds will do. I’ve never played with perl, but how different can it be right? I openned dnscurl.pl in an editor and found what looked like the right reference to CURL on line 190 (your results may vary), which looks like this:

my $curl_output_lines = run_cmd_read($CURL, "--progress-bar", "-I", "--max-time", "5", "--url", $url, "--insecure");

Simply changing the number 5 to a 30 seemed to solve the problem because the next time I called dnscurl.pl it all worked splendiferously.

1. I got the useless message because I hadn’t created an AndroidManifest.xml file, new projects don’t have them by default, so right-click on the project, go to properties, then Android Manifest, and create one. Fill in some details, the file needs to exist anyway, but you can get away with not having one until you put an ImageView in there.

2. Then when you rebuild the project you’ll get a sensible message out of aresgen.exe, which for me was an incorrectly spelt resource name. My .png file was in the correct directory (Resources/Drawable) but I’d misspelled it by one letter.

Hope this saves someone else 30 minutes of searching around for answers.

RewriteEngine On
RewriteRule ^.*$ http://example.com/$0 [R=301,L]

Obviously switch example.com for your domain name and any requests which hit this folder or below will be redirected and keep the subfolders, filename and any arguments that were passed on. You can also do some interesting things with the user agent. For instance, if you want to only forward search engines you can add a rewrite condition such as the following which will only redirect altavista and lycos’ search engine bots. This is called cloaked forwarding and is usually against the terms and conditions of the search engines (ie, they will penalise you if they think you’re forwarding their bots but not users), so don’t do it unless you really hate that user agent!

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} AltaVista [OR]
RewriteCond %{HTTP_USER_AGENT} lycos
RewriteRule ^.*$ http://example.com/$0 [R=301,L]

Setting up SVN looks tricky at the outset, but it really isn’t. Assuming you’ve already got an AWS account…:

1. At the AWS management console “Launch Instance”, go to “community AMI’s” and select Alestic’s latest 32-bit Ubuntu build (you can find a list at alestic.com – click on the region you want (ie US-East 1), and then copy the AMI-1234567 and paste it into the search box on the “community AMI” AWS page. There should be a star next to it to indicate that it will be free micro instance. Click select to use that AMI.

2. Make sure the instance type is micro. If you can’t select a micro instance then you need to use a different AMI (or a different region). Then follow the options and set up the server as you would normally. I won’t cover that here (there are plenty of tutorials out there for setting up an instance), though do get yourself an elastic IP and assign it to the new instance.

3. Now your instance is running you need to open up TCP port 3690 for the SVN service to hear from the outside. Find the security group that your instance is using, select it, click on the “inbound” tab, and add a new rule: TCP, 3690, 0.0.0.0/0 and then click “add rule” and then “apply rule changes”.

4. SSH into your instance and run the command “sudo apt-get install subversion” to install svn

5. Create the directory to be used for subversion, for example “mkdir ~/svn”. Your projects will go inside this folder. The newer instances may require you sudo the commands if they don’t work – if you do use sudo you’ll need to change the permissions or ownership on the svn folder and subfolders.

6. Create the project: “sudo svnadmin create ~/svn/myproject”, again you may need to set the permissions here “sudo chmod -R g+rws ~/svn”

7. Under the new project directory (in the conf subfolder), edit svnserve.conf, remove the # before “anon-access” and change it to “none”, then remove the # before “auth-access” and “password-db” lines. Remove the # before the “realm=” and give it a custom name.

8. Edit passwd, add a line entry with the user name(s) and password(s) to be used for access.

9. The SVN server will work locally, but we need it to respond to remote instructions, so set up the service by calling “svnserve -d -r ~/svn” Now in your SVN client of choice commandline (why?), TortoiseSVN (windows) or RabbitSVN (linux) you can access via the SVN protocol: svn://[elastic ip address]/[project name] using the username and password you set in the passwd file in step 8.

If your instance is rebooted you’ll need to SSH back in and repeat step 9. In order to have it start automagically at boot up follow Michał Wojciechowski’s instructions under “Svnserve Initialization Script” which worked for me!

Find will return a list of files and directories, one per line which can be piped into xargs so that the same command can be repeated on each file found, for instance:

find *.png

Will find .png files in the current directory. If you wanted to change the permissions on just these .png files you could then pipe the output into xargs:

find *.png | xargs chmod 775 -v

The other day I wanted to change the permissions for a subfolder called “logs” that was in 10+ folders:

find . -maxdepth 2 -name "logs" | xargs chmod 755 -v

Naturally, one should test the find to see what is returned before piping the result into an xargs, especially if you’re doing something that cannot be easily undone (like rm for instance).

The final thing that is worth knowing about is argument replacement. All the above examples assume that it is ok to put the result of the find command on the end of the xargs command, but if you were doing a rename operation or a copy this wouldn’t be the case. In the case below I’ve used the -I xargs command to define an identifier which will be swapped for the result from find:

find . -maxdepth 2 -name "logs" | xargs -I xxx mv xxx xxx.l

So if one of the lines coming from find is “subfolder/logs” then the mv command executed will be:

mv subfolder/logs subfolder/logs.l

A useful tool.