In working with a few other APIs, they eventually add rate limits, or get slower to respond to requests as their user base grows. To avoid these issues I decided to cache calls to the Readability API instead of doing them live on the site, client side. It isn’t anything terribly complicated, but I’ll review the code for the script.

#!/usr/bin/python

import readability
from datetime import date, timedelta
from sqlalchemy import *

These are the required libraries for the script.

'''		readability config		'''
token = readability.xauth('read_username', 'read_apikey', 'read_email', 'read_pass')
rdd = readability.oauth('read_username', 'read_apikey', token=token)

These lines are the config for the oauth call to the Readability API. You’ll need to acquire your API Key from the Readability site and then substitute all the parts here.

'''		sqlalchemy setup		'''
engine = create_engine('mysql://db_username:db_passwd@db_host/db')
connection = engine.connect()

In searching and talking to some people it looked like SQLalchemy was going to be the best way to get Python to work with MySQL. This library made this script much easier than I expected.

'''		Fetch new bookmarks since yesterday		'''
yesterday = date.today() - timedelta(1)

for b in rdd.get_bookmarks(added_since=yesterday.strftime('%m-%d-%y')):
	result = connection.execute("INSERT INTO bookmarks VALUES(NULL, " + str(b.id) + ", '" + b.article.title.encode("utf-8") + "', '" + b.article.url.encode("utf-8") + "')" )

For your initial pull of your bookmarks into your database you would just need to change the for statement to “rdd.get_bookmarks()”. The data that is saved in the database, as you can see, is the readability bookmark id, the title and the URL for the story. There is more available in the API, but this should be sufficient to display a ‘currently reading’ list in a widget on a webpage.

The end of this would then be to schedule this script in cron to run once a day. If you find yourself saving bookmarks more often, you may want to update these schedules.

The database structure and code are on Github

Awhile back I started using Amazon S3 for cloud storage online and for website backups. The more I used it, I started becoming more interested in their other offers in the AWS suite. I finally sat down with their calculator and started doing the math to figure out how much it would cost to at least move this blog over to full Amazon services. Beyond paying for features I wasn’t using, another thing that always weighed on me was single point of failure for my websites. They had backups, and I could restore them fairly quickly. But how resilient were they if the hosting company had an outage? What if I wasn’t around to run the site restore? I wanted a more resilient web environment, partly for myself, but for other people I host as well.

I’ve always ran my websites on the normal LAMP (Linux/Apache/MySQL/Perl|PHP|Python) stack. Simplest way to get this going in the AWS system was to get a reserved EC2 instance setup, and I chose the Amazon distro of Linux for now. I’m not going to go through a step by step setting up LAMP, there are plenty of guides for that. I’d really like to talk about Amazon’s ability to provide me the cheapest redundancy that I could find in a reliable hosting company.

Here is a diagram of what I setup for this site. I’ve since added a load balancer in front of the server, but the rest is the same.

Diagram of jimwalsh.name.

If Amazon were to have a regional outage, they have datacenters in multiple regional zones you can deploy servers into. Many of their services are managed services, meaning that you dont need to install and run the applications like you would traditionally. For example some of their managed services are DNS, Load Balancing, Messaging, Queuing, Caching and Database to name a few. You would have to explore the pricing options to see if the managed service is worth it over setting it up manually on a server yourself. For me getting this off the ground I went with managed DNS and Load Balancing and Messaging. Database was a bit expensive, outside of the trial period, for a small blog. If you had a large application it quickly becomes a very attractive offering though.

The versatility and agility in which you can deploy servers let’s you architect just about anything you can imagine. I went from a single server that could fail and I would be offline until my provider fixed the issue, potentially requiring me to run a restore. To now having an image of my server ready to deploy in any regional zone if needed. Then nightly backups to get the image up to speed. I’m only using that method now because it keeps costs down. It would be just as easy to have hot spare servers in other regional zones and just turn them on automatically if another one goes down. I can even put up a few more servers behind the load balancer and just run a normal web farm if I needed the power. Those are just a few of the possibilities, all with pricing that is very competitive to other places.

They also have something that is rather unique in their auto scaling offering. You can define thresholds that can kick off jobs to create/start extra servers to handle additional load, as well as jobs to reduce it back down as well. Very cool stuff.

All in all I’ve been very pleased with the move. Setup was simple and easy and the options available give me more recovery and resiliency options than I’ve ever had in the past. Definitely check into it if you are starting to outgrow your host.

They chose to blame the developers.

Myself and others have clamored for quite awhile that vendor prefixes should stay in their development trees and not enter their production releases until they are in the official CSS spec. The vendors could then pressure CSS WG to move more quickly and with more agility than they are now. W3C’s slow pace at formalizing some of these definitions is really why we are in the place we are now. Opera, is more or less calling developers lazy because they were ‘only’ including webkit prefixes instead of the ‘standard’ five prefixes that are required to have a rule properly work and failback in all browsers. So let me get this straight. Opera is calling out developers for being lazy about addressing a problem that the browser manufacturers are responsible for creating? That makes sense to me, how about you?

The effort should not be spent adding other vendors prefixes to your product. If Opera felt that the web is making a bad shift towards excluding standards, and ‘standardizing’ on using one particular vendor prefix, then you should fight with the standards group and get traction made there. Instead they went the other direction, threw their hands up and just went the easy route of just adding those prefixes to their product.

Within the next few weeks I would expect Firefox and Internet Explorer to announce the same. Why would they choose to be ‘left behind’? Time and effort needs to be directed towards advancing the standards at a greater rate. With the recent explosion of CSS preprocessors I think it’s fair to say that it’s time to move beyond hand writing CSS code. There should be an Open Source product that all the vendors and the community could contribute. This product should produce standards based CSS that works in all browsers, with the same results across the board. It’s 2012, why are we still fighting with CSS and cross browser compatibility? Let’s move the web forward, but now even further.

Maybe Opera should kick off that Open Source CSS Generator project, and help repair this mess they are only helping to make worse.

For now we’ll focus on the various ways I can use the S3 storage on my Mac. One of the best FTP clients is Panic’s Transmit. They added the ability to transfer files to S3 some time ago. It also has an option to mount any of the sites you setup in the application as drives. Using Transmit proves to be a very quick and easy way to get files up to S3.

I started looking into various methods I could use to mimic Dropbox’s filesystem monitoring and what tools I could use. Not many of the people I chat with about web design and development talk about unit testing. I used it when redeveloping the new Gamegossip homepage and backend, and many programmers use it. What some people do is to setup their unit testing software to automatically check files when they change. So I started looking into these types of apps that were watching the filesystem and sending notifications. The end result being that I could pickup these changes, log them, and then sync them to S3.

That search lead me many places and some of them I’m still working on, but I got a quick and dirty solution working so I thought I’d share that first. The solution involves an app called JetS3t. The initial setup for JetS3t is trivial so I wont go into detail with that. What I wanted to do with this was schedule some directories to be synchronized and schedule that to happen every so often. I settled in on wanting it to check every five minutes. Knowing that some uploads were probably going to take longer than 5 minutes I needed to figure out a way to not launch multiple syncs of the same data. Here is what I came up with.

Write a wrapper script that checks to see if the sync script is running. If it isn’t, then fire off then sync script. I wanted to do this PID based but that wasnt working as well as I wanted so I just used trusty old ps.

s3wrapper.sh

#!/usr/bin/bash

if ps -ef | grep -v grep | grep synchronize.sh ; then
  echo "Already running."
  exit 99
fi

/jets3t/bin/synchronize.sh DOWN jpw/Books /Users/jwalsh/Documents/Books/* > /var/log/s3upload.log &
/jets3t/bin/synchronize.sh UP jpw/Books /Users/jwalsh/Documents/Books/* > /var/log/s3upload.log &

You just keep adding lines like the last two lines to add other folders to sync up to S3. The syntax is pretty easy, as noted in the JetS3t documentation. We are using ‘UP’ for the upload and ‘DOWN’ to download, the next part is the S3 bucket and folder inside it. The last part is the folder to sync, and then logging the actions. You may want to follow this up by adding a logrotate entry so that the log doesnt get too large. We need to use both UP/DOWN to ensure you capture any changes fully.

Once you have that script written you just need to schedule it in Crontab. You type the following to start the Crontab editor.

crontab -e

This line will run the script every five minutes. If you are not familiar with Crontab syntax, there are generators and tutorials around to assist. Once in the Crontab editor, normal vi commands will work.

*/5 * * * * /jets3t/s3wrapper.sh

Now your folder will be checked and synchronized every five minutes as long as your computer is on. The upgrade to this script would be me to find something or write something that integrates with FSEvents API to detect changes to monitored folders and then kick off the syncs then, instead of scheduling them. That version would also have to include caching the changes in case the Internet connection is not available. The advantage to all of this may be minimal to some, others may love being able to have full control over their data. This method should at least get you on the right path. Good luck and let me know how this works out for you!

Twitter has a doc on their various Embedded Tweets options. But we are going to focus on the oEmbed version. We will need some things setup before we get started with the meat and potatoes of this exercise.

We will be utilizing the ID of a Twitter post for this example. This is the long number in the URL of a Tweet. In the new Twitter layout just click a Tweet and hit Details. You will then be taken to the URL for the post and you can copy the number at the end of the URL for this. Example: https://twitter.com/#!/jpwalsh1/status/145224413200130048. The API also allows you to just send the whole URL if you choose to write your app that way as well.

Now we need to setup a database for the caching of the Tweets. Here is some code you can use to do that.

CREATE TABLE `twitter` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `tweetid` bigint(30) unsigned NOT NULL,
  `html` text NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=MyISAM  DEFAULT CHARSET=utf8 COMMENT='tweet cache';

With that setup, now create a new PHP file and add your typical code you use to connect to your database. We are going to create two functions, one to check if the Tweet is in the cache, and one to add it to the cache.

function check_cache($tweet){
	
	$qCheck_cache = "SELECT * from twitter where tweetid =" . $tweet;
	$rCheck_cache = mysql_query($qCheck_cache);
	
	if (mysql_num_rows($rCheck_cache) == 0) {
		// Tweet isn't cached, grab and save.		
		$html = add_cache($tweet);
		
	} else {
		// Tweet is in the Cache, grab it.
		$cached_tweet = mysql_fetch_assoc($rCheck_cache);
		$html = stripslashes($cached_tweet["html"]);
	}
		
return $html;
	
} // end check_cache

Now let’s review what this function does.

Lines 3-4: Builds and checks for the Tweet in the database.
Line 6: Checks to see if the query to find the Tweet returned anything.
Line 8: Call the add_cache function to add this Tweet to the database and return the html to be displayed.
Lines 12-13: Pulls the Tweet out of the database, removes extra slashes in the HTML required to store it in the database and returns it.

Now we will check out that add_cache function and see what it does.

function add_cache($tweet){
	
	$twitterapi_url = "https://api.twitter.com/1/statuses/oembed.json?id=";
	$twitterapi_url = $twitterapi_url . $tweet;
	
	$curl = curl_init($twitterapi_url);
	curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
	$response = curl_exec($curl);
	curl_close($curl);
	
	$json_content = json_decode($response, true);
	
	$html = $json_content['html'];
	$qInsert_cache = "INSERT INTO `twitter` (`tweetid`, `html`) VALUES (" . $tweet . ",'" . addslashes($html) ."')";
	$rInsert_cache = mysql_query($qInsert_cache);
	
	return $html;

} // end add_cache

Lines 3-4: Build the API URL
Lines 6-9: Leverage CURL to fetch the API URL and JSON return.
Line 11: Decode said JSON content
Line 13: Grab the HTML element out of the JSON array.
Lines 14-15: Build the SQL to add the Tweet to the database. Adding slashes in the HTML code so that it goes into the database properly, and then execute the query.
Line 17: Return the HTML to be displayed.

Now to use these functions in this test page you will use this code.

$tweet_id = $_GET["id"];	
$tweet_display = check_cache($tweet_id);

echo $tweet_display;

Line 1: Grab the ID variable from the URL.
Line 2: Call the functions, passing the Tweet ID and saving the result.
Line 4: Print the result.

Now obviously there is no error checking shown, and this is more a technical demo than something you should copy and paste into your site for use. But this should give you a better understanding how to cache the result from an oEmbed API. Let me know what you think, or what you have come up with as well. You can also add in how you would like to expire your cache, if at all.

Here is the example URL
http://jimwalsh.name/demos/oembed/oembed.php?id=145224413200130048
You can change the ID in the URL to the ID of another Tweet to see it change.

The functions are also available on Github.

It also turns out that if I would have read the man page of tar for a second I would have found the simple manner in which you can create incremental tar files.

-g, --listed-incremental F
    create/list/extract new GNU-format incremental backu

As you can see the -g option will create our (.snar) file that will store what is in the backup. Before a full backup you will want to remove .snar file so that all the files will be grabbed by tar. I use the Ruby wday function to determine what day of the week it is, zero is Sunday, the day I want to run full backups. I also append the three character abbreviation for the day of the week at the end of the backup file name. This is partly because I wanted to store all the backups for a week in one S3 bucket, so we needed unique names. You will want to backup your .snar file too, dont forget about it.

Here is the new Filesystem backup section.

## Filesystem Backup ##
#

site_file = "#{backup_name}_#{t.strftime('%a')}.tar.gz"

# On Sunday do Full Backup
if t.wday == 0
# Delete incremental file
        system("rm -f fs_backup.snar")
end

# Run filesystem backup
system("tar -cz -g fs_backup.snar -f #{site_file}")
#

Now we move on to the MySQL incremental backup. This was a bit more involved. This requires a MySQL configuration change to enable Binary Logging if you don’t already have this enabled. I enabled it by placing ‘log-bin’ as the first line in my my.cnf and then restarted MySQL and the binary log showed up in the MySQL directory. You will now need to alter the grants setup on your backup user account. You will need to grant Reload and Super to the system. After that is complete you will need to grant Select and Lock on the individual databases that you want to backup with this user account.

Sample Grants

mysql> show grants for backup@localhost;
+-----------------------------------------------------------------------------------------------------------------------+
| Grants for backup@localhost                                                                                           |
+-----------------------------------------------------------------------------------------------------------------------+
| GRANT RELOAD, SUPER ON *.* TO 'backup'@'localhost' IDENTIFIED BY PASSWORD '[encoded passwd]' |
| GRANT SELECT, LOCK TABLES ON `sweetDatabase`.* TO 'backup'@'localhost'                                                          

Now that you have your user account setup, and MySQL creating binary logs we can talk about backing them up. Instead of creating individual files for each database, we will dump out all the databases into one file. This seems to be the easiest way to get the backups working with binary logging, if anyone knows a way to get each database individually let me know and I’ll post an update.

Again on Sunday we will run a full MySQL dump of all the databases. In the mysqldump command when we pass –flush-logs it will rotate the binary log. We will also pass –delete-master-logs so that it removes all the old binary logs so that we don’t manually have to manage the logs filling the drive. I’m debating finding a way to have these logs removed during incrementals too, since we upload the logs each day, it seems redundant to keep them on the system and upload them each day of the week.

For our incremental backups all we need to do is rotate the binary logs and then back them up. I will grab all but the newest log, and copy them to the temporary backup directory so they get uploaded to Amazon and the FTP. You will need to add each database that you want backed up to the mysql dump command, one after another separated only by spaces. The locations of the binary logs can be changed in the my.cnf, and even your file path may be different than what I have listed below, so update that path to match where your logs are located.

New Database backup section.

## Database Backup ##
#

if t.wday == 0
#Today is Sunday, run the full backup
        full_backup_cmd = "mysqldump -u #{db_user} -p#{db_pass} --flush-logs --delete-master-logs --master-data=2 --add-drop-table --lock-all-tables --databases database1 database2 database3 | gzip -9 > db-backup.sql.gz"
        system(full_backup_cmd)

else
#Roll logs for incremental backup
        system('mysqladmin -u#{db_user} -p#{db_pass} flush-logs')

        #Copy old logs to directory to be backed up

        logs = Dir.glob("/var/lib/mysql/mysqld-bin.[0-9]*").sort
        logs_archive = logs[0..-2]
        logs_archive.each do |log|
                 system("cp #{log} #{backup_dir}")
        end #end copy
end #db backup
##

The last part to do now is to clean up your backup directory, so just update your rm command you may be using or whatever technique you chose to use.

This time around we’ll be writing the script in Ruby, mostly because Amazon provides an sdk that will make it very easy for us to upload our files to their S3 service. I’m going to assume that you signed up for an S3 account (free for a year) and you grabbed your two security keys, and now we’ll jump into the script. This first version of the script is only doing full backups. I hope to set some time aside and get the incremental parts created.

Script Configuration

Here we will look at the required libraries from Ruby, and you can read through the config section it is very self explanatory, so we wont spend much time talking about this. This script does assume that you already created a bucket in S3, if you want to create one each upload, that is just one more line of code you can add to this script. I got lazy and I use the same user/pass for my db backup account and my remote FTP account. This is easily separated if you require that as well.

#!/usr/local/bin/ruby
require 'rubygems'
require 'aws/s3'
require 'net/ftp'
require 'time'
require 'fileutils'

include AWS::S3

## Timestamp for backup
t=Time.now
timestamp = t.strftime('%d-%m-%Y')

## Config
# Connect to S3
AWS::S3::Base.establish_connection!(
  :access_key_id     => 'S3 Access Key',
  :secret_access_key => 'S3 Secret Key'
)
s3_bucket = 'S3 bucket to upload the backups'
db_user = 'mysql/ftp backup username'
db_pass = 'mysql/ftp backup password'
site_dir = '/dir/to/files/to/backup/*'
url = 'hostname/ip of place we want to FTP the files off to'
directory = 'remote ftp directory to upload into'
backup_name = 'name for filesystem tarbal'
##

MySQL Database Backup

Note: Before you can run this chunk of code you will need to configure your MySQL backup account properly. I recommend doing grants for each database that you want to backup. The ‘SELECT’ and ‘LOCK TABLES’ are the only permissions that you need to grant to the account.

We start this snippet by building the command to obtain a list of databases from the server. We then loop through this list of databases, generating a mysqldump command, and then executing it. If the script comes to a database that it doesnt have access, it will skip it and no harm will be done. The last two lines in the loop were a clever way to get the loop to wait for the mysqldump command to finish before doing the next object in the array. Otherwise it will go so fast you’ll have multiple mysqldump threads running and you get errors about hitting your open files limit quite quickly. If some of you have a more elegant way to get around this issue, leave a comment or send me a tweet.

## Database Backup ##
#

command = "mysql -u #{db_user} -h localhost -p#{db_pass} -Bse 'show databases'"

IO.popen(command) do | databases |
	databases.each_line do | db |
		dbase = db.split.join("\n")
		file = "#{dbase}.backup.sql.gz"
		dumpdb = "mysqldump -u #{db_user} -h localhost -p#{db_pass} #{dbase} | gzip -9 > #{file}"
		output = IO.popen(dumpdb)
		output.readlines	
	end
end
##

Filesystem Backup

The filesystem backup is quite simple. I set my backup path to be the folder all my websites are hosted. Then I just backup this entire folder into one large tarball. Currently S3 has a 5GB file limit, so depending on how large your tarball gets you may want create multiple tarballs. You should also ensure that the user account you choose to run this script has access to all these folders so you get all the files into the tarball.

## Filesystem Backup ##
#

site_file = "#{backup_name}.tar.gz"
system("tar -czf #{site_file} #{site_dir}")

#

Amazon S3 Upload

Now that we have all the files created, let’s upload them to S3. This snippet starts by finding the bucket you already created, and then clearing it out, preparing it for the upload. I decided to go this route because I wanted to keep yesterdays backup in the S3 system, meanwhile I’ll keep a copy of each backup on my remote FTP in the event that I need something older than yesterdays data. After we empty the S3 bucket, we then upload all the files in the current directory.

## S3 Upload ##
#

bucket = AWS::S3::Bucket.find(s3_bucket)
#empty old backup in S3
while(!bucket.empty?)
	begin
		bucket.objects.each do |object|
		object.delete
		end
	end
end
#upload to S3
uploads = Dir["*"]
uploads.each do |uploadfile|
	AWS::S3::S3Object.store(uploadfile, open(uploadfile), s3_bucket)
end
#

Offsite FTP and Cleanup

One last section to go, the Offsite FTP section. We connect to the FTP, authenticate, and then, just like for S3, we upload all the files in the current directory. Then close the FTP connection and your backup is complete! The next command will remove all the SQL and filesystem backup files from the directory. The FTP commands are pretty straightforward so if you need to navigate or setup your FTP in a different manner, it shouldn’t take too much effort. The only remaining task is to schedule this to run as often as you like in your favorite scheduling utility. I end up running the script as root in crontab. Thanks again for reading and check back as hopefully I’ll get the incremental backup code written soon.

## FTP Offsite ##
#

ftp=Net::FTP.new
ftp.connect(url,21)
ftp.login(db_user, db_pass)
ftp.chdir(directory)
ftp.mkdir(timestamp)
ftp.chdir(timestamp)

uploads.each do |f|
	ftp.putbinaryfile(f)
end

ftp.close
##

## Remove backup files
system ('rm -f *.gz')
#

First login to Bitly with the account you will be using and locate your API key in your settings.

Then let’s take a look at the Bitly API definition. Specifically the shorten method. Since I’m using this in a web application many of the variables that the shorten method are going to be hardcoded, as the only variable will be the long url. Here are all the possible options for the shorten method (via Bitly docs).

• format (optional) indicates the requested response format. supported formats: json (default), xml, txt.
• longUrl is a long URL to be shortened (example: http://betaworks.com/).
• domain (optional) refers to a preferred domain; either bit.ly default or j.mp. This affects the output value of url.
• x_login (optional) is the end-user’s login when make requests on behalf of another bitly user. This allows application developers to pass along an end user’s bitly login.
• x_apiKey (optional) is the end-user’s apiKey when making requests on behalf of another bitly user. This allows application developers to pass along an end user’s bitly apiKey.

Whenever possible I use JSON as my return from APIs, choose whichever you are comfortable with, but the rest of this example will using JSON. If your application wants to pass the users login/apikey use the above values, otherwise you will just pass your own, minus the leading ‘x_’.

Let’s dive into the function.

function create_bitly_url($url)
 {
 // Form the Bitly api call URL
 $bitly = 'http://api.bitly.com/v3/shorten?login=ENTER_YOUR_LOGIN_NAME_HERE&apiKey=ENTER_YOUR_KEY_HERE' . '&longUrl=' . urlencode($url) . '&format=json';

This is the beginning of the function, and we build the Bitly API URL. Note that the long url you are passing needs to be URL Encoded. Pass the appropriate format you want to deal with as the return as well.

 // Fetch Bitly URL
 $response = file_get_contents($bitly);

You can use whatever method you prefer to grab urls here as well.

 $json = @json_decode($response,true);</pre>
return $json['data']['url'];
 }

The first line will take the returned code from the last line and turn it into a usable php array. The second line will return the short url from the data returned. Let’s take a closer look at the array returned from the Bitly API.

Array
(
    [status_code] => 200
    [status_txt] => OK
    [data] => Array
        (
            [long_url] => http://gamegossip.com/
            ['url'] => http://bit.ly/lcabpe
            [hash] => lcabpe
            [global_hash] => ksYvP7
            [new_hash] => 0
        )
)
URL: 1

(quotes added around url to get around some WordPress funny business)

The API documentation goes into details on the various possibilities for some of the data. I thought it would be nice to include what the array looks like that is returned to give some context to the line of code that is pulling out the short URL value. This was ran on a URL that was already in the Bitly system, as you can tell from the ‘New Hash’ being set to ‘0’. Depending on what you were doing with your application, you may want to consider testing the results a bit more than we do in this function.

 $short = create_bitly_url('http://gamegossip.com/');

This is the call to create the short URL.

echo 'URL: '. $short;

Display the URL to confirm all is well.

There it is, like I said not much to this one. This will give you a quick and dirty way to add Bitly URL shortening to any project you may be working on. Enjoy!

This code and database structure are now available on Github!

$NTPHosts = Get-VMHost | Select-Object Name,@{Name="NTPServer";Expression={$_ |
Get-VMHostNtpServer}}, @{Name="NTPRunning";Expression={($_ |
Get-VMHostService |
Where-Object {$_.key -eq "ntpd"}).Running}} |
Sort-Object -Property "NTPRunning", "NTPServer"

Using that command will scan through all of the ESX hosts in a Virtual Center and gather up their NTP server list as well as if NTP is running or not on the host.

To export the result out to a file you can run the following command.

$NTPHosts | Export-Csv c:\ntpinfo.csv -NoTypeInformation