Rails 3 WebDAV Tutorial with Custom Resources, Authentication with Devise, and User Specific Routing

The great gem DAV4Rack and its creator Chris Roberts deserve a huge shout-out.

Note: The tutorial is part of a Wiki and is subject to change.

Update: I built a sample app for this and it is available on Github: github.com/bryanrite/dav4rack-example-devise-subdirectories

I will explain how this happens and some steps you can implement to mitigate it.

Most discussion seems to be focused very much on the obvious security difference: that data in the cookie is stored in plain text (well its stored in Base64, but that’s trivially convertible).  Storing sensitive information or stateful data in a session, be it server side or cookie store, is bad practice and for most of us storing simple reference information, like the logged in user’s id, this is not usually much of a concern.

Lets use the example: our cookie stores the logged in user’s ID or nothing if the user isn’t logged in. This is a common scheme used in many of the rails screencasts and guide books.

Instead of storing a session_id which identifies a server side storage record to get the user’s ID, we store the user ID in the cookie. Generally this user ID isn’t sensitive, its probably in the user’s profile URL or within the code somewhere.

If you’re following the above rule and only storing simple reference information, your cookie isn’t much different than a server-sided store.  The HMAC digest within the Rails CookieStore cookie prevents someone from tampering and changing the cookie, so we cannot change our cookie to a different user ID and be logged in as that user, same way it would be difficult to guess a different session_id and access someone else’s session.

Since both implementations rely on a cookie being passed anyways, they have the same security concerns, and are equally susceptible to replay and fixation attacks.

Ok, so what is our concern then?

There is one major difference that never seems to be brought up.  If (and when!) a CookieStore cookie is stolen:

By default, a CookieStore session will never become invalid.

By this, I mean if I steal an authenticated Cookie, I can use it to access the site as that user. This is a common attack called Session Hijacking or Sidejacking but with server-sided storage it can be mitigated.

Generally, with a server side store, you delete the session_id and accompanying data when a user logs out or times out.  Any stolen session_ids are no longer valid because that session_id no longer exists.  If the valid user never logs out and the attacker keeps sending requests, they can keep the session alive, but this can still be detected and stopped.

With the cookie based store, even if the valid user logs out and an expiry date is put on the cookie, an attacker can change the expiry date and replay the cookie at any time.  It will always be valid as there is no session_id to compare against and the cookie expiry is not guarded by the HMAC digest.

The Rails Guides suggest using reset_session to stop hijacking, but this does not help for CookieStore, there is no session identifier to reset!

Really?

Give it a try yourself! On any of your CookieStore based rails app:

1. Load up Firefox and install the Live HTTP Headers or Tamper Data plugin, i’ll use Live HTTP Headers.
2. Log into your app.
3. Start Live HTTP Headers
4. Go to a page that shows you if you are logged in or not.
5. On the Live HTTP Headers modal, select the main request header, its usually the top most one. Pretend you are an attacker and read this off a wireless network.
6. Log out of your web app.
7. Back on the Live HTTP Headers modal, you are now the attacker logging in with stolen cookie: replay the main request header.
8. You will now be logged into your app. No matter what you do, the attacker can save that stolen cookie and replay it any time from anywhere and log back in.

 Uh oh! How can we stop it?

Well, stopping it is quite easy, and I’ll explain a couple of ways how, but the real reason I bring this up is because it isn’t obvious to people using the default session store there is a huge concern here. Everyone talks about how never to store sensitive information in the cookie, but getting an authenticated cookie, by default, gives you life-time pass to that user’s account. That seems much worse! Without a security measure, your application has a huge hole, and none of the rails tutorials, documentation, or screencasts seem to mention this.

No matter what authentication library you use: Devise, Authlogic, or if you roll your own, they are all susceptible because they all use whatever session store you decide.

The best and easiest solution is simply to use SSL.  Not just on your login forms and actions, but your entire site, or at least any pages where you have sessions turned on.  With SSL on, the user will not be able to replay your cookies and the entire attack vector is shut down.  Rails 3.1 has a handy force_ssl switch you can use, and you can use something like:

:secure => Rails.env.production?

in your config.session_store declaration to ensure the cookie is only served over SSL.

If you don’t or can’t use SSL, try implementing a timeout and a nonce within the HMAC protected portion of the cookie.

For example, managing a session timeout yourself by updating the expiry date on every request unless the expiry has passed creates a non-editable timeout on the session and will invalidate it after a specific time. Of course the attacker can still touch your app to keep the timeout alive indefinitely, but it helps.

In addition to a timeout, adding a nonce, even something simple, can help invalidate existing cookies. Storing a hash based on the user’s last login and/or logout time can invalidate stolen cookies every time the valid user logs in/out. This, coupled with the timeout can mitigate hijacking and puts it on par with server-based session management schemes.

Regardless of the session store mechanism you use, they’re all susceptible to attack unless you’re using SSL. Unfortunately, by default, Rails’ CookieStore gives you a no-fuss lifetime pass instead of a day-pass.

I will discuss some realistic ways of preventing brute force attacks, taking a valid user’s experience into account, explain why a lot of tutorials out there on brute force prevention are not effective, and a solution to the problem.

Why Do I Need Brute Force Protection?

Actually, hopefully you don’t.  There are great services out there that you can outsource your authentication to like OpenID, Facebook Connect, and Google Auth (OpenID).  The same way you probably outsource your comment spam protection, these services have more resources tasked to prevent unauthorized access to accounts than you can bring to bare.  In addition, some of these services offer a form of two-factor authentication to their users, usually in the form of an SMS message, making the task of brute forcing an account very difficult if not impossible.

If you are unable or unwilling to integrate an external authentication service, and its pretty common to offer them only as an option anyways, then you will need a brute force solution.  A typical username/password authentication system, especially with public user registration, is very susceptible and most guides you read typically only talk about preventing SQL Injection.

Why Are Most Other Tutorials Wrong?

Most sites and forums suggestions suffer from two major design flaws: they punish valid users (CAPTCHAs, account locking) and they expect the attacker to act like a regular user (Session or IP tracking).

Turing tests are a usability problem.  They can generally have a high defeat rate, sometimes in the range of 90%, and can greatly lower your site’s conversion rate.  They punish real users and don’t stop automated attempts significantly enough.

Account locking (X failed login attempts locks your account for an hour) opens the avenue for a DoS attack on your website. Continuous brute force attempts cause accounts to be locked, locking accounts generally produces a different error message, so when an account gets locked the attacker can start to map valid usernames, further making it easier to lock them out.

Another incorrect assumption is that you can use Cookies, Sessions, or IPs to prevent brute force.  Sessions are based on a cookie or a URL parameter, and if you’re attempting to brute force a login, you aren’t going to allow the target to track your attempts by helping them save your state: Session’s can’t be trusted.  IP tracking is not going to help an attack as you can spoof or use proxies to change your IP.  Also, a large number of users behind a particular proxy or NAT’d router could give you a false-positive: IPs can’t be trusted.

So How Are We Going to Do This?  Punish the Machines.

Brute force works by trying all permutations until it stumbles across the correct one.  Generally this would be done by a machine because the key space of a reasonable password policy is too large to do manually.  We can exploit this man vs. machine to make sure our valid users are kept happy while punishing machines.

The faster password attempts can be made, the faster the password will be found, so lets exploit this by introducing 3 pieces of entropy: speed, scope, and complexity.  We can introduce each of these items in a way that does not punish valid users with the assumption: A valid user will not try their password X times without starting to experience unwanted effects. Several permutations is insignificant for a brute force attempt, but likely a lot for a valid user, and we’ll bring the onset of unwanted effects slowly, to help steer the user to the recover password system.

Speed: by increasing the response time of our authentication request, it takes longer for each attempt, which takes longer to find the correct password.  At a certain number of attempts we can determine that it is likely not a valid user and begin to increase the response time artificially to a point where each attempt takes a maximum amount of time making the entire permutation unreasonable.

Scope: We can greatly increase the size of the problem of a brute force attempt by having the same response, regardless if the username is a valid one or not:  the attacker doesn’t know if its attacking a real account or just wasting time.  Any error messages and response times should be the same.

Complexity: Increasing the complexity of each attempt simply makes it more difficult, causing the failure count to go up, which will increase the time per request.  By introducing a Turing test at a certain time, like after 10 failed attempts, we are likely to avoid the valid user and just punish the machine, changing the form and introducing a third problem to solve.

Putting It All Together

Below is a example pseudo-code-ish implementation using reCAPTCHA as the Turing test.  This should be easily ported to C#, PHP, Java, or Ruby.

protected void Login()
{
    string email = this.textboxEmail.Text.Trim().ToLower();
    string password = this.textboxPassword.Text.Trim();
 
    if (String.IsNullOrEmpty(email) || String.IsNullOrEmpty(password))
    {
        // TODO: Display "form is incomplete" error message.
        return;
    }
 
    User user = UserFactory.ByEmail(email);
    // We have a Database Table or Cache of keys and counts; in this
    // case: email address and failed attempt count. Since we track
    // failures whether the user account exists or not, we store this
    // value separately from the user.
    int failedAttempts = UserFactory.GetFailedLoginCount(email);
 
    // Run our Brute Force Proection, regardless if the user is valid.
    bool captchaRequired = BruteForceProtection(email, failedAttempts);
 
    // Check that the user submitted and passed a captcha. If they
    // didn't redirect them back to the login using a URL parameter
    // to indicate to show the captcha, and increment the counter.
    if ((captchaRequired) && ((!Captcha_Submitted) || (!Captcha_Passed))
    {
        UserFactory.IncrementFailedLoginCount(email);
        // TODO: Display "captcha failed" error message.
        Redirect("/LoginPage/?captcha=1");
    }
 
    // The user doesn't exist or the passwords don't match.
    if ((user == null) || (user.Password != User.EncryptPass(password)))
    {
        UserFactory.IncrementFailedLoginCount(email);
        // TODO: Display "invalid login" error message.
        return;
    }
 
    // Account is authenticated successfully.
 
    // Clear failed attempts count.
    UserFactory.ClearFailedLogins(email);
 
    // Log the user as you normally would:
    Authenticate.Login(user);
 
    Redirect("/Profile/");
}

private bool BruteForceProtection(string email, int failedCount)
{
    // The metrics used are somewhat arbitrary, but seem to work well.
    if (failedCount >= 30)
    {
        // At this point we are pretty positive its a brute force attempt:
 
        // At the 30th failure, we send an email to support about the brute force attempt.
        if (failedCount == 30)
            Email.SendEmail("Brute Force Attempt Detected.",
            String.Format("Attempted brute force of email: {0} from IP: {1}", email, RemoteIP));
        // Sleep between 30 seconds and 1 min per request
        Thread.Sleep(Math.Max(failedCount * 1000, 60000));
        // Enable Captcha
        return true;
    }
    else if (failedCount >= 15)
    {
        // Sleep between ~9 and ~21 seconds
        Thread.Sleep(failedCount * 600);
        // Enable Captcha
        return true;
    }
    else if (failedCount >= 3)
    {
        // Sleep between 1.5 seconds and 7 seconds.
        Thread.Sleep(failedCount * 500);
        // No Captcha
        return false;
    }
 
    return false;
}

As you can see, as the number of failed attempts increase, the time and complexity increase.  At a certain point, we are notified of the brute force, and the entropy increases to the point where each request takes long enough that brute force becomes an unreasonable.  We are also notified, via email in this case, of the offending IPs and at risk usernames, so we have ample time to do something about it.

Problems to Watch For:

• Horizontal Brute Force – We’ve looked at brute force attempts vertically, attacking a username, but an attacker could use a common password and attack the system horizontally, trying different usernames.  You can simply add hashed password to the key table from the above solution, storing a count for each failed password attempt and lock them down that way.
• Password Recovery: Your authentication system might be top notch, but often password recovery functionality is not.  Ensure you don’t give away valid usernames and that your recovery system is well implemented.
• Account Locking: We don’t actually “Lock” accounts, but we do implement a large enough timeout to make it uncomfortable for real users if a DoS attack is attempted – requiring them to wait when logging in.  The email sent above should help mitigate DoS attacks, and its much better than actually locking the account.  You could potentially lower the max second wait time, or decrement the failed login attempts after a specific time frame, so over time it goes back to zero.
• Registration - If an account that doesn’t exist is attempted to be brute forced and that account is later registered by a valid user, the initial login time for our new user could be a long time, so perhaps clear the failed login count on newly registered accounts.

I will explain some useful tips we learned while implementing Internationalization in our Cashless Schools project.

Understanding Globalization and Localization

Internationalization can be roughly broken down into two sections: Globalization and Localization, and understanding the difference between these is important.

When we talk about internationalization, people mainly think about multiple languages.  For example, we want to offer our project in English and Chinese.  What we’re really saying is we want to localize our project with two different languages.  Essentially replacing the text depending on the user’s preference.

Now say that project is selling a product… are we going to change the currency symbols to match the language?  Are all your prices going to be converted to Yuan for your Chinese speaking users?  In some cases, yes; in a lot of cases, no, we still want to display currency and date formats in a single culture.  This is the difference between Localization, language, and Globalization, culture.

Take for instance the Apple Store.  The Apple Store is Globalized for several different cultures.  When you go to the Chinese store, it will be selling its products to people in China, using the Chinese language and culture (zh-CN) is the right thing to do.  Now say they offered a version of its USA store in Mexican-Spanish.  Apple would likely not display currency in Pesos, it is still the USA store, they just want to make it more friendly to native Spanish speakers.  In this case we would use the US culture (en-US) and Mexican-Spanish language (es-MX): the text would be displayed in Spanish, but the currency symbols would continue to be dollars.

ASP.Net allows us to specify these settings separately:

Thread.CurrentThread.CurrentCulture = CultureInfo.CreateSpecificCulture("en-US");
Thread.CurrentThread.CurrentUICulture = CultureInfo.CreateSpecificCulture("es-MX");

What’s so important about the Culture

In my experience, cultural settings are most important in two areas: currency and date format.

Currency can have different currency symbols ($, €, ¥), different thousands separators, different decimal separators.  Dates can be displayed in different formats (m/d/y, d/m/y).

Since these can differ from our localization settings, we need a way to dynamically show them based on the culture (not the language).  ASP.Net has the handy .ToString() method which can take standard identifiers like “C” or “D”.  ASP.Net will automatically output the values based on your current culture, but when you want to customize the output, check out the following libraries:

System.Globalization.CultureInfo.CurrentCulture.NumberFormat
System.Globalization.CultureInfo.CurrentCulture.DateTimeFormat

They will have everything you need, from the current cultures date format strings to the currency symbol.  Below are a couple of Extensions I commonly use to display data.  As you can see they take the format from the culture and the language from the ui culture.

public static string ToLongDateNoWeekDayString(this DateTime date, bool abbrieviate)
{
    string dateFormat = System.Text.RegularExpressions.Regex.Replace(System.Globalization.CultureInfo.CurrentCulture.DateTimeFormat.LongDatePattern, @",?\s*dddd?,?", "");
 
    if (abbrieviate)
        return date.ToString(System.Text.RegularExpressions.Regex.Replace(dateFormat, @"MMMM", "MMM"), System.Globalization.CultureInfo.CurrentUICulture);
    else
        return date.ToString(dateFormat, System.Globalization.CultureInfo.CurrentUICulture);
}
 
public static string ToLongDateTimeNoWeekDayString(this DateTime date, bool removeSeconds)
{
    string pattern = (removeSeconds) ? @"(,?\s*dddd?,?)|(:ss)" : @",?\s*dddd?,?";
    return date.ToString(System.Text.RegularExpressions.Regex.Replace(System.Globalization.CultureInfo.CurrentCulture.DateTimeFormat.FullDateTimePattern, pattern, ""), System.Globalization.CultureInfo.CurrentUICulture);
}
 
public static string ToCulturalCurrencyWithoutSign(this decimal amount)
{
    System.Globalization.NumberFormatInfo ni = (System.Globalization.NumberFormatInfo)System.Globalization.CultureInfo.CurrentCulture.NumberFormat.Clone();
    ni.CurrencySymbol = "";
    return amount.ToString("C", ni).Trim();
}

Regional Dialects

With the culture, we always have to specify a region.  For example, our site’s culture cannot be English (en), it has to be USA English (en-us), Canadian English (en-ca), UK English (en-gb), etc. Because all of these have slightly different settings, there is no “English” culture.

This is not true with language.  We can create generalized resource files as well as specify regional or dialect specific files.  ASP.Net will automatically select the best resource file to use.  For example, we can create resource files for general english and override it with regional specific one.  All we have to do is name the resource files properly:

App_LocalResources
- Example.aspx.en.resx
- Example.aspx.en-CA.resx
- Example.aspx.es.resx
- Example.aspx.es-DO.resx
- Example.aspx.es-MX.resx

Above we have 5 different resource files:

• en : General English.
• en-CA : Canadian English (we have more u’s!)
• es : Spanish
• es-DO : Dominican Republic Spanish
• es-MX : Mexican Spanish

ASP.Net will select the most specific file based on your UI Culture.

Localization Files and Variables

Another important feature of localization is the use of variables in our localization strings.  Languages can be very complex and it is important to remember you cannot just break up a string around a variable.

For a very simple example, “You just added War and Peace to your Shopping Cart.”, is a common type of string you’ll need.  The variable War and Peace can be replaced with any item you are adding to your shopping cart.  You may be tempted to localize the string as:

1. You just added
2. to your Shopping Cart.

And output it around a variable on your page.  Since in different languages, the structure of the sentence can vary greatly, it is highly recommended to use String.Format and numbered variables.  This allows the localized versions of this sentence all the flexibility they need.  You would want to localize the string like:

You just added {0} to your Shopping Cart.

And display it with:

String.Format(GetLocalResourceObject("example").ToString(), item.Name)

The same idea goes for logical paragraphs or basic formatting.  Instead of breaking a paragraph up into multiple separate sentences, always try to keep logically grouped text together as much as possible, and I personally have no problems allowing basic HTML formatting in localized strings.  This allows for smooth translations, as the more the text is broken up, the choppier the translations will be.

Plurals

The use of plurals is an important and difficult syntax to master.  In English, and some other languages, we only worry about 2 cases, maybe 3:

• You have multiple items.
• You have a single item.
• You have zero items.  (usually the same as multiple, but in some cases it can be awkward)

This can be managed by storing the 2-3 different strings in our resource files and extending GetLocalResourceObject to include an identifier for which string to use based on the count.

In many languages, it can be more complex then that.  In Polish for example, the grammar for 1, 2-4, and 0 or 4+ is different, and changes again after 20.  GNU’s gettext has a pretty good solution to this dynamic type of pluralization, but ASP.Net does not, you will have to come up with your own solution if that need arises in your project.

Using the Drobo, there were a couple of gotchas.  It doesn’t have sshd enabled, no bash shell, no rsync (or rsnapshot which would make this easier) and more.

I will describe how I setup an automated, passwordless, incremental differential, hard-linked backup using rsync across ssh.

Whats an incremental differential hard-linked backup?

It is essentially the same as Apple’s Time Machine system.  We create a backup using only the files and directories that have changed, but still store it as a full backup on the backup media.  In other words, every time we backup, we will only transfer what has changed, but if you look at that backup on the remote computer, all the files will be there.

How do we do this?  Utilizing hard-linking in the filesystem.  Essentially we are creating two “pointers” that reference the same file (inode).  That way each backup folder has its own copy of the file, but we only store the file once.  If you run the command stat on a file or directory, you can see in the output how many links to that inode there are, or run ls -i to see the inode you are pointing to.  New versions of that file won’t overwrite the old ones, the new backups will reference a different inode, keeping your historical data.

This helps for several reasons.  When we need to restore the entire filesystem, we can just move all the files from one directory, rather than merging the incrementals into a delta and applying it to the full backup to get our latest system and we only transfer what has changed which dramatically reduces bandwidth, time, and space.

Let’s get started…

First things first, setup your Drobo as the instructions specify and be sure to enable DroboApps.  Create a share on the Drobo for your backups.  I’ll name mine backup, if you use a different name, edit the below scripts as necessary.

You will then need to install the Dropbear SSH and Rsync apps for the following to work.  [Note: You do not need the Apache or Admin apps, but you might find them useful.]

At this point you should be able to login via SSH to your Drobo using the default SSH credentials, at the time of writing for the DroboFS its root and root.  We’ll update the root SSH password using the following command:

# /mnt/DroboFS/Shares/DroboApps/dropbear/root_passwd

This will reset and persist our root password between reboots.

[Note: I know that allowing root SSH access is not ideal but I have so far been unable to stop the drobo from allowing it if SSH is enabled... perhaps a commenter will have a solution?]

Now we need to edit the rsync config file:

# vi /mnt/DroboFS/Shares/DroboApps/rsync/rsyncd.conf

You should see one share named [drobofs]. We want to replace it so that the file looks like:

uid = root
gid = root
pid file = /mnt/DroboFS/Shares/DroboApps/rsync/rsyncd.pid

[drobofs]
        path = /mnt/DroboFS/Shares/backup
        comment = Backup Share
        read only = false

[Note: if you named your backup share different, you'll need to specify it here.]

Alright, now to provide passwordless login.  From the machine that has the data you want backed up, generate your SSH keys.  You will likely not want to use a passphrase and I’ll leave you to secure SSH as you see fit (perhaps read up about forced-commands-only).

user@server:~> ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/user/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/user/.ssh/id_rsa.
Your public key has been saved in /home/user/.ssh/id_rsa.pub.
The key fingerprint is:
f2:13:a7:23:75:da:4e:35:a7:32:61:af:43:e1:a0:53 user@server

Copy what is contained in id_rsa.pub. Now, on the Drobo we have to create the authoized_keys file to enable us to login with the key we just created. The required directory and files do not exist by default so, from the Drobo:

mkdir ~/.ssh
vi ~/.ssh/authorized_keys

Now copy the contents of id_rsa.pub into the authorized_keys file. You should now be able to SSH from the server to the Drobo without needing to enter a password.

Now that our apps are setup on the Drobo and communications will be seamless, we’re going to create two scripts, one to rotate and manage the backups on the Drobo and one to do the actual rsync-ing.

The first script will reside on the Drobo in the root of the backup folder.  In our example:

/mnt/DroboFS/Shares/backup

I’ll call it rotate_backups.sh and it’ll look like the following.  Edit as necessary for your own purposes, and please note, the Drobo doesn’t have the bash shell, so we’re using good old #!/bin/sh

#!/bin/sh
 
# How many backups would you like to keep, each time you run
# the backup script, a new one will be created, so if you want:
# Daily for a week, script goes cron daily and enter 7.
# Hourly for 3 days, script goes cron hourly and enter 72 (24 hours x 3 days)
NUMOFBACKUPS=7
 
# Where are we backing up to?
BACKUPLOC=/mnt/DroboFS/Shares/backup
 
# Delete the oldest backup
NUMOFBACKUPS=`expr $NUMOFBACKUPS - 1`
if [ -d $BACKUPLOC/backup.$NUMOFBACKUPS ]; then
        echo "delete backup.$NUMOFBACKUPS"
        rm -Rf $BACKUPLOC/backup.$NUMOFBACKUPS
fi
 
# Move each snapshot
while [ $NUMOFBACKUPS -gt 0 ]
do
        NUMOFBACKUPS=`expr $NUMOFBACKUPS - 1`
        if [ -d $BACKUPLOC/backup.$NUMOFBACKUPS ] ; then
                NEW=`expr $NUMOFBACKUPS + 1`
                mv $BACKUPLOC/backup.$NUMOFBACKUPS $BACKUPLOC/backup.$NEW
                echo "Move backup.$NUMOFBACKUPS to backup.$NEW"
        fi
done

This script will delete your oldest backup and move the others down the line… so your latest backup will always be backup.0, 1 run old is backup.1, 2 runs old is backup.2, etc.

Now that our backups are managed properly we will create the actual rsync script.  This will be run on the computer you want to back up from, and I suggest you add it to the crontab so it runs as often as you want it to. Edit BDIR, BSERVER, REMOVEDIR, and EXCLUDES as necessary.

#!/bin/sh
 
# Directory to backup.
BDIR=/data_to_backup
 
# Remote server (should match the passwordless SSH credentials
# we setup earlier).
BSERVER=root@drobo
 
# Full path of the directory on the Drobo to backup to.
REMOTEDIR=/mnt/DroboFS/Shares/backup
 
# A list of files or directories to exclude.
EXCLUDES=/home/brite/backup/excludes.txt
 
# Activate our rotate_backups script on the drobo.
ssh $BSERVER $REMOTEDIR/rotate_backups.sh
 
OPTS="--force --ignore-errors --delete-excluded --exclude-from=$EXCLUDES --delete -av --rsync-path=/mnt/DroboFS/Shares/DroboApps/rsync/rsync"
 
# Do the rsync.
rsync $OPTS -e 'ssh -p 22' --link-dest=../backup.1 $BDIR $BSERVER:$REMOTEDIR/backup.0/

Add that script to your hourly/daily/whatever you want crontab and you should be good. The first time you run it you’ll be transferring the entire backup directory so it might take a long time and you’ll have an error message about --link-dest backup.1 not existing, but you can ignore that. Subsequent backups should run without a hitch.

Is This a Job?

No, not really.  What we’re looking for is a person interested in working on projects with us, getting in at the ground floor and collaborating on ideas brought to the table by our collective and external clients.

A lot of us have day jobs, these are projects and tasks we work on in our spare time.  They are passion projects for some, and great money-making ideas for others.  All projects are run on an opt-in basis, meaning you only work on what you want to be a part of.

How Does this Work?

You would complement our team of developers and marketers by designing the graphics and interfaces used on websites, mobile applications, etc, in a collaborative environment.

We have a very informal atmosphere because the people working with you are extremely passionate about what they do.  We chat over beers at the pub, sit down at a whiteboard on the weekend, or skype together.

As part of a self managing collective, you would be expected to live up to your word; we ask for nothing else.

How Much Does it Pay?

A lot of the ideas brought forth by our collective are bootstrapped, meaning the people involved aren’t paid upfront but the revenue of our hard work is typically shared equally between those involved.

Some projects are brought to us by clients, and revenue sharing is openly discussed before projects are taken on.

What Do I Need To Have?

The ability to create beautiful logos, graphics, and interfaces any way you deem necessary.  Design the look and feel of projects through collectively contributed mood boards and ideas.

Design with interactivity, simplicity, beauty, and passion!

Conclusion

It’s an opportunity to work on projects and bring all our diverse skills together and create world-class products and projects.  If you’re interested, contact me at bryan@bryanrite.com

The problem is greater when the content you want to Slug is UTF-8 encoded and contains non-ASCII characters. How do you slug a word like: Iñtërnâtiônàlizætiøn?

My ongoing redo of Footstops, which now creates slug’d URLs from UTF-8 user generated content, has ventured me into such territory and I’ll share my slug method with you. The one caveat is that its power relies on the awesome iconv library, which has come enabled by default since PHP 5.0.0, and easily installable in PHP 4.2+, so make sure you have that, if not, remove the line – it still works, just not nearly as well. I also make the assumption that your data is encoded in UTF-8, which is fairly safe because it is pretty backward compatible, but if you are working in a different charset, please adjust as necessary.

The method is short and sweet.

1. First we use iconv to translit the UTF-8 string into ASCII. This converts the UTF-8 string into an ASCII equivalent, but also translate non-ASCII characters into their ASCII appearing equivalents: ë becomes e.

iconv('UTF-8', 'ASCII//TRANSLIT//IGNORE', $string);

2. We then remove all the unwanted characters from the URL.

preg_replace('/[^a-zA-Z0-9 -]/', '', $url);

3. We convert it to lowercase (which is just a preference for consistency), make sure its between our max string length (we don’t want a 64 character slug, 40-50 characters is probably lots), and remove any surrounding whitespace.

trim(substr(strtolower($url), 0, $maxLength));

4. Finally we replace any whitespace or our separator character with a single instance of the separator character, to remove multiples. I prefer an underscore as a word separator rather than a dash (traditional slug separator) as it may conflict with an actual hyphen in the string but in the final version you’ll see it’s easy to default to your own preference.

preg_replace('/[s' . $separator . ']+/', $separator, $url);

So we put it all together with a couple of options:

public static function ToSlug($string, $maxLength=40, $separator='_') {
	$url = iconv('UTF-8', 'ASCII//TRANSLIT//IGNORE', $string);
	$url = preg_replace('/[^a-zA-Z0-9 -]/', '', $url);
	$url = trim(substr(strtolower($url), 0, $maxLength));
	$url = preg_replace('/[s' . $separator . ']+/', $separator, $url);
	return $url;
}

Calling ToSlug('Iñtërnâtiônàlizætiøn    is the greatest! ') produces the slug:

internationalizaetion_is_the_greatest

perfect for a friendly URL!

Update: Commentors have been reporting this fix is not working on Snow Leopard or MAMP 1.8.4

Installing ImageMagick via MacPorts is simple enough:

sudo port install ImageMagick

but trying to run it via the MAMP stack gives you the apache error:

dyld: Symbol not found: __cg_jpeg_resync_to_restart
  Referenced from: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ImageIO.framework/Versions/A/ImageIO
  Expected in: /Applications/MAMP/Library/lib/libjpeg.62.dylib

or something similar. I was able to get it to go, but you need to break the sandbox, so it may not be a viable solution for everyone.

Once ImageMagick is installed, head to:

/Applications/MAMP/Library/bin/envvars

Here we want to comment out the two lines:

#DYLD_LIBRARY_PATH="/Applications/MAMP/Library/lib:$DYLD_LIBRARY_PATH"
#export DYLD_LIBRARY_PATH

and add the line:

export PATH="$PATH:/opt/local/bin"

What we’re doing is using our systems default DYLD Library path, not the sandboxed one, and exporting our MacPorts bin directory into the sandbox PATH, so it can execute your ImageMagick bins like convert without specifying the full path to it.

Presto! I’m not sure of the full consequences of doing this but I haven’t run into any problems yet and nothing seems to break as a result. Always backup the original files tho!

Now that its been added to the backports repository, its a very simple to upgrade.

If you’re never added anything from backports before, listen up. Backports are all the libs that have feature upgrades (not security related) that won’t show up in normal apt-get upgrade repositories. You could simply add the following line to your sources.list:

deb http://archive.ubuntu.com/ubuntu hardy-backports main universe multiverse restricted

On a default Hardy install it should already be in there, just commented out. So you can uncomment the two backports lines.

This will make all the libs on your machine upgrade to the newest… probably not what you want as you won’t get security updates for the backported versions. Instead we can use something called Pinning, which is available on most apt based operating system.

What we do is add the repository to our sources.list as above but put its priority lower then default, so that the normal repositories take precedent. Then we can on, a per lib basis, upgrade only the libraries we expressly want upgraded from backports.

In the file /etc/apt/preferences (you may have to create it), add the following lines:

Package: *
Pin: release a=hardy-backports
Pin-Priority: 400

We’re setting the hardy-backports repository to have a lower then default priority, so it won’t override the official repositories.

Update your apt cache: sudo apt-get update

Now if you do an apt-get upgrade subversion you shouldn’t see any changes… so how do we install from backports?

sudo apt-get upgrade -t hardy-backports subversion

You can see, its the same line, except we set the target of the upgrade to come from the backports repository.

Now you can keep your servers on the main repositories but still update a library here and there if you really desire new functionality.

/dev/md0:
        Version : 00.90.03
  Creation Time : Sun Nov 16 14:13:20 2007
     Raid Level : raid5
     Array Size : 732587712 (698.65 GiB 750.17 GB)
  Used Dev Size : 244195904 (232.88 GiB 250.06 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Wed Dec 31 10:41:15 2008
          State : clean, degraded
 Active Devices : 3
Working Devices : 3
 Failed Devices : 1
  Spare Devices : 0

...

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       17        1      active sync   /dev/sdb1
       2       8       33        2      active sync   /dev/sdc1
       3       8       49        3      active sync   /dev/sdd1
       4       8        1        -      faulty spare

One of the drives in my fileserver had died! Time to back up and get that sucker running again.

Please note: The following is only a guide to help you replace a failed disc. I cannot guarantee this will work for you, but it is what I do and has worked every time without any data loss.

As you can see, it is a 4 disc software RAID 5 array with no hotswap spares. The following should work for most single disc failure situations in RAID 1, 5, or 6.

It appears that sda has bailed on me. First things first, backup the machine. If anything happens, you can rebuild from scratch.

You can see the faulty disc has already been removed from the array, but if yours hasn’t been removed yet, the commands:

mdadm --manage /dev/md0 -f /dev/sda1
mdadm --manage /dev/md0 -r /dev/sda1

will mark it as failed (so it can be removed) and remove the sda1 partition.

Shutdown the machine and switch out the harddrives, make sure you only replace the faulty drive, don’t mess up the order of the drives cause it’ll be a pain to get it back in.

Boot up the machine. Your RAID array will be in the same degraded state. We need to partition the new drive exactly the same way we partitioned the drives in the existing array. Luckily this is a one-liner with sfdisk:

sfdisk -d /dev/sdb | sfdisk /dev/sda

The above code will dump the partition table of sdb (or use any of the functioning drives) and pipe it to sfdisk to partition sda the same way. It should only take a second.

Then we can simply add the new drive to the array:

mdadm --manage /dev/md0 -a /dev/sda1

Bam! If you take a look at cat /proc/mdstat or mdadm --detail /dev/md0 you should see that the array is recovering (with a percentage done).

After the recovery is done, you’ll be back to new and clean!

Good luck!