Kevin van Zonneveld

Loosely Typed Code Deserves Triple Equality

2013-04-23T10:18:00+02:00

In loosely typed languages such as JavaScript or PHP, using == to compare values is bad practice because it doesn’t account for type, hence false == 0 == '' == null == undefined, etc. And you may accidentally match more than you bargained for.

If you want you can limit unintented effects & bugs this may lead to, it’s often wise to use ===.

In the process of converting legacy codebases to use these triple equality operators, I find that as a rule of thumb you can almost always force triple equality in case of comparing variables against non-numerical strings.

There’s just never a case where you want the text 'Kevin' to pass for the boolean true, or the number 3. And if that can still happen in your legacy codebase, you’ll want to limit those risks rather sooner than later. Even if that breaks things that now accidentally, work.

Switching legacy code that compares against numerical strings, numbers, or booleans is trickier. For instance user input often arrives as a string, so bluntly changing age == 4 to age === 4, will now no longer match the user input age, and introduce more problems than it solves. Addressing these cases needs more thought & care.

To change all of the text-compare cases, here’s a regex that can, to a degree, automate this otherwise much more painful process.

([0-9a-zA-Z\_]\s+[!=])=(\s+['"][^0-9\-\.])
$1==$2

You could use this in a programs like Vim or Araxis, or escape it and then use it in sed. Make sure the changes are under source control and reviewed & tested before committed & pushed.

I tested this on legacy JavaScript and PHP projects. Here you can see the potential issues caught in php.js

Improvements welcome.

Scrape All Text from a Domain

2013-04-19T11:32:00+02:00

Here are some commands to download the most important pages of your site as plain text (determined by MAX_DEPTH), and save it into one big <DOMAIN>.txt file.

This could come in handy when you want to have everything checked for grammar & spelling errors.

After the spellcheck you’d still have to search through your codebase / database to find & fix the culprits, but this should already save you some time in discovery.

(obtain_site_text.sh) download

#!/usr/bin/env bash -e
#
# Downloads a site's text to 1 text file, so you can easily
# have it grammer/spellchecked
#
# Requires: wget, html2text
# Recommened: pandoc vs html2text
# Improve at: http://kvz.io/blog/2013/04/19/obtain-all-text-from-your-website/
#

[ -z "${DOMAIN}" ]    && echo "Cannot continue without DOMAIN. " && exit 1
[ -z "${EXCLUDE}" ]   && EXCLUDE="*.css,*.js,*.rss,*.xml,*.png,*.jpg,*.jpeg,*.gif,*.flv,*.swf,*.mp4,*.mov,*.mp3,*.wav"
[ -z "${MAX_DEPTH}" ] && MAX_DEPTH="1"
[ -z "${OUTPUT}" ]    && OUTPUT="./${DOMAIN}.txt"
[ -z "${TMPDIR}" ]    && TMPDIR="/tmp"
[ -z "${TXTENGINE}" ] && [ -x "$(which pandoc)" ]    && TXTENGINE="pandoc +RTS -K16m -RTS -t markdown -o- -f html -i"
[ -z "${TXTENGINE}" ] && [ -x "$(which html2text)" ] && TXTENGINE="html2text -nobs"
[ -z "${TXTENGINE}" ] && echo "Cannot continue without pandoc or html2text. " && exit 1

wget \
  --adjust-extension \
  --convert-links \
  --directory-prefix="${TMPDIR}" \
  --level="${MAX_DEPTH}" \
  --no-parent \
  --recursive \
  --reject="${EXCLUDE}" \
  --restrict-file-names=windows,lowercase \
"http://${DOMAIN}"

[ -f "${OUTPUT}" ] && rm -f "${OUTPUT}"
find "${TMPDIR}/${DOMAIN}" -type f -print0 -name '*.html' \
  |while read -d $'\0' file; do
  echo "imported by ${0} from: $(echo "${file}" |sed "s@^${TMPDIR}/${DOMAIN}@@")" >> "${OUTPUT}"
  echo "==================================" >> "${OUTPUT}"
  ${TXTENGINE} "${file}" >> "${OUTPUT}"
  echo -e "\n\n\n\n" >> "${OUTPUT}"
done

echo ""
echo " Combined text file ready in ${OUTPUT}"
echo " To cleanup after this script, type: rm -rf \"${TMPDIR}/${DOMAIN}\""
echo ""

Required:

brew install wget html2text # or apt-get install wget html2text

Run it:

DOMAIN=kvz.io ./obtain_site_text.sh

Recommended:

If you can install Pandoc, the resulting text output will be in Markdown and of much higher quality.

Improvements are more than welcome!

Migrate Redis Keys Without RDB Files

2013-04-16T15:28:00+02:00

Recently we moved the Transloadit status page from an unmanaged EC2 instance to the Nodejitsu platform. We kept status uptime history in redis, and obviously I wanted to preserve that data.

For the new setup I did not have access to the filesystem, I only had a redis port to talk to. So instead of rsyncing the .rdb file I used rync replication to migrate the data between instances.

Here’s the redacted terminal transcript, the sending host is old-redis.transloadit.com, the receiving host is new-redis.transloadit.com.

# First let's connect to the new host
$ redis-cli -h new-redis.transloadit.com
redis new-redis.transloadit.com:6379> AUTH ...:...
OK

# Let's start with a clean slate, first make sure it stands alone
redis new-redis.transloadit.com:6379> SLAVEOF NO ONE
OK

# WARNING! This will trash all existing data on the receiving host
redis new-redis.transloadit.com:6379> FLUSHALL
OK

# Now setup replication so that redis copies all keys from old to new
redis new-redis.transloadit.com:6379> SLAVEOF old-redis.transloadit.com 6379
OK

# Wait until INFO shows `role:slave`, and the
# same amount of keys as on the old server.
redis new-redis.transloadit.com:6379> INFO
...
role:slave
...
OK

# Check some random keys to see if you have all the data.
redis new-redis.transloadit.com:6379> LRANGE history 0 1
 1) "{\"message\":\"All services operational\",\"date\":\"2013-04-15\"}"
OK

# Looks good, now let's disable slave mode, make it stand alone again
redis new-redis.transloadit.com:6379> SLAVEOF NO ONE
OK

And that’s it, you’ve copied all the redis keys without using .rdb files.

Hope this helps, let me know if you have suggestions.

Let's Make DNS Outage Suck Less

2013-03-27T14:56:00+01:00

This article was featured on the Hacker News frontpage. Feel free to checkout comments there.

Unfortunately the Linux DNS resolver has no direct support for detecting and doing failovers for DNS servers. It keeps feeding requests to your primary resolving nameserver, waits for a configured timeout, attempts again, and only then tries the second nameserver.

This typically means nearly 30s delay for all request as long as your primary nameserver is unreachable. It doesn’t learn to directly target your secondary nameserver so long as there is trouble.

Even with the most optimal configuration, the delay will still be measured in seconds per request. For many requests, that’s many more seconds.

I wanted to solve this.

Mainly because over at Transloadit our Amazon EC2 resolving nameserver (172.16.0.23) is unreachable too often. When it happens it causes big delays and queues in some processes and even downtime as we rely on domain->ip translation. For instance customer could tell us to download 1000 images from different urls, watermark them, upload them to an sFTP server.

I wanted solid failover to Google / Level3’s nameservers in case Amazon’s went down again. And then failback as soon as possible, because Amazon can resolve server33.transloadit.com hostnames to LAN IPs where applicable, resulting in lower latency for instance-to-instance communication, when encoding machines need to work together.

But whatever the usecase, there’s a need for better way to failover.

Ideally one that does not involve more local proxy daemons, external services, keepalived VRRP IPs, etc. as that would just introduce more complexity and Single Point Of Failures. It should be transparent, archaic and at most rely on crontab.

So I wrote nsfailover in bash, and have it replace the resolve-configuration when needed. It’s rugged, easy to debug, hard to break, and has been working really well for us so far.

Running it is pretty simple, too. Configuration such as NS_1 is done via environment variables, here’s an example where I set it globally using export:

ubuntu@abril:/srv/current$ export NS_1=172.16.0.23
ubuntu@abril:/srv/current$ sudo nsfailover.sh
2013-03-27 14:18:22 UTC [     info] Best nameserver is primary (172.16.0.23)
2013-03-27 14:18:22 UTC [     info] No need to change /etc/resolv.conf
ubuntu@abril:/srv/current$

Or maybe you want also want to define the backup nameserver (defaults to Google’s), and just pass the config to this process:

ubuntu@abril:/srv/current$ NS_1=8.8.8.8 NS_2=8.8.4.4 sudo -E nsfailover.sh
2013-03-27 15:01:53 UTC [     info] Best nameserver is primary (8.8.8.8)

 # Written by /srv/current/stack/bin/nsfailover.sh @ 20130327150153
 nameserver 8.8.8.8
 options timeout:3 attempts:1
 search compute-1.internal

2013-03-27 15:01:53 UTC [emergency] I changed /etc/resolv.conf to primary (8.8.8.8)
ubuntu@abril:/srv/current$

Tight!

Now if you save this in crontab with a timeout:

crontab -e
# By default, NS_2 is Google, NS_3 is Level3, so only your NS_1 is required:
* * * * * timeout -s9 50s NS_1=172.16.0.23 nsfailover.sh 2>&1 |logger -t cron-nsfailover

it turns out, Bob indeed is your uncle : )

The logger pipe will make all output go to syslog so to get notified when a failover happens, just scan for [emergency] in the cron-nsfailover tag. In our case, Papertrail receives our syslog and I made it report to Campfire when this happens.

There’s more documentation available on Github. Let me know if you have any improvements or send me a pull request : )

Find duplicate input with MySQL

2013-03-04T11:32:00+01:00

Back to basic, let’s brush up on some SQL : )

At my company we have employees creating customer accounts every day. Sometimes we make mistakes, for instance, we forget to check if the company already was a customer (maybe 10y ago they may have had a product).

Duplicate accounts can cause all sorts of problems, so I wanted way to detect them with SQL.

The problem was, the company names may have been entered with different punctuation, whitespace, etc. So I needed similar names to surface from the depths of our database, not just exact matches (that would have been too easy : )

For the solution I turned to SOUNDEX for fuzzy matching similar sounding company names, and then review the results myself (false positives are possible, but since they would be few, it becomes a simple task to doublecheck) and report back to our company.

I thought I’d share

partly because it could be useful to others (obviously this could be used to detect all kinds of user generated typos and similar entries);
mostly because I’m curious to find if there is a better (more performant) way to write this query.

Do you know how? Leave a comment : )

-- Select all the individual company names that have a
-- soundex_code that occurs more than once (I now use a subquery for that)
SELECT
  `id`,
  `customer_name`,
  SOUNDEX(`customer_name`) AS soundex_code
FROM `customers`
WHERE SOUNDEX(`customer_name`) IN (
  -- Subquery: select all soundex_codes that occur more than once,
  -- (this does not return the individual company names that share them)
  SELECT SOUNDEX(`customer_name`) AS soundex_code
  FROM `customers`
  WHERE 1 = 1
    AND `is_active` = 1
    -- More specific criteria to define who you want to compare
  GROUP BY soundex_code
  HAVING COUNT(*) > 1
)
ORDER BY
  soundex_code,
  `customer_name`

This e.g. returns:

`id` `customer_name`  `soundex_code`
F.S. Hosting     F2352
FS hosting       F2352
R  Schmit        R253
R Schmit         R253

By the way: The SQL is formatted according to my old SQL Formatting blogpost. Exactly 4 years after I published it, I still find it a pretty useful code convention.

Introducing BASH3 Boilerplate

2013-02-26T00:19:00+01:00

There are many cases where using BASH to code ends up overcomplicating things, but due to it’s portability there are many cases that make it the best tool for the job.

When I do hack on BASH scripts, I often find there are things like logging, configuration, commandline argument parsing that:

I need everytime
Take quite some effort to get right, and
Keep you from your actual work.

Here’s an attempt at bundling those things in a generalized way so that they are reusable as-is in most of my (and hopefully your) programs.

Goals

I’m going to be pragmatic and be violating some good practices like DRY and the things I know about sourcing, submodules, package management, etc.

I feel that when a program consists of multiple files, BASH may very well not be the right tool for the job.

I propose people copy-paste main.sh, then delete the parts they don’t need, and write the rest of their script below ### Runtime.

Copy-pasting and removing what you don’t need may not seem very rocket-scientific or 2013, but in fact, neither is BASH.

Keeping things simple, small and lightweight should be key here.

Aiming for portability, I’m targetting BASH 3 (OSX still ships with 3 for instance). If you’re going to ask people to install BASH 4 first, you might as well pick a more advanced language as a dependency.

The main template should only have features that are needed 80% of the time you write BASH.

Other cool functions can be adopted by the repository, but should not be referenced by the main.sh template and function purely as a resource.

Features

Structure
Configuration by environment variables
Configuration by commandline arguments (definitions parsed from help info, so no duplication needed)
Magic variables like __FILE__ and __DIR__
Logging that supports colors and is compatible with Syslog Severity levels

Conventions / Recommendations

It’s the task of the script’s caller to: Redirect the output to appropriate locations
It’s the task of the script’s caller to: Correctly set the PATH variable (think cronjobs)
Use spaces vs tabs so you can copy-paste parts directly into the console, without BASH’s automcomplete firing on every tab character
Use set -eu, so it will break on every error and undefined variable. If you’re expecting an error, add ||true to your command. set -eu is better that relying on #!/bin/bash -eu, that will be ignored when sourced or executed like bash main.sh.
Checkout The 10 Commandments of Logging
Adhere Syslog. Sure, skip notice, alert, critical, if you need just 3 levels. You should still crash in emergency, debug in debug, inform in info. The world has had plenty smart folks creating new cool logstandards, dealing with them does not make life easier though.

Code

(main.sh) download

#!/bin/bash
#
# Template to write better bash scripts. More info: http://kvz.io
# Version 0.0.1
#
# Usage:
#  LOG_LEVEL=7 ./template.sh first_arg second_arg
#
# Licensed under MIT
# Copyright (c) 2013 Kevin van Zonneveld
# http://twitter.com/kvz
#

### Configuration
#####################################################################

# Environment variables
[ -z "${LOG_LEVEL}" ] && LOG_LEVEL="6" # 7 = debug -> 0 = emergency

# Commandline options. This defines the usage page, and is used to parse cli opts & defaults from.
# the parsing is unforgiving so be precise in your syntax:
read -r -d '' usage <<-'EOF'
  -f   [arg] Filename to process.
  -t   [arg] Location of tempfile. Default="/tmp/x"
  -d         Enables debug mode
  -h         This page
EOF

# Set magic variables for current FILE & DIR
__FILE__="$(test -L "$0" && readlink "$0" || echo "$0")"
__DIR__="$(cd "$(dirname "${__FILE__}")"; echo $(pwd);)"


### Functions
#####################################################################

function _fmt ()      {
  color_ok="\x1b[32m"
  color_bad="\x1b[31m"

  color="${color_bad}"
  if [ "${1}" = "debug" ] || [ "${1}" = "info" ] || [ "${1}" = "notice" ]; then
    color="${color_ok}"
  fi

  color_reset="\x1b[0m"
  if [ "${TERM}" != "xterm" ] || [ -t 1 ]; then
    # Don't use colors on pipes or non-recognized terminals
    color=""
    color_reset=""
  fi
  echo -e "$(date -u +"%Y-%m-%d %H:%M:%S UTC") ${color}$(printf "[%9s]" ${1})${color_reset}";
}
function emergency () { echo "$(_fmt emergency) ${@}" || true; exit 1; }
function alert ()     { [ "${LOG_LEVEL}" -ge 1 ] && echo "$(_fmt alert) ${@}" || true; }
function critical ()  { [ "${LOG_LEVEL}" -ge 2 ] && echo "$(_fmt critical) ${@}" || true; }
function error ()     { [ "${LOG_LEVEL}" -ge 3 ] && echo "$(_fmt error) ${@}" || true; }
function warning ()   { [ "${LOG_LEVEL}" -ge 4 ] && echo "$(_fmt warning) ${@}" || true; }
function notice ()    { [ "${LOG_LEVEL}" -ge 5 ] && echo "$(_fmt notice) ${@}" || true; }
function info ()      { [ "${LOG_LEVEL}" -ge 6 ] && echo "$(_fmt info) ${@}" || true; }
function debug ()     { [ "${LOG_LEVEL}" -ge 7 ] && echo "$(_fmt debug) ${@}" || true; }

function help () {
  echo ""
  echo " ${@}"
  echo ""
  echo "  ${usage}"
  echo ""
  exit 1
}

function cleanup_before_exit () {
  info "Cleaning up. Done"
}
trap cleanup_before_exit EXIT


### Parse commandline options
#####################################################################

# Translate usage string -> getopts arguments, and set $arg_<flag> defaults
while read line; do
  opt="$(echo "${line}" |awk '{print $1}' |sed -e 's#^-##')"
  if ! echo "${line}" |egrep '\[.*\]' >/dev/null 2>&1; then
    init="0" # it's a flag. init with 0
  else
    opt="${opt}:" # add : if opt has arg
    init=""  # it has an arg. init with ""
  fi
  opts="${opts}${opt}"

  varname="arg_${opt:0:1}"
  if ! echo "${line}" |egrep '\. Default=' >/dev/null 2>&1; then
    eval "${varname}=\"${init}\""
  else
    match="$(echo "${line}" |sed 's#^.*Default=\(\)#\1#g')"
    eval "${varname}=\"${match}\""
  fi
done <<< "${usage}"

# Reset in case getopts has been used previously in the shell.
OPTIND=1

# Overwrite $arg_<flag> defaults with the actual CLI options
while getopts "${opts}" opt; do
  line="$(echo "${usage}" |grep "\-${opt}")"


  [ "${opt}" = "?" ] && help "Invalid use of script: ${@} "
  varname="arg_${opt:0:1}"
  default="${!varname}"

  value="${OPTARG}"
  if [ -z "${OPTARG}" ] && [ "${default}" = "0" ]; then
    value="1"
  fi

  eval "${varname}=\"${value}\""
  debug "cli arg ${varname} = ($default) -> ${!varname}"
done

shift $((OPTIND-1))

[ "$1" = "--" ] && shift


### Switches
#####################################################################

# debug mode
if [ "${arg_d}" = "1" ]; then
  set -x
  LOG_LEVEL="7"
fi

# help mode
if [ "${arg_h}" = "1" ]; then
  help "Help using ${0}"
fi


### Validation
#####################################################################

[ -z "${arg_f}" ]     && help   "Setting a filename with -f is required"
[ -z "${LOG_LEVEL}" ] && emergency "Cannot continue without loglevel. "


### Runtime
#####################################################################

# Exit on error. Append ||true if you expect an error.
# set -e is safer than #!/bin/bash -e because that is nutralised if
# someone runs your script like `bash yourscript.sh`
set -e

# Bash will remember & return the highest exitcode in a chain of pipes.
# This way you can catch the error in case mysqldump fails in `mysqldump |gzip`
set -o pipefail

debug "Info useful to developers for debugging the application, not useful during operations."
info "Normal operational messages - may be harvested for reporting, measuring throughput, etc. - no action required."
notice "Events that are unusual but not error conditions - might be summarized in an email to developers or admins to spot potential problems - no immediate action required."
warning "Warning messages, not an error, but indication that an error will occur if action is not taken, e.g. file system 85% full - each item must be resolved within a given time. This is a debug message"
error "Non-urgent failures, these should be relayed to developers or admins; each item must be resolved within a given time."
critical "Should be corrected immediately, but indicates failure in a primary system, an example is a loss of a backup ISP connection."
alert "Should be corrected immediately, therefore notify staff who can fix the problem. An example would be the loss of a primary ISP connection."
emergency "A \"panic\" condition usually affecting multiple apps/servers/sites. At this level it would usually notify all tech staff on call."

Contribute:

OSX Productivity: Dropbox your Screenshots

2013-02-18T16:02:00+01:00

I often share screens with co-workers by Campfire, Github, or mail. Visualizing something can save you a lot of typing. Show people what button shade doesn’t look quite right, instead of explaining in 1000+ characters. Share a load graph without saving & attaching images, or handing out basic auth credentials. The list goes on & on. Once you make it a joy to share, you’ll find use-cases on a daily basis, and it is my believe you’ll lose less time on typing and miscommunication.

At least.. that’s what I’m experiencing, using the following tricks.

The Normal flow

Press SHIFT + COMMAND + 4
(optionally you spress SPACE to switch to Window-capturing)
Drag an area that you want to share with co-workers
A screenshot is saved to ~/Desktop

And now you’ll have to find your way to the screenshot. That seems like effort.

The Better flow

But having many windows open, now it’s a pain to access the screenshot.

One thing you could do is make the Desktop folder more accessible by adding it to your Dock. You can make it even better by setting it up as a Fan, ordered by Date Modfied.

Now, your latest screenshot will always be the bottom-first item from your new Dock fan. Now you can drag & drop it right into your open Github/Campfire app like a pro, without ever minimizing the apps your already working in, or firing up Finder windows to hunt down your screen.

The Ultimate flow

To add even more sugar to this, you could tell OSX to save screenshots into Dropbox by default. This way you can use screens as quick notes, distributed across all of your machines; or you could tell Dropbox to quickly make a public link of your screen, and share a URL with anyone.

Setup

Here’s how to set up the Ultimate flow, assuming you have Dropbox. Without Dropbox, it’s still cool, just remove /Dropbox from the following commands.

mkdir -p ~/Dropbox/Screenshots/
defaults write com.apple.screencapture location ~/Dropbox/Screenshots/
killall SystemUIServer

Now open your Dropbox in the GUI

open ~/Dropbox/

Drag the Screenshots next to your Downloads & Trash icons in your Dock
Right click: Fan
Right click: Date Modified

Tight! Now make some screenshots as explained in the Normal flow and see them automatically added to the fan, most recent always the first clickable item, and Dropbox syncing them up instantly.

Undo

Don’t like it? Wow you are one tough customer. But here we go : )

defaults write com.apple.screencapture location ~/Desktop/
killall SystemUIServer

And just remove the folder from your Dock. That’s all : )

Alternatives

Now the article already mentions Dropbox is very optional, but of course you can also substitute Dropbox with (free) competitors such as Sparkleshare if you’re into that.

It was also mentioned by Graham Weldon that you can use CloudApp for a similar workflow. It’s free & available in the Mac App Store.

Too many authentication failures for root

2013-02-12T13:13:00+01:00

I recently had an annoying encounter with the error message: Too many authentication failures for root. I found out this can be caused because you’ve hoarded too many SSH keys : )

So serves me right, but let’s see what happens exactly.

Consider at this paste:

» ssh -vvv root@67.23.163.74 2>&1 |egrep '(public|fail)'
debug3: Could not load "/Users/kevin/.ssh/id_rsa" as a RSA1 public key
debug3: Could not load "/Users/kevin/.ssh/id_dsa" as a RSA1 public key
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering DSA public key: /Users/kevin/.ssh/id_dsa
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password
debug1: Offering RSA public key: /Users/kevin/.ssh/kevin_201011_true
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password
debug1: Offering RSA public key: /Users/kevin/.ssh/transkey.pem
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password
debug1: Offering RSA public key: /Users/kevin/.ssh/kevin_201211_true
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password
debug1: Offering RSA public key: /Users/kevin/.ssh/kevin_201205_true
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password
debug1: Offering RSA public key: /Users/kevin/.ssh/id_rsa
debug2: we sent a publickey packet, wait for reply
Received disconnect from 67.23.163.74: 2: Too many authentication failures for root

It tries 6 different SSH keys it found laying around on my system before asking for a password. I knew SSH automagiclly looks for the best way to authenticate you. No problem there.

But the serverside counts each try as a failure! Apparently SSHD was configured to only allow so many (6) failures for a user (root), hence SSH never even bothered asked for the password that should be working for this box.

This server was freshly delivered by a colo provider, so I really could not get in any other way. To fix, I could have cleaned up my keys, but instead I disabled login by Public key authentication:

ssh -o PubkeyAuthentication=no root@67.23.163.74

And this fixed the issue.

Afterwards I could set up proper public keys and disable password login altogether.

Tricky though, so I thought I’d share : )

Keep Mounted Network Drives Alive on OSX

2013-02-11T23:12:00+01:00

I love my NAS but because I tried to save a little money it does not run SABnzbd very well.

I’ve tried different approaches but find myself ending up downloading on OSX as it writes to a network share on my NAS. Too bad, but I’m archiving this one under the section first world problems.

The challenge I have now though, is when my Mac goes to sleep, my mounts disappear, and SABnzbd writes to the local filesystem instead. Cause as far as my downloading program could tell, it was already writing to a local filesystem, so it will just keep on doing that until my Mac’s disk is at 100%.

I wrote a little script to prevent that.

You may not be running SABnzbd, but there are obviously many other use cases where you want a network mount to persist. Especially if you are automating something outside of the GUI.

With some small adjustments this could work for Linux/NFS/SMB as well.

Here we go

(remounter.sh) download

#!/usr/bin/env bash -e
# Monitors mounts, checks if they're writable, remounts if necessary.
# For OSX / AFP.
# If it had to remount, exits with code 1 so you can easily chain
# other scripts in such an event. e.g.:
#
#  NASPASS=******** ./remounter.sh || ./restart_downloader.sh restart
#

function _log () {
  local level="${1}"
  local str="${2}"
  echo "[$(date "+%Y-%m-%d %H:%M:%S")] ${level}: ${str}"
}
function info () {
  _log "INFO" "${1}"
}
function err  () {
  _log " ERR"  "${1}"
}
function crit () {
  _log "CRIT" "${1}";

  if [ "${REBOOTONCRIT}" = 1 ]; then
    echo "Warning, CRIT happened and reboot on crit was specified. "
    echo "Rebooting in 50 seconds"
    shutdown -r +50
  fi

  exit 1
}

[ -n "${NASSHARES}" ]    || NASSHARES="downloads video"
[ -n "${NASHOST}" ]      || NASHOST="nas.local"
[ -n "${NASUSER}" ]      || NASUSER="${USER}"
[ -n "${STAMPFILE}" ]    || STAMPFILE=".remounter.stamp"
[ -n "${NASPASS}" ]      || crit "Please set the shares password like so: NASPASS=******** ${0}"
[ -n "${FORCEREMOUNT}" ] || FORCEREMOUNT=0
[ -n "${REBOOTONCRIT}" ] || REBOOTONCRIT=0

function mount_exists () {
  local share="${1}"
  local dir="/Volumes/${share}"

  echo "$(mount |egrep -i "^//${NASUSER}@${NASHOST}/${share} on ${dir} " |wc -l)"
}

function mount_writable () {
  local share="${1}"
  local dir="/Volumes/${share}"

  local stamp="$(date "+%Y-%m-%d %H:%M:%S")"
  local stampfile="${dir}/${STAMPFILE}"

  echo "${stamp}" > "${stampfile}" || true

  local found="$(cat "${stampfile}")"

  [ -f "${stampfile}" ] && rm "${stampfile}"

  if [ "${found}" != "${stamp}" ]; then
    echo "0"
  else
    echo "1"
  fi
}

function remount () {
  local share="${1}"
  local dir="/Volumes/${share}"
  local stampfile="${dir}/${STAMPFILE}"

  info "remounting ${share}... "

  if [ -d "${dir}" ]; then
    # Try 3 times because this happened once:
    # //kevin@nas.local/downloads on /Volumes/downloads (afpfs, nodev, nosuid, mounted by kevin)
    # //kevin@nas.local/video on /Volumes/video (afpfs, nodev, nosuid, mounted by kevin)
    # //kevin@nas.local/downloads on /Volumes/downloads (afpfs, nodev, nosuid, mounted by kevin)
    # //kevin@nas.local/video on /Volumes/video (afpfs, nodev, nosuid, mounted by kevin)
    for i in `seq 1 3`; do
      umount -f "${dir}" || true
      sleep 1

      [ "$(mount_exists ${share})" -eq 0 ] && [ "$(mount_writable ${share})" -eq 0 ] && break
    done

    [ "$(mount_writable ${share})" -ne 0 ] && crit "Could not unmount ${share}. Still writable"
    [ "$(mount_exists ${share})" -ne 0 ] && crit "Could not unmount ${share}. Still exists"
  fi

  mkdir -p "${dir}"
  mount_afp -i "afp://${NASUSER}:${NASPASS}@${NASHOST}/${share}" "${dir}"
}

for share in ${NASSHARES}; do
  if [ "${FORCEREMOUNT}" = 1 ] || [ "$(mount_exists ${share})" -ne 1 ] || [ "$(mount_writable ${share})" -ne 1 ]; then
    [ -z "${missing}" ] || missing="${missing} "
    missing="${missing}${share}"
    remount "${share}"
  fi
done

if [ -z "${missing}" ]; then
  info "Mounts in tact. All good"
  exit 0
fi

info "Had to remount. "
exit 1

I set up a cronjob so it runs every minute like so:

crontab -e
* * * * * /location/remounter.sh || /location/restart_downloader.sh

To illustrate, here’s remounter.sh in action. This happened while I was away, apparently my Mac went to sleep (otherwise you’d see a stamp every minute), and that totally broke the mounts afterwards, multiple times in different ways.

[2012-12-25 22:48:02] INFO: Mounts in tact. All good
[2012-12-25 22:49:01] INFO: Mounts in tact. All good
[2012-12-26 06:38:00] INFO: remounting downloads...
[2012-12-26 07:32:00] INFO: remounting downloads...
umount: /Volumes/downloads: not currently mounted
/blogscripts/remounter.sh: line 55: /Volumes/downloads/.remounter.stamp: Permission denied
cat: /Volumes/downloads/.remounter.stamp: No such file or directory
/blogscripts/remounter.sh: line 55: /Volumes/downloads/.remounter.stamp: Permission denied
cat: /Volumes/downloads/.remounter.stamp: No such file or directory
[2012-12-26 16:33:00] INFO: remounting downloads...
umount: /Volumes/downloads: not currently mounted
/blogscripts/remounter.sh: line 55: /Volumes/downloads/.remounter.stamp: Permission denied
cat: /Volumes/downloads/.remounter.stamp: No such file or directory
/blogscripts/remounter.sh: line 55: /Volumes/downloads/.remounter.stamp: Permission denied
cat: /Volumes/downloads/.remounter.stamp: No such file or directory
[2012-12-26 17:26:45] INFO: remounting video...
[2012-12-26 17:26:48] INFO: Had to remount.
[2012-12-26 17:27:04] INFO: Mounts in tact. All good
[2012-12-26 18:21:04] INFO: Mounts in tact. All good
mount_afp: AFPMountURL returned error -1069, errno is -1069
mount_afp: AFPMountURL returned error -5023, errno is -5023
[2012-12-27 01:45:02] INFO: Mounts in tact. All good
[2012-12-27 01:46:04] INFO: Mounts in tact. All good
[2012-12-27 01:47:06] INFO: Mounts in tact. All good

Would be a great idea to /blog/2012/12/31/lock-your-cronjobs it.

And that’s that : )

Vagrant Tip: Sync VirtualBox Guest Additions

2013-01-16T13:43:00+01:00

Quick tip. If you lose your Vagrant mounts after kernel upgrades in your virtualbox, you’ll need to reinstall your VirtualBox Guest Additions. Same is true when you upgrade Vagrant, etc.

It’s just a real pain and people usually avoid it by never upgrading. Or delve in once they accidentally do. But there’s actually a nice & automated way of keeping your VM’s guest additions in sync with virtualbox.

I was fed up with missing mountpoints and messages like these:

$ vagrant up
[default] Importing base box 'ubuntu-12.04-64bit'...
[default] The guest additions on this VM do not match the install version of
VirtualBox! This may cause things such as forwarded ports, shared
folders, and more to not work properly. If any of those things fail on
this machine, please update the guest additions and repackage the
box.

Guest Additions Version: 4.1.22
VirtualBox Version: 4.2.6

So, Vbguest plugin to the rescue!

To install, you type this one time in the directory that also keeps your Vagrantfile:

# For vagrant < 1.1.5:
# vagrant gem install vagrant-vbguest

# For vagrant 1.1.5+ (thanks Lars Haugseth):
vagrant plugin install vagrant-vbguest

On succesful execution, the output should look something like this:

Fetching: micromachine-1.0.4.gem (100%)
Fetching: vagrant-vbguest-0.6.4.gem (100%)
Successfully installed micromachine-1.0.4
Successfully installed vagrant-vbguest-0.6.4
2 gems installed
Installing ri documentation for micromachine-1.0.4...
Installing ri documentation for vagrant-vbguest-0.6.4...
Installing RDoc documentation for micromachine-1.0.4...
Installing RDoc documentation for vagrant-vbguest-0.6.4...

And that’s it. From now on every vagrant up will check & install the correct guest additions right after booting:

[default] Waiting for VM to boot. This can take a few minutes.
[default] VM booted and ready for use!
[default] GuestAdditions 4.2.6 running --- OK.

And if it’s outdated, here’s what happens:

$ vagrant up
[default] Booting VM...
[default] Waiting for VM to boot. This can take a few minutes.
[default] VM booted and ready for use!
[default] GuestAdditions versions on your host (4.2.10) and guest (4.2.6) do not match.
stdin: is not a tty
Reading package lists...
Building dependency tree...
Reading state information...
linux-headers-3.2.0-23-generic is already the newest version.
dkms is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 126 not upgraded.
[default] Copy iso file /Applications/VirtualBox.app/Contents/MacOS/VBoxGuestAdditions.iso into the box /tmp/VBoxGuestAdditions.iso
stdin: is not a tty
mount: warning: /mnt seems to be mounted read-only.
[default] Installing Virtualbox Guest Additions 4.2.10 - guest version is 4.2.6
stdin: is not a tty
Verifying archive integrity... All good.
Uncompressing VirtualBox 4.2.10 Guest Additions for Linux..........
VirtualBox Guest Additions installer
Removing installed version 4.2.6 of VirtualBox Guest Additions...
Removing existing VirtualBox DKMS kernel modules ...done.
Removing existing VirtualBox non-DKMS kernel modules ...done.
Building the VirtualBox Guest Additions kernel modules ...done.
Doing non-kernel setup of the Guest Additions ...done.
You should restart your guest to make sure the new modules are actually used

Tight! : )

Lock your Cronjobs, Enjoy your Sleep

2012-12-31T11:15:00+01:00

If you use EC2 you may have heard of Tim Kay’s aws commandline tool. It provides access to most of Amazon’s API and is less cumbersome than Amazon’s own CLI utilities in day to day use.

A lesser known tool by Tim Kay is solo. It’s basically one line of Perl, but it’s incredibly useful to defeat a common problem with cronjobs: overlap.

The Problem

You’ve probably dealt with this before; you write a pretty neat yourscript.sh, schedule it to run every or so minute on your production server. One night, your server reaches a load of 90 and you get pagerdutied to fix this.

You login, which takes about 15 minutes, succeed executng ps auxf, and it appears your server now has 8325 instances of yourscript.sh running. What happened?!

Maybe there was an infinite loop in your script, maybe there were NFS timeouts, you tried to update a database that had write-locks during backup, but whatever the cause; there was overlap 8324 times, and this should never happen. Not even once.

The Solution

One way to defeat it, is to write perfect code and have 0 external dependecies that can increase your script’s execution time.

But since that is never going to happen ; ) I recommend taking a look at solo.

Tim Kay realised that operating systems typically can only ever have 1 process listening on a port, and makes clever use of that as a locking mechanism.

The Flow

/usr/bin/solo -port=3000 /path/to/yourscript.sh
Solo tries to open port 3000
Can it open port 3000?
- Start yourscript.sh
Can’t open port 3000?
- Never mind, yourscript.sh is probably still running, will try again next time

Naturally this beats working with lock/PID files, because an open port is directly tied to a runnin process, and chances of inconsistency and having to detect and cleanup orphaned PID files, are zero.

The Example

Your crontab could e.g. look like this:

crontab -e
*    * * * * /usr/bin/solo -port=3001 /path/to/yourscript1.sh
*    * * * * /usr/bin/solo -port=3002 /path/to/yourscript2.sh
*/10 * * * * /usr/bin/solo -port=3003 /path/to/yourscript3.sh

You can now be sure that only one instance of each script will run at any given time.

Clever chaps may realise this can be used as a keepalive system for daemon-like scripts. However I suggest looking into monit or upstart for that.

The Installation

This is what makes solo great, it has basically 0 dependencies (ok Perl, but I’ll assume you have that) and is a breeze to deploy.

sudo curl -q https://raw.github.com/timkay/solo/master/solo -o /usr/bin/solo \
  && sudo chmod +x $_

Happy crontab -eing, and happy dreams with few pagerduties, you’ve earned it : )

The Alternative

As Jason mentioned, if your jobs don’t need to necessarily finish running, but just restart without overlap, another option is to use timeout:

crontab -e
* * * * * timeout -s9 50s /path/to/yourscript1.sh

The Next Level: Go Distributed with Cronlock

Ok so solo is the bomb in terms of simplicity for running 1 instance of 1 script on 1 server.

But what if you want to make sure only 1 instance of 1 script can run throughout many servers? Install the cronjobs on just one server? Hm.. what if it goes down. That means someone will have to intervene, chances are they will forget, and your nodes aren’t really expendable.

Especially in volatile environments where nodes come & go as they please, you want cronjobs to be the responsibility of the collective, not just 1 machine.

For this purpose I wrote Cronlock.

The good

You can deploy all nodes equally, install all cronjobs on all servers, if a node goes down, another will make sure your jobs are executed

The bad

It relies on a central redis server. If your cluster already relies on redis, you’re not adding reliability or a SPOF. If your cluster doesn’t, reconsider using Cronlock.

The ugly

I use straight up BASH for everything. I don’t even use redis-cli to communicate with Redis. This is because I want deployment to be as easy as with solo. Just a

sudo curl -q https://raw.github.com/kvz/cronlock/master/cronlock -o /usr/bin/cronlock \
  && sudo chmod +x $_

and you’re set.

You can visit Cronlock on Github for docs on how configure it.

Installing Hubot on Ubuntu

2012-11-20T13:27:00+01:00

We used to run Hubot on Heroko until it crashed, not sure what happened exactly but we didn’t bother bring it back due more pressing issues within our company.

Then I saw one of the most georgeous presentations ever, Intergalactic Javascript Robots from Outer Space, and it got me excited to run a Hubot again.

This time I wanted to Deploy Hubot onto UNIX. I followed the excellent article, and made some adjustments below in this blogpost to accomodate better system integration and copypastability : )

First off

I’m assuming you use Campfire. Let’s create a bot user by the name hubot there. Make sure has a working email address because Campfire will want to validate it.

Hint: It’s fun to give your hubot@yourcompany.com a nice avatar at gravatar.com.

All done? Let’s install!

Install

# Prerequisites
aptitude install build-essential libssl-dev git-core redis-server libexpat1-dev logtail

# If you don't have node 0.6+
NODE_VERSION="v0.6.9"
NODE_BASE="node-${NODE_VERSION}"
pushd /usr/local
  wget http://nodejs.org/dist/${NODE_VERSION}/${NODE_BASE}.tar.gz
  tar xf ${NODE_BASE}.tar.gz -C src && cd src/${NODE_BASE}
  ./configure && make && make install
popd

# Coffeescript
npm install -g coffee-script

# Hubot
pushd /usr/local
  git clone git://github.com/github/hubot.git && cd hubot
  npm install
popd

Note that npm is bundled with node since v0.6.3.

Hubot system user

adduser --disabled-password --gecos "" hubot

Upstart script

cat <<EOF > /etc/init/hubot.conf
description "Hubot Campfire bot"

# Campfire-specific environment variables. Change these:
env HUBOT_CAMPFIRE_ACCOUNT='companyname' # the one in: <companyname>.campfirenow.com
env HUBOT_CAMPFIRE_ROOMS='123456'
env HUBOT_CAMPFIRE_TOKEN='afafafafafafafafafafcdcdcdcdcdcdcdcdcdc'

# Subscribe to these upstart events
# This will make Hubot start on system boot
start on filesystem or runlevel [2345]
stop on runlevel [!2345]

# Path to Hubot installation
env HUBOT_DIR='/usr/local/hubot/'
env HUBOT='bin/hubot'
env ADAPTER='campfire'
env HUBOT_USER='hubot' # system account
env HUBOT_NAME='bot' # what hubot listens to
env PORT='5555'

# Keep the process alive, limit to 5 restarts in 60s
respawn
respawn limit 5 60

exec start-stop-daemon --start --chuid \${HUBOT_USER} --chdir \${HUBOT_DIR} \
  --exec \${HUBOT_DIR}\${HUBOT} -- --name \${HUBOT_NAME} --adapter \${ADAPTER}  >> /var/log/hubot.log 2>&1
EOF

Now don’t forget to vim /etc/init/hubot.conf and change the 3 HUBOT_CAMPFIRE_ environment variables.

To start, type:

start hubot

To check the log, type:

logtail /var/log/hubot.log

Use

Now check Campfire to see if Hubot entered the room, and tell him to send you an image; type:

bot: image me donuts

Some other commands to get you started may be retrieved by issuing:

bot help

This will show you:

bot <user> is a badass guitarist - assign a role to a user
bot <user> is not a badass guitarist - remove a role from a user
bot animate me <query> - The same thing as `image me`, except adds a few parameters to try to return an animated GIF instead.
bot convert me <expression> to <units> - Convert expression to given units.
bot die - End hubot process
bot echo <text> - Reply back with <text>
bot help - Displays all of the help commands that Hubot knows about.
bot help <query> - Displays all help commands that match <query>.
bot image me <query> - The Original. Queries Google Images for <query> and returns a random top result.
bot map me <query> - Returns a map view of the area returned by `query`.
bot math me <expression> - Calculate the given expression.
bot mustache me <query> - Searches Google Images for the specified query and mustaches it.
bot mustache me <url> - Adds a mustache to the specified URL.
bot ping - Reply with pong
bot pug bomb N - get N pugs
bot pug me - Receive a pug
bot show storage - Display the contents that are persisted in the brain
bot show users - Display all users that hubot knows about
bot the rules - Make sure hubot still knows the rules.
bot time - Reply with current time
bot translate me <phrase> - Searches for a translation for the <phrase> and then prints that bad boy out.
bot translate me from <source> into <target> <phrase> - Translates <phrase> from <source> into <target>. Both <source> and <target> are optional
bot who is <user> - see what roles a user has
bot youtube me <query> - Searches YouTube for the query and returns the video embed link.

Going forward

You can write your own Hubot Scripts, or use those by others.

Happy hubbing : )

Highlevel testing with CasperJS

2012-11-03T12:29:00+01:00

If you’ve written a webapp and you want to ensure that critical parts such as the signup process stay working, the best would be to have an actual user go through that process every time you change your codebase. But since that’s is both tedious & expensive, the second best thing is to automate a chrome browser (webkit engine anyway) to do this for you, and upload screenshots if anything unexpected happens.

Welcome to CasperJS!

From the CasperJS website: “Casperjs is an open source navigation scripting & testing utility written in Javascript and based on PhantomJS — the scriptable headless WebKit engine. It eases the process of defining a full navigation scenario and provides useful high-level functions, methods & syntactic sugar”.

Install

OSX

If you use Homebrew, you can install both CasperJS and PhantomJS using this command:

brew install casperjs

Ubuntu

To install both PhantomJS and CasperJS into /usr/local with this layout:

/usr/local/bin/casperjs  -> /usr/local/casperjs-1.0.0/bin/casperjs
/usr/local/bin/phantomjs -> /usr/local/phantomjs-1.8.0-linux-x86_64/bin/phantomjs

Please execute the following (Warning: purges all previous versions of CasperJS inside /usr/local/)

PHANTOM_VERSION="1.8.0"
PHANTOM_BASE="phantomjs-${PHANTOM_VERSION}-linux-x86_64"
if [ "$(/usr/local/bin/phantomjs --version 2>/dev/null)" != "${PHANTOM_VERSION}" ]; then
  DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes install libfontconfig1-dev
  pushd /usr/local/
    wget --quiet http://phantomjs.googlecode.com/files/${PHANTOM_BASE}.tar.bz2
    tar -jxvf ${PHANTOM_BASE}.tar.bz2 && rm ${PHANTOM_BASE}.tar.bz2*
    ln -nfs /usr/local/${PHANTOM_BASE}/bin/phantomjs /usr/local/bin/phantomjs
  popd
fi

CASPER_VERSION="1.0.0"
CASPER_BASE="casperjs-${CASPER_VERSION}"
if [ "$(/usr/local/bin/casperjs --version 2>/dev/null)" != "${CASPER_VERSION}" ]; then
  pushd /usr/local/
    rm -rf *casperjs*
    wget --quiet https://github.com/n1k0/casperjs/tarball/${CASPER_VERSION}
    tar -zxvf ${CASPER_VERSION} && rm ${CASPER_VERSION}*
    mv *casperjs* /usr/local/${CASPER_BASE} # n1k0-casperjs-e629586
    ln -nfs /usr/local/${CASPER_BASE}/bin/casperjs /usr/local/bin/casperjs
  popd
fi

phantomjs --version
casperjs --version

Use

Notes

You can write CaspjerJS scripts in Javascript or Coffeescript. CasperJS will just switch interpreters based on the extension of the script you feed it. For small projects like these I prefer Coffeescript.

Note that CasperJS is not node.js and though compatible with require, you cannot use any npm modules.

To check if your .coffee files are valid, I recommend running them through coffeelint (npm install -g coffeelint).

Make sure you run at least RC3 if you want to capture screenshots of timeouts as well.

Example

Here’s an example script that shows some different tricks, I’ve commented along the way. Some gems:

Anytime a testcase fails, a .png is saved. CasperJS will exit with code 1 so it’s really easy to detect a fail then upload this screenshot to campfire for example. This is possible using just curl and your campfire api keys.
Anytime a page contains: Error or Exception, a fail is automatically triggered without the need to write additional asserts for this. It can be disabled on a URL basis (in this case /nonexistent is allowed to throw these texts).

## Setup
##########################################################################

utils  = require("utils")
casper = require("casper").create
  verbose: true
  logLevel: "warning"
  exitOnError: true
  safeLogs: true
  viewportSize:
    width: 1024
    height: 768

testhost   = casper.cli.get "testhost"
screenshot = casper.cli.get "screenfile"

casper
  .log("Using testhost: #{testhost}", "info")
  .log("Using screenshot: #{screenshot}", "info")

if not testhost or not screenshot or not /\.(png)$/i.test screenshot
  casper
    .echo("Usage: $ casperjs test project.coffee --ignore-ssl-errors=yes --testhost=<testhost> --screenfile=<screenshot.png>")
    .exit(1)

## Hooks
##########################################################################

# Capture screens from all fails
casper.test.on "fail", (failure) ->
  casper.capture(screenshot)
  casper.exit 1

# Capture screens from timeouts from e.g. @waitUntilVisible
# Requires RC3 or higher.
casper.options.onWaitTimeout = ->
  @capture(screenshot)
  @exit 1

# Scan for the word notice|warning|error|exception by default
casper.on "step.complete", (page) ->
  # Skip urls that can contain 'error'/'exception'
  u = casper.getCurrentUrl()
  if (u == "https://#{testhost}/nonexistent")
    return

  @test.assertEval ->
    !$('div#content').text().match(/(notice|warning|error|exception)/i)
  , "no notices, warnings, errors or exceptions in #{u}"

## Testcases
##########################################################################

# This is an app that has everything (even the /news page) behind a login.

# try to access nonexistent when logged in (don't 404, we only tell customers what exists and what not)
casper.start "https://#{testhost}/nonexistent", ->
  @test.assertHttpStatus(302, "nonexistent should 302 when logged not in (can't show guests what exists)")
  @test.assertUrlMatch /\/customers\/login/, "redirect to login"

# open /news/ without login, errors out, should go to login,
casper.thenOpen "https://#{testhost}/news/", ->
  @test.assertTextExists "I could not give you access to", "cannot access news without login"
  @test.assertUrlMatch /\/customers\/login/, "redirect to login"
  @test.assertTitle "Please login", "login page title is the one expected"
  @test.assertExists "form[action=\"/customers/login/\"]", "login page must have a form with customer/login action"
  @fill "form[action=\"/customers/login/\"]", { "data[Customer][username]": "janedoe", "data[Customer][password]": "jsdi32ld!" }, true

# redirect to landing page /news/
casper.then ->
  @test.assertUrlMatch /\/news/, "redirected to landing page after login"

# notice login twice
casper.thenOpen "https://#{testhost}/customers/login", ->
  @test.assertTextExists "You are already logged in", "notice already logged in"

# try to access admin page
casper.thenOpen "https://#{testhost}/admin/tickets", ->
  @test.assertTextExists "I could not give you access to ", "prohibit to access admin page"

# try to access nonexistent when logged in
casper.thenOpen "https://#{testhost}/nonexistent", ->
  @test.assertHttpStatus 404, "nonexistent should 404 when logged in"

# dashboard has panels
casper.thenOpen "https://#{testhost}/customers/dashboard", ->
  @test.assertTitle "Dashboard", "customer dashboard title is ok"
  @test.assertEvalEquals ->
    __utils__.findAll(".user-dashboard div.accordion-heading").length
  , 8, "found 8 customer dashboard panels"

# calculate storage price
casper.thenOpen "https://#{testhost}/storage_accounts/add", ->
  @evaluate ->
    $("#StorageAccountBytesMax").val("10737418240")
    $("#StorageAccountPassword").val("dlfksfag!1")
    $("#StorageAccountEmail").val("janedoe@example.com")
    $("#StorageAccountBytesMax").change()

  @waitFor ->
    @evaluate ->
      $("#billabe_buy").text() != "Calculating..."
  , ->
    @test.assertSelectorHasText "#billabe_buy", "45.00", "10gb is 45.00 euros for janedoe"

# unowned invoice: prohibit
casper.thenOpen "https://#{testhost}/invoices/view/201100493", ->
  @test.assertTextExists "Invoice not found", "prohibit access to invoice of another customer"

# owned invoice: allow and check it's price is 12 cents
casper.thenOpen "https://#{testhost}/invoices/view/201100975", ->
  @test.assertTextExists "Subtotal", "my invoice has a subtotal"
  @test.assertEval ->
    $("td.total").text().indexOf("0.12") > -1
  , "invoice 201100975 total is 12 cents"


## Bombs away
##########################################################################

casper.run ->
  @test.renderResults true

See what we did? In just ~100 LoC we make sure this app deals correct prices for new products, protects people’s invoices and admin pages from unauthorized access, makes sure the login system functions & redirects correctly, and that no page except /nonexistent has any errors on it. If any of these conditions aren’t met, a screenshot is made.

Run

To run it, type something like:

casperjs \
  test \
  ./tests/project.coffee \
  --ignore-ssl-errors=yes \
  --testhost=staging.exampleproject.com \
  --screenfile=./webroot/fails/screenshot.png # || script to upload screenshot.png to campfire.

Ideally, you’d wrap this up in a script and plug it into your Continuous Integration server so that it gets run on every change.

Alternatively

While still developing, it’s really pleasant to have your Mac open the screenshot automatically after any fail:

rm -f ~/Desktop/screen.png \
 ; casperjs test ./tests/main.coffee --ignore-ssl-errors=true --testhost=www.example.local --screenfile=~/Desktop/screen.png \
|| open ~/Desktop/screen.png

Conclusion

There are also paid services you can outsource this to. Most of them offer a lot more features such as also testing against FF, IE, Opera, Mobile, etc. so it may make sense for you to use one of those. Some I know in no particular order:

As for some advantages of rolling this out yourself:

customize to your needs, run on your own CI server
the tests & actual code are stored in the same repository, hack on your code, hack on your tests, it’s all versioned and coupled, this makes it easy and fun to update your tests.
no monthly fees
and as you’ve noticed it’s actually not hard to do anymore, thanks to PhantomJS & CasperJS

Reverse a Multibyte String in PHP

2012-10-09T13:07:00+02:00

PHP’s strrev is not safe to use on utf-8 strings because it reverses a string one byte at a time. So if a character consists of multiple bytes it cannot be preserved as an entity in the reversed result.

There is no Multibyte String alternative to strrev either.

We did some googling, but strangely enough all solutions we encountered were either invalid or incredibly heavy memory/code wise.

For example:

using utf8_decode only works if your characters in the string exist in the ISO-8859-1 character set
using preg_match_all seems weirdly over-engineered
a simpler preg_match_all works, but on a 2MB string PHP was already using 150MB of memory. This is actually what sparked our search when when @renan_saddam noticed his PHP port of Github’s email_reply_parser choked on a 2MB multibyte email.

What we came up with

Is dead simple, but I’m putting it online anyway since it’s apparently not common good.

(mb_strrev.php) download

<?php
function mb_strrev ($string, $encoding = null) {
  if ($encoding === null) {
      $encoding = mb_detect_encoding($string);
  }

  $length   = mb_strlen($string, $encoding);
  $reversed = '';
  while ($length-- > 0) {
      $reversed .= mb_substr($string, $length, 1, $encoding);
  }

  return $reversed;
}
?>

Example:

<?php
echo    strrev('Gonçalves') . "\n"; // returns sevla??noG
echo mb_strrev('Gonçalves') . "\n"; // returns sevlaçnoG
?>

In our tests, the above function was factor 5x more efficient in regards to memory consumption than the preg_match_all solution.

Hope this helps

Quick server debugging with WTF

2012-10-03T16:30:00+02:00

If something weird is happening, you want to know everything that’s going on on a server, as fast as possible.

At these times, you will be very happy to have a simple alias wtf installed that you can type immediately after logging into a server, and see all that it’s busy with.

tldr; oneliner (okay, three-liner)

[ -z "$(fgrep 'alias wtf' ~/.bash_aliases)" ] \
  && echo "alias wtf='tail -f /var/log/{dmesg,messages,*{,/*}{log,err}}'" \
  >> ~/.bash_aliases && . ~/.bash_aliases

Explained

Normally you could type

tail -f /var/log/messages
tail -f /var/log/syslog
tail -f /var/log/mysql/error.log
tail -f /var/log/nginx/php-errors.log
#etc

But you can have all this goodness in one command:

tail -f /var/log/{dmesg,messages,*{,/*}{log,err}}

Now this is slightly hard to type, so we’ll save it in an alias:

alias wtf='[...]'

Cool. You can now just type wtf.

However next time you login, the alias is lost. To make it persist, we store it in your ~/.bash_aliases. This file gets sourced everytime you log into bash (if you use zsh, you will know what to do):

echo "[...]" >> ~/.bash_aliases

But this only stores the alias for your next logins. To make your current session profit, we source it now:

. ~/.bash_aliases

One more thing, you should just be able to copy paste this, without the alias getting added twice, so we make sure it does not exist yet:

[ -z "$(fgrep 'alias wtf' ~/.bash_aliases)" ]

(directly using fgrep here could cause an exit 1 if ~/.bash_aliases does not exist yet, in which case we still want to continue, so we check with -z instead)

And we put all this together as shown above in the oneliner, and we can start using wtf directly.

On a mac?

Add the following to your ~/.bash_profile

if [ -f ~/.bash_aliases ]; then
. ~/.bash_aliases
fi

Courtesy of @dogmatic and myself

: )

Blog with Octopress and Github pages

2012-09-25T15:08:00+02:00

This article aims to provide a compact tutorial for setting up an Octopress blog from scratch on OSX.

Since so many people encounter Mountain Lion breakage, the article starts with a few pointers how to get Homebrew, Git, make, Ruby going first.

Please look closely at the steps if you really want to do each of them. For instance I replace rvm with rbenv, you may want to skip that.

Replace kvz with your Github username everywhere.
Replace kvz.io with your domain name everywhere.
Replace subl with your favorite editor everywhere such as mvim, mate, or mou.

Upgraded to Mountain Lion?

Make sure you can build something again

Upgrade Xcode
Launch Xcode
Select Xcode -> Preferences from the menu bar.
Select the Downloads tab.
Install “Command Line Tools”.

“Boots into OS X, and my terminal takes FOREVER to boot. Fixing permissions on the /usr/local directory fixed most of that”

sudo chown -R `whoami` /usr/local

Fixing homebrew

brew update
brew outdated|xargs brew install
brew tap homebrew/dupes
brew install apple-gcc42 git
brew upgrade

Switch from rvm to rbenv

Remove rvm

rvm implode

Install rbenv instead, and Ruby 1.9.3-p194 which is required for Octopress

brew install rbenv
brew install ruby-build
eval "$(rbenv init -)"
rbenv install 1.9.3-p194
rbenv global 1.9.3-p194

Environment

This should be in your ~/.bash_profile, ~/.zshrc or ~/.zprofile

export PATH="$HOME/.rbenv/bin:$PATH"
eval "$(rbenv init -)"
# required for https://github.com/imathis/octopress/issues/144
export LC_CTYPE=en_US.UTF-8
export LANG=en_US.UTF-8

Source it now so you won’t have to open another tab first

source ~/.zprofile || source ~/.zshrc || source ~/.bash_profile

Make sure you have no other Ruby versions in your $PATH

echo "# ${PATH}"
# /Users/kevin/.rbenv/shims:/Users/kevin/.rbenv/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin

e.g., mine had a legacy /Users/kevin/.gem/ruby/1.8/bin in it, causing Jekyll to segfault.

Installing Octopress as your blog

git clone git://github.com/imathis/octopress.git kvz.io
cd kvz.io
ruby --version # should read ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-darwin12.2.0]
gem install bundler
rbenv rehash
bundle install

Deploying to Github project pages (http://username.github.com/project[gh-pages])

Let octopress know about this

rake setup_github_pages
# Repository url: git@github.com:kvz/kvz.io.git
rake install
rake generate && rake deploy

Note: Different from Github use pages (http://username.github.com/username.github.com[master]).

Setup a domain

echo 'kvz.io' > source/CNAME

Put these DNS records in place

A     record from kvz.io     to 204.232.175.78  (Cannot be a CNAME!)
CNAME record from www.kvz.io to kvz.github.com

Makefile

So you’ll only have to type make blog to push online ALL THE THINGS!

Edit ./Makefile

(Makefile) download

all: tweets dependencies blog

setup:
  rake setup_github_pages\[git@github.com:kvz/kvz.io.git\]

unpub:
  subl .
  fgrep -rIi "published: false" ./source/_posts | awk -F: '{print "subl " $$1}' |bash

tweets:
  rake twitter
  make blog MSG="updated twitter"

comments:
  rake build_comments

dependencies:
  (cd ~/workspace/kvz.io 2>/dev/null || (cd ~/workspace && git clone git@github.com:kvz/kvz.io)) && git pull
  (cd ~/workspace/bash3boilerplate 2>/dev/null || (cd ~/workspace && git clone git@github.com:kvz/bash3boilerplate.git)) && git pull
  (cd ~/workspace/kvzlib 2>/dev/null || (cd ~/workspace && git clone git@github.com:kvz/kvzlib.git)) && git pull
  (cd ~/workspace/transloadit-api2 2>/dev/null || (cd ~/workspace && git clone git@github.com:transloadit/transloadit-api2.git)) && git pull
  (cd ~/workspace/nsfailover 2>/dev/null || (cd ~/workspace && git clone git@github.com:kvz/nsfailover)) && git pull

blog:
  git pull && \
  bundle install && \
  rake integrate && \
  rake build && \
  rake generate && \
  rake deploy && \
  git add .; \
  git commit -am "blog update $$(date +%Y-%m-%d)"; \
  git push origin master

.PHONY: blog%

Don’t forget to commit the source for your blog.

To add your project’s master as the new origin. This is what my .git/config looks like:

[core]
  repositoryformatversion = 0
  filemode = true
  bare = false
  logallrefupdates = true
  ignorecase = true
[remote "origin"]
  fetch = +refs/heads/*:refs/remotes/origin/*
  url = git@github.com:kvz/kvz.io.git
[remote "octopress"]
  url = git://github.com/imathis/octopress.git
  fetch = +refs/heads/*:refs/remotes/octopress/*
[branch "master"]
  remote = origin
  merge = refs/heads/master

This way you can pull from octopress/master to receive updates, and push to origin/master to save your posts.

Start blogging

New article

rake new_post\["Blog with Octopress"\]
subl source/_posts/$(date +%Y-%m-%d)-blog-with-octopress.markdown

Add an about page

Add & edit like so

rake new_page\[about\]
subl source/about/index.markdown

Add a link to it in the navigation

subl source/_includes/custom/navigation.html

Add an about panel

subl source/_includes/custom/asides/about.html

and now add it to your _config.ym so it will be included on every page:

default_asides:
- custom/asides/about.html

More info

Bombs away : )

make blog MSG="Updated blog"

Thanks

Revisiting Faster PHP Sessions

2011-04-29T14:00:00+02:00

“Simplicity is prerequisite for reliability.”
Edsger W. Dijkstra How Do We Tell Truths That Might Hurt?

As our experience grows, we learn from past mistakes and discover what’s truely important in reliable systems. When designing systems, simplicity is an often heard mantra, but it isn’t getting applied nearly as much as spoken off. I’m guilty of this too. I think it’s mainly because engineers love to, well, engineer : ) and will naturally try to outsmart problems by throwing more tech at it.

Article vs Article

In the light of this, I revisit my 2008 article Enhance PHP session management. The article explains how you can use a central memcache server to store sessions for performance & scalability purposes.

Having a shared something when you can avoid it is asking for problems, and I was just throwing unneeded tech at this: network protocols, pecl modules, configuration. All vulnerable to bugs, maintenance, performance penalties and outage.

Using 2007 article Create turbocharged storage using tmpfs, we can defeat some of this over-engineering and take a simpler approach to speeding up sessions in PHP. We’ll store them decentralized in memory by mounting RAM onto the existing /var/lib/php5 session directories throughout your application servers, which I will call nodes from now on.

Make Session Dir Live in RAM

Add this to your /etc/fstab:

# Make PHP Sessions live in RAM
tmpfs /var/lib/php5 tmpfs size=300M,atime 0 0

This will make sure the 300MB RAM device will be available on your next reboot as well.

300MB is a lot.

You can decrease it later on by changing the /etc/fstab entry and

executing mount -o remount /var/lib/php5

Activate & Migrate Existing Sessions

Then execute:

# Create a temporary place for current sessions
mkdir -p /tmp/phpsessions/
# Move current sessions to it
mv /var/lib/php5/* /tmp/phpsessions/
# Activate our ramdisk
mount -a
# Move the current sessions back
mv /tmp/phpsessions/* /var/lib/php5/
# Remove the temporary placeholder
rmdir /tmp/phpsessions

Advantages

What’s nice about saving sessions in a tmpfs device compared with saving in memcache is:

you can migrate to this solution without logging people out : )
nothing needs to be installed
instead of throwing errors, it degrades gracefully as disk storage if implementation fails
you can restart/flush/upgrade any existing memcache instances without people losing sessions
it uses the default /var/lib/php5 directory, so no .ini changes, and PHP’s garbage collector will still purge old sessions
it takes away a bottleneck & single point of failure in your architecture
it’s just a mountpoint, so existing monitoring tools will automatically trigger alerts when you need to allocate more space
no locking issues with ajax calls (though I believe fixed in memcached-3.0.4beta)
no protocol overhead
less tech, so less prone to errors & bugs, easier upgrade process

Decentralizing

Now this doesn’t work in clusters without Sticky Sessions. But you’ve got to ask yourself: in huge clusters, do you really want Shared Sessions? The bigger the cluster, the more vulerable you’ll become as it really only adds a bottle-neck & single point of failure to your architecture.

With decent loadbalancers like EC2’s ELB, Pound, HAProxy it becomes childsplay to implement Sticky Sessions so that people keep ending up on the node that has their session.

When you’re designing to tolerate failure, this architecture is much more robust than depending on anything shared.

Yes, some people will be logged out when you shut down a node (vs all when your session store goes down).

To counter you could:

drain a node’s connections before you take it into planned maintenance, this way nobody is affected
rsync sessions between nodes if it’s crucial that all sessions survive outage.

This could even be automated where nodes can cover for eachother. If it’s worth the investment depends on your application. Are your nodes likely to go down completely? How many customers will get logged out? What kind of data is lost?

Even if your session store is clustered and uses persistent storage like Redis or MySQL (not the right tool for the job people): network outage, maintenance and misconfiguration can hurt you badly, logging out all customers or worse, throwing errors throughout your platform.

Problems will be bigger and harder to solve.

Whereas if the RAM mountpoint fails, /var/lib/php5 just degrades gracefully as normal disk-based storage. Making sessions slower on that 1 node, but at you’ll still be serving customers.

I welcome your thoughts on this!

Imported comments

These were imported from my old blog. Please use disqus below for new comments

Guadalupe on 2012-04-12 04:51:41
Thank you for the auspicious writeup. It in fact was a amusement account it.
Look advanced to far added agreeable from
you! However, how can we communicate?

mindzap on 2012-04-04 13:39:34
Its possible now to rank no.1 on all search engine with our unique white hats SEO techniques. Rank on the top and get maximum traffic which means maximum sales.Make your site popular online with our seo services, search engine optimization, link building, social media marketing services. High search engine rank assured.

Coach Bags on 2012-03-01 08:16:25
How’s it going? I enjoyed reading through this publish. My husband and I have been researching for this kind of article with the longest time and We know that your details about the issue at hand is spot on. I’ll be certain to introduce this posting to my neice. Can you tell me how to acquire your new RSS feed? Continue to keep on blogging!

Maciej Lisiewski on 2011-12-31 08:10:35
Sessions in ramfs is a very, very bad idea.

1. The obvious: when the machine crashes all sessions are lost. While sometimes that is acceptable it’s a definite no-go for ecommerce sites that share session id with cart id (common practice to reduce number of cookies).

2. Limited (compared to other storage solutions) capacity. With small sessions you’ll end up with a 1-4 KB session files. That means that 300MB you are using allows 75-300k concurrent sessions. Seems a lot til you remember that webcrawlers from Google, Yahoo, Bing, etc don’t use cookies and each request equals new session. With a large website number of requests in millions per day is not uncommon. Boom, suddenly no new session can be created, 502.

3. As with any file storage you loose all the benefits of storing sessions in relational database - the most basic being an ability to modify session of a specific user (log him out for example) - this way you can actually store permissions in session and avoid querying permissions per request.

I went for a hybrid solution:
- sessions are stored both in database and key-store (APC/memcached/whatever I plug in depending on site size)
- database is checked only if key-store returns no match - if there is a match in db it’s copied to key-store.
- database is written to only if session value is modified or if time since last persistent (db) store is longer than predefined interval
- most popular/aggressive crawlers are matched by user agent and no session is created for them

Result is fast. Not as fast as tmpfs, but close enough and it supports sharding out of the box (add modulo of request id to select memcached and/or db shard and you’re done) without the need to sync files. No fancy clusters - just add new box if you need it and change divisor in config.
Garbage collection is dead simple - cron job that deletes entries from db that had no persistent store for predefined interval - key-store expires on it’s own.
You can kill any memcached instance - new one will load all the sessions from db (there will be a load spike, but each will be read just once).
It degrades gracefully if at least db works. If it doesn’t you’re fucked anyway ;-)

webhoster on 2011-12-30 05:36:51
You got a point about load balancers. At least for the presentation layer.

Still might need to cluster a central data layer.

Janak on 2011-12-01 11:47:42
AWESOME! Instant, noticeable improvement. And super useful for me as I have some servers with pretty slow Disk I/O.

bird on 2011-11-30 08:46:27
nice work

Mike on 2011-11-01 15:56:10
Well now, sticky sessions are a big nono, since those will hinder your very purpose for using a cluster.

If I have 3 servers, and I use sticky sessions and I do have a flash flood, I would get:
- webserver1 - 10% load
- webserver2 - 150% load
- webserver3 - 40% load
instead of using all three in an even way.

Maybe sticky would be nice for 10-20+ servers in a cluster, numbers that would led the statistics work and assure an almost even distribution.

But having inbalance in an small cluster is like not having the cluster at all. We do LB for performance _and_ failover not only for failover.

David on 2011-08-25 18:17:09
Hi,

I implemented this as we had a lot of issues with session_start taking up to 3 seconds, just because of opening files, and our page loads decreased drastically!

Since this was an absolutely fantastic result, we decided to save our cached objects (which were now cached into a database), in ram as well… translations are now saved in a serialized array into ram, and still has a copy in the database.

page loads for some of our larger pages went from 7 seconds to & lt;0.3 seconds.

So yes - our profiling shows that this is a GREAT result. - we just put in 4 GIGS of additional RAM, and off we go!

Thanks for a great post! This absolutely saves our day, and helps us maintain good structured code, as we didn’t want to start compromising by taking shortcuts in our code & quot;just for performance reasons& quot;, as we see happen so many times, unfortunately.

Steve Clay on 2011-05-11 16:57:47
Does any storage over 300M roll over to disk like & quot;virtual memory& quot;? If not, how do you handle too many/too big session files?

Ondra on 2011-05-09 10:55:41
We have a more techy solution - db with memcached.

http://www.keboola.com/blog/php-sessions-with-memcached-and-a-database-session-in-the-cloud-done-right/

Coret on 2011-05-07 12:24:25
When I tried to move my session store to the newly created tmpfs it ran out of space (inodes). To counter this I added & quot;nr_inodes=400k& quot; to the fstab entry.

Also, try to specify a good mode, so your webserver can read/write session files (something like mode=777 in your fstab entry, or less for more security).

First impression: the i/o wait on my server is significantly down!

Question: I use & quot;3;/var/lib/php5& quot; as session.save_path, so there’s a directory structure of 3 levels deep. When my server is rebooted now, this directory structure has to be regenerated… Should I just abandon this approach and just use 1 directory with 235K files?

Ben Wong on 2011-05-06 00:16:32
Interesting idea. I’d like to see some benchmarks of the performance of this vs straight up disk access sessions.

I’d be surprised if there is a huge performance improvement over regular disk based sessions. Unless you’re on something like Amazon’s ELB which isn’t a real disk anyways.

A technique that I’m doing is using signed data in cookies as a session storage technique.

Pros:

1. no server side state
2. no disk access to check
3. No locking

Cons:

1. bigger cookies (but still & lt; 100 bytes)
2. no server side invalidation of a session w/out some server side state. We use memcache for this, but in a simple, does this key exists? Ok cookie is valid.

For the application I’m working on we’ll be seeing hundreds of req/second and keeping less state makes scaling out a lot easier.

Also each user is sending requests concurrently so locking *all* requests until the current one finishes really slows down performance. Instead, to prevent race conditions on data we let the database do atomic operations and always code knowing that data could have changed from the time you read to the time to write back to the db. Compare/Swap patterns are great here.

For security, so users can’t forge their own cookies we:

a) sign them w/ sha1
b) expire them with a timestamp (int, embedded into data)

When the cookie comes in, we validate the data (json serialized), the signature and that the cookie has not expired.

If everything is OK we process the request and issue a *new* cookie.

Frank de Graaf on 2011-05-05 14:24:46
This is a great idea. What I like most is that you get rid of the dependencies of other applications and that you don’t have to change your ini settings for it. As you said, it would make it easier to maintain and to migrate. Excellent article, cheers.

Revisiting Spaces and Tabs

2011-03-31T14:00:00+02:00

This article in 50 words: I used to prefer spaces vs tabs, now I don’t care so much, think it’s more important that you can easily switch on a per-project basis. Have some thoughts on how conventions should be established, and I’ll demonstrate bash code that can convert your codebase to a new standard.

Back in the day

I used to prefer spaces over tabs so code would look consistent throughout the monospace universe. No matter what crappy viewer people ended up using on it: as long as it supported a monospace font, your code looked as intended. I also did some work for PEAR and they enforced spaces in their thorough Coding Standards. Which was later adopted by many frameworks, sometimes with small deviations.

Quest for the holy Convention

After years of spaces in my code I started using CakePHP and their standard was tabs. Nothing to get hung up over, but after a while my code started intermingling with other Cake developer’s code and that’s when it gets a little hairy.

So I started using tabs there cause in my view conventions are much like traffic rules as I mentioned before in SQL Formatting

It’s irrelevant if people drive on the right or left side of the road, as long as they do the same

At Transloadit we started using 2 spaces for JavaScript as it’s the way of node.js. And then there’s a little Ruby project I started hacking on and they also like 2 spaces.

Adopt many Conventions

Coding standards change. Within a project, organization, framework, and even language. Or they change for you simply because you contribute to different aforementioned forms.

Instead of trying to enforce one preference throughout all of my projects, I adopt the rules of the domain at hand. In this order:

Project > Organization > Framework > Language

(where conflicting, left wins from right)

On a side-note, I think it’s the convention-designer’s responsibility to align his conventions with the layer they’re building on as much as possible. So frameworks should look at their language. companies should look at their framework. This makes for consistently looking codebases. And that helps encouraging involvement. Nobody likes messing with code that suffers from poor housekeeping.

In order to be flexible about this, it helps a lot if your IDE supports per-project settings (I currently use both NetBeans & Vim, and they do an fine job at that). In NetBeans it’s easy to mess up though cause it’s pretty much indentation agnostic. So sometimes you won’t notice you’re filling a 4-spaces file with tabs, ruining the code in other views/editors.

Once that happens, or maybe if you’re porting big chunks of ‘legacy’ code to a new standard that’s closer to your layer, you’ll need decent conversion scripts.

Switch Conventions

There are many pages on converting spaces to tabs on Linux or Mac, but I wasn’t satisfied as they:

Also change non-leading whitespace (which may not be what you want, e.g. a tab-indented document could still use spaces to promote readability around assignments, or inside big strings)
Don’t support multiple levels of indention
Can’t be run from command-line (e.g. depend on IDE)
Are specific to a language (indent / astyle)
Messed up my indentation (expand / unexpand)

In an attempt to come up with a reliable tabs vs spaces converter that you can simply run inside a directory and will traverse your source files, I’d like to share a couple of lines of BASH.

Warnings:

Only do this when your source is under version control, these snippets make no backups! So execute, test, verify, commit. Or hit git reset --hard if you don’t like it (leave a comment for improvement!)
Currently processes .php, .ctp, .js, .css, .sh. But can easily be modified to do other extensions as well.

Ubuntu

# 4 Spaces to tabs
find -P  . -type f -regextype egrep -regex '.*\.(php|ctp|js|css|sh)$' -print0 | xargs -0  sed -i"" -e ':repeat; s/^\(\t*\)    /\1\t/; t repeat'

# extra: Strip any trailing whitespace
find -P  . -type f -regextype egrep -regex '.*\.(php|ctp|js|css|sh)$' -print0 | xargs -0  sed -i"" -e 's/[[:blank:]]*$//g'

# extra: Strip any trailing blank lines (http://www.eng.cam.ac.uk/help/tpl/unix/sed.html)
find -P  . -type f -regextype egrep -regex '.*\.(php|ctp|js|css|sh)$' -print0 | xargs -0  sed -i"" -e :a -e '/^\n*$/{$d;N;ba' -e '}'

# extra: Strip any trailing PHP closing tags
find -P  . -type f -regextype egrep -regex '.*\.(php|ctp)$' -print0 | xargs -0 sed -i"" -e :a -e '/^
*$/{$d;N;ba' -e '}'

# extra: Check the PHP files for syntax errors
find -P  . -type f -regextype egrep -regex '.*\.(php|ctp)$' -exec php -l {} \; > /dev/null

Mac

On a Mac? You need GNU sed! - Read below.

# 4 spaces to tabs
find -P -E . -type f -regex '.*\.(php|ctp|js|css|sh)$' -print0 | xargs -0 gsed -i"" -e ':repeat; s/^\(\t*\)    /\1\t/; t repeat'

# tabs to 2 spaces
find -P -E . -type f -regex '.*\.(php|ctp|js|css|sh)$' -print0 | xargs -0 gsed -i"" -e ':repeat; s/^\(\(  \)*\)\t/\1  /; t repeat'

# extra: Strip any trailing whitespace
find -P -E . -type f -regex '.*\.(php|ctp|js|css|sh)$' -print0 | xargs -0 gsed -i"" -e 's/[[:blank:]]*$//g'

# extra: Strip any trailing blank lines (http://www.eng.cam.ac.uk/help/tpl/unix/sed.html)
find -P -E . -type f -regex '.*\.(php|ctp|js|css|sh)$' -print0 | xargs -0 gsed -i"" -e :a -e '/^\n*$/{$d;N;ba' -e '}'

# extra: Strip any trailing PHP closing tags
find -P -E . -type f -regex '.*\.(php|ctp)$' -print0 | xargs -0 gsed -i"" -e :a -e '/^
*$/{$d;N;ba' -e '}'

# extra: Check the PHP files for syntax errors
find -P -E . -type f -regex '.*\.(php|ctp)$' -exec php -l {} \; > /dev/null

Run into problems?

Please let me know, I’ll update the article so that these lines become the perfect converters.

On a Mac? You need gnu-sed

Mac OSX (BSD) has a cripled sed. This illustrates my point:

# On Mac:
$ echo "1|2|||5||7|" | sed -e ': repeat; s/||/|NULL|/; t repeat'
1|2|||5||7|

# On Linux:
$ echo "1|2|||5||7|" | sed -e ': repeat; s/||/|NULL|/; t repeat'
1|2|NULL|NULL|5|NULL|7|

Luckily you can get GNU sed for Mac OSX just as well. Get homebrew, then run:

brew install gnu-sed

And change all of sed references to gsed.

That’s it

How do you deal with changing coding standards?

Imported comments

These were imported from my old blog. Please use disqus below for new comments

Marc-André on 2012-04-30 23:35:13
Thanks. Very helpful to deal with my team who decided that tabs were better. Even for Ruby.

b b on 2011-11-29 06:08:06
kjyg kjhgku uhgu

sujon on 2011-08-10 23:25:14
Live sports on your pc

& lt;a href=’http://livestreaming-sports.com’& gt;LIve streaming sports on your pc such as NFL, Rugby, soccer,NBA…& lt;/a& gt;

sujon on 2011-08-10 23:18:04
IT is a good site for sports

& lt;a href=’http://livestreaming-sports.com’& gt;LIve streaming sports on your pc such as NFL, Rugby, soccer,NBA…& lt;/a& gt;

joe on 2011-04-11 08:23:22
nice info.thanks.

Optimize your Synology NAS for downloading

2011-02-28T13:00:00+01:00

I recently bought a NAS so my data is safe & available, with the benefit of being low power / noise / heat. I’ve considered Netgear, QNAP, but decided to go for a Synology as it was affordable, still had a big community, decent reviews & Time Machine support.

I wanted 4 bays so that I could use RAID5 and only lose 25% space on fault tolerance instead of (RAID1) 50%. Synology has 2 offerings in the 4-bay home-user range this year: the DS411+ (fast) and the DS411j (slow).

I figured as long as it can blast a bandwidth adequate for 1080p over my network, I’d save myself some money (300$ vs 600$), heat and power consumption that come with the more powerful + version.

However now that it’s here I want it do download from newsgroups and am running into performance issues with my junior edition.

No worries. With a little bit of hacking you can squeeze just enough performance out of this thing to make sense of it all.

Here’s how I turned my budget NAS that’s mediocre at 8 things into a more powerful one that’s good at 3 things: downloading / file serving / backups.

Warning

This article assumes you’re somewhat skilled in Linux. By applying these suggestions you could seriously mess up your Disk Station.

I’m doing this on a DS411j running DSM 3.0. Your mileage may vary.

Downloading

In an earlier article I described how to install SABnzbd. After testdriving it for a while I was never able to get it to download above 3MB/s (2 average). Where as nzbget (the program used by Synology’s own Download Station), peaks at 8MB/s (6 average).

Although I really like that SABnzbd automatically unpacks your downloads, these speed differences made me decide to go back to nzbget. The j is just not powerful enough to do SABnzbd at these speeds, and I can write auto-unpackers myself.

Optimal Config

I found that optimal speeds can be reached by letting your Synology download with 8 connections on 1 single download. With these settings the load reaches 11, so don’t expect your NAS to do anything else while it’s busy. But at least you’re saturating your connection.

If you want it to multitask, limit it to 1 connection on 1 single download at any time, but you won’t see it peak beyond 2MB/s.

If you use it for torrents as well, you don’t want 1 slow torrent blocking the rest of the queue. In that case, set it to 2 to 3 connections with 2 to 4 threads each for optimal downloading.

Turn off unused protocols

Decide on 1 file-sharing protocol (I chose Mac File service cause all my systems speak it and use Time Machine, but SMB/Windows is typically the right choice). Disable the rest in your configuration panel, saving a few precious MBs of RAM.

This is all just done from your web-interface.

SSH access

Before you can do any hacking on your Synology, turn on SSH access in the web-interface’s control panel. You can now type: ssh root@<nas ip>. Followed by sh. The root password is the same as admin password.

AppStore : )

Get your hands on ipkg, which is like your Synology’s secret AppStore. From here on, it’s much easier to install cool additional software.

Turn off media indexers to free up CPU & Memory

When I logged in to see what was eating up my NAS’ resources, I saw a lot of processes running that I don’t need such as thumbnail generators and media indexers (ffmpeg & convert). They were endlessly consuming 100% CPU, leaving nothing for my other tasks.

Any currently available NAS is a terrible media streamer. And that’s ok, just get yourself an AC Ryan ($80) or Boxee Box ($250) to do that instead and dedicate your NAS to less tasks.

In my case that meant killing off all these wannabe media processes that are eating up your poor handheld CPU with 128MB RAM (every MB we’ll save from this point forward counts to faster download speeds : )

So if you don’t use the Photo/Media/iTunes station and would like more power for other tasks, consider turning off indexers:

1) Turn off all services in the bottom configuration panel (iTunes, everything except Download Station, unless you’re going to use SABnzbd for this)

2) Login as root via SSH and stop all indexing:

/usr/syno/etc/rc.d/S??synoindexd.sh stop
/usr/syno/etc/rc.d/S??synomkflvd.sh stop
/usr/syno/etc/rc.d/S??synomkthumbd.sh stop
killall -9 convert
killall -9 ffmpeg
# If you don't use Download Station (but e.g. SABnzbd instead):
# /usr/syno/etc/rc.d/S??pgsql.sh stop

3) Make sure they won’t restart on your next reboot.

chmod -x /usr/syno/etc/rc.d/S??synoindexd.sh
chmod -x /usr/syno/etc/rc.d/S??synomkflvd.sh
chmod -x /usr/syno/etc/rc.d/S??synomkthumbd.sh
# If you don't use Download Station (but e.g. SABnzbd instead):
# chmod -x /usr/syno/etc/rc.d/S??pgsql.sh

Hint) After a DSM firmware upgrade, you need to repeat these steps.

Custom cleanup & rename script cause SAB is too slow

Building your own cleanup scripts can be fun (and risky). If you want to get into it, you’ll need some system tools at your disposal.

Here’s what I cooked up to take care of my downloads:

(sabunpacker.sh) download

#!/opt/bin/bash
# @todo: Don't delete parent dir if Dir == Root
# @todo: Root = $1 - But what about series!

set +x
export PATH="/opt/bin:/opt/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/syno/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/syno/bin:/usr/syno/sbin:/usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/syno/bin:/usr/syno/sbin:/usr/local/bin:/usr/local/sbin"

# Locking
LockFile="/volume1/downloads/nas_is_unpacking.lock"
if [ -f "${LockFile}" ]; then
  echo "Lockfile still exists: ${LockFile}. Aborting"
  exit 0
fi
trap "{ rm -f ${LockFile} ; exit 255; }" EXIT
date > ${LockFile}

echo "Running ${0} on $(date)"

Root="/volume1/downloads"
Home="$(pwd)"
Purged=""

# Downloadstation
echo "Looking for downloadstation tasks..."
Prevdir=""
find ${Root}/_queue -mmin +5 -iname '*.nzb' -o -iname '*.torrent' |sort | while read File; do
  Dir="$(dirname "$File")"
  if [ "${Prevdir}" != "${Dir}" ]; then
      cd "${Dir}"
      echo ""
      echo "= $(pwd)"
      echo "================================================================================================"
  fi
  # Process the first par file in this directory thanks to |sort
  /opt/bin/downloadstation add "${File}"
  if [ $? -eq 0 ]; then
      echo "Successfully added ${File}; purging file"
      rm -f "${File}"
  else
      echo "Unable to add ${File}"
  fi  
  Prevdir=$Dir
done
cd "${Home}"

# PAR
echo "Looking for files to repair..."
Prevdir=""
find ${Root} -mmin +5 -iname '*.par2' |sort | while read File; do
  Dir="$(dirname "$File")"
  if [ "${Prevdir}" != "${Dir}" ]; then
      cd "${Dir}"
      echo ""
      echo "= $(pwd)"
      echo "================================================================================================"
      # Process the first par file in this directory thanks to |sort
      par2 r "${File}"
      if [ $? -eq 0 ]; then
          echo "Successfully repaired; purging par files"
          rm -f *.par2
          rm -f *.PAR2
      else
          echo "Unable to repair; purging entire directory"
          Purged="${Purged}${Dir}\n"
          cd ..
          rm -rf "${Dir}"
      fi
  fi
  Prevdir=$Dir
done
cd "${Home}"

# RAR
echo "Looking for rar files to unpack..."
Prevdir=""
find ${Root} -mmin +5 -iname '*.rar' |sort | while read File; do
  Dir="$(dirname "$File")"
  if [ "${Prevdir}" != "${Dir}" ]; then
      cd "${Dir}"
      echo ""
      echo "= $(pwd)"
      echo "================================================================================================"
      # Process the first rar file in this directory thanks to |sort
      unrar e -y -o+ -p- "${File}"
      if [ $? -eq 0 ]; then
          echo "Successfully unpacked; purging rar files"
          rm -f *.rar
          rm -f *.r[0-9][0-9]
          rm -f *.s[0-9][0-9]
          rm -f *.t[0-9][0-9]
      else
          echo "Unable to unpack; purging entire directory"
          Purged="${Purged}${Dir}\n"
          cd ..
          rm -rf "${Dir}"
      fi
  fi
  Prevdir=$Dir
done
cd "${Home}"

# 7zip
echo "Looking for 7zip files to unpack..."
Prevdir=""
find ${Root} -mmin +5 -iname '*.7z.001' |sort | while read File; do
  Dir="$(dirname "$File")"
  if [ "${Prevdir}" != "${Dir}" ]; then
      cd "${Dir}"
      echo ""
      echo "= $(pwd)"
      echo "================================================================================================"
      # Process the first 7zip file in this directory thanks to |sort
      7z x "${File}"
      if [ $? -eq 0 ]; then
          echo "Successfully unpacked; purging rar files"
          rm -f *.7z.[0-9][0-9][0-9]
      else
          echo "Unable to unpack; purging entire directory"
          Purged="${Purged}${Dir}\n"
          cd ..
          rm -rf "${Dir}"
      fi
  fi
  Prevdir=$Dir
done
cd "${Home}"

# Move 1 Dir up & rename to parent dir
echo "Looking for files to clean..."
Prevdir=""
find ${Root} -mmin +5 -iname '*.mkv' -o -iname '*.avi' |sort | while read File; do
  Dir="$(dirname "$File")"
  Parent="$(dirname "$Dir")"
  if [ "${Prevdir}" != "${Dir}" ]; then
      cd "${Dir}"
      echo ""
      echo "= $(pwd)"
      echo "================================================================================================"
      rm -f *.1 2> /dev/null
      rm -f *.2 2> /dev/null
      rm -f *.nzb 2> /dev/null
      rm -f *.nfo 2> /dev/null
      rm -f *.par2_hellanzb_dupe0 2> /dev/null
      rm -f *.sfv 2> /dev/null
      rm -f *.srr 2> /dev/null
      rm -f *.segment000[0-9] 2> /dev/null
      # in the middle
      rm -f *[.-][Ss][Aa][Mm][Pp][Ll][Ee][.-]*.{mkv,avi,mpg,srs} 2> /dev/null
      # at the end
      rm -f *[.-][Ss][Aa][Mm][Pp][Ll][Ee].{mkv,avi,mpg,srs} 2> /dev/null
      # at the beginning
      rm -f [Ss][Aa][Mm][Pp][Ll][Ee][.-]*.{mkv,avi,mpg,srs} 2> /dev/null
      # complete
      rm -f [Ss][Aa][Mm][Pp][Ll][Ee].{mkv,avi,mpg,srs} 2> /dev/null

      # Synology media thumbs
      rm -rf @eaDir
  fi
  Prevdir=$Dir
done
cd "${Home}"

# Move lonely files 1 dir up & rename to parent dir
echo "Looking for lonely files to promote 1 directory up..."
Prevdir=""
find ${Root} -mmin +5 -iname '*.mkv' -o -iname '*.avi' -o -iname '*.ts' |sort | while read File; do
  Dir="$(dirname "$File")"
  Parent="$(dirname "$Dir")"
  if [ "${Prevdir}" != "${Dir}" ]; then
      cd "${Dir}"

      if [ "$(ls -l |grep -v 'total ' |wc -l)" = "1" ]; then
          Basedir="$(basename "${Dir}")"
          Newname="$(echo "${Basedir}")"
          Ext=${File##*.}
          Newname="${Newname}.${Ext}"

          #cmd="mv \"${File}\" \"${Parent}/${Newname}\" && rmdir \"${Dir}\""
          mv "${File}" "${Parent}/${Newname}" && rmdir "${Dir}"
          echo "promoted: ${Parent}/${Newname}"
      fi
  fi
  Prevdir=$Dir
done
cd "${Home}"

## TV Episodes
# Please use FileBot instead. Much better results.
#if [ "${1}" = "tvnamer" ]; then
# echo "Looking for tv episodes to rename..."
# tvnamer -r --batch /volume1/video/series
#fi

# REPORT
if [ -n "${Purged}" ]; then
  echo ""
  echo "Had to purge these directories cause they were damaged beyond repair:"
  echo -e "${Purged}"
fi

echo "Done"

It runs every 15 minutes by cron, will remove broken downloads, unpack complete downloads, move lonely files 1 directory up, delete a bunch of unwanted extensions, etc. It makes a few assumptions (e.g. downloads must be in /volume1/downloads), so be sure to only use it for inspiration.

It’s a work in progress, and improvements are more than welcome.

Downloadstation CLI

To have your Synology scan a directory for new download tasks, you can use Downloadstation CLI.

ipkg install python24 py-pgsql py24-mx-base
wget -O- http://downloadstation.jroene.de/downloadstation > /opt/bin/downloadstation
chmod 755 /opt/bin/downloadstation

With the command

downloadstation add <nzbfile>

The download will be added to the queue. If you use an adaptation of my unpacker script, it will already automatically scan /volume1/downloads/_queue for any new torrent or nzb task.

Tools

These programs may take up a little bit of space, but won’t be active in memory until you call upon them (except for cron), so feel free to install without performance loss:

ipkg install vim bash bash-completion less rsync mtr \
  sudo tshark htop openssl mlocate perl ack hdparm sysstat dstat \
  bzip2 unrar unzip zlib p7zip wget

wget -O- --no-check-certificate https://raw.github.com/timkay/solo/master/solo > /usr/bin/solo
chmod a+x /usr/bin/solo

Optionally do ipkg install clamav so you can run clamscan on freshly downloaded files and check them for viruses (I decided not to).

Renaming files

There’s a neat program called tvnamer that will rename all your TV series files.

Install:

ipkg install python25 py25-setuptools git
cd /volume1/@tmp
git clone https://github.com/dbr/tvnamer.git
cd tvnamer
python setup.py install
ln -s /opt/local/bin/tvnamer /usr/bin/tvnamer

Use:

tvrenamer -r /volume1/video/tv

FileBot is even better but requires a GUI.

Crontab

Crontab works slightly different than on more high-level Operating Systems.

Here’s how to edit your crontab:

vim /etc/crontab

Every job needs a user prefix. e.g. root:

*/15 * * * * root /usr/bin/solo -port=1111 /volume1/video/unpacker.sh 1>&2 > /volume1/@tmp/unpacker.log

When you’re done editing the new crontab, reload it by executing:

/usr/syno/etc.defaults/rc.d/S??crond.sh stop
/usr/syno/etc.defaults/rc.d/S??crond.sh start

Tmux or Screen

If you start programs from within tmux, you can close your SSH session without killing it. You can check back later on it with tmux attach || tmux.

This makes it perfect to run cleanup/rename scripts in while you’re still experimenting and need to check up on them regularly.

Tmux similar to screen, but I think it’s a bit easier to deal with (just tmux attach || tmux is all).

However screen is a lot easier to install thanks to ipkg, so pick your poison.

Screen

ipkg install screen

Tmux

ipkg install libevent optware-devel ncurses-dev

# http://forum.synology.com/enu/viewtopic.php?f=90&t=30132
mkdir /opt/arm-none-linux-gnueabi/lib_disabled
mv /opt/arm-none-linux-gnueabi/lib/libpthread* /opt/arm-none-linux-gnueabi/lib_disabled
cp /lib/libpthread.so.0 /opt/arm-none-linux-gnueabi/lib/
cd /opt/arm-none-linux-gnueabi/lib/
ln -s libpthread.so.0 libpthread.so
ln -s libpthread.so.0 libpthread-2.5.so

cd /volume1/@tmp
wget http://sunet.dl.sourceforge.net/project/tmux/tmux/tmux-1.4/tmux-1.4.tar.gz
tar -zxvf tmux-1.4.tar.gz
cd tmux-1.4
export CC=gcc
export CFLAGS="-L /opt/lib -I  /opt/include/ncurses"
./configure --prefix=/opt # prefix is not supported. So we'll need some symlinks
make # This will take a while
make install
ln -s /opt/lib/libevent-1.4.so.2 /usr/lib/libevent-1.4.so.2
ln -s /opt/share/terminfo/* /usr/share/terminfo/

Imported comments

These were imported from my old blog. Please use disqus below for new comments

BC on 2012-09-10 18:49:16
Thanks for taking the time to write this up. I just upgraded to DSM 4.1 and I had to revisit this issue. This time I found a work-around. I’m just using the DS to archive photos (not view them) so I didn’t care at all about thumbnail generation (or indexing). By moving the photo files OUT of the & quot;/Photos& quot; directory, DSM will ignore indexing/thumbnail creation. (By default it will index and create thumbnails for anything put into a folder called & quot;/Photos& quot;. It seems to have done the trick (I just created a separate shared folder called & quot;/Archives& quot; and made sure that under the Media Indexing Service (in Control Panel) that this folder was NOT set to be indexed under the & quot;Indexed Folder& quot; tab. (& quot;/Photos& quot; is a default that cannot be removed from Indexing).

On a separate note, I would like to start using my Synology for downloading, but need to set it up with a VPN first (I want a secure tunnel). I use Viscosity on my Macs as a VPN client but I’m confused as to how to set up a VPN *client* on the Synology (not set it up as a VPN *server*).

Any suggestions?

Daniel Orum on 2012-08-21 09:57:56
Hello all.

First of all, thanks for a great blog. very useful.

But im having a problem with sab and my synology nas 207+ which i hope you guys could help out with.

Im experiencing that the sab program is stopping regularly on my nas. If i download with 200kbps it can stay running for half a day, if i turn the speed up to 800kbps it normally shuts down after 2-3 hours. Im running sab, Couch Potato, python and sickbeard on my nas. They are all from Superzebulon.

I have followed your blog, and i have stopped the processed that i dont use. Byt it dosen’t seem to make a difference. I also tried following this tutorial: (crontab job)

http://thanatosblog.wordpress.com/

And under & quot;trouble with sab& quot;, i tried following the part where i entered a line in Crontab.

But my problem is that i cant access crontab. Even with the commands:

crontab -e
or
vim crontab
Putty tells me: crontab: file not found

And i cant, for the life of me, figure out what to do next. So even with the tweaks on this great blog, im still experiencing that sab is shutting down. Could soneone please help me figure out how i avoid this? And how do i get access to the crontab as described in this blog?

Sorry for partially referring to another blog, but i just dont know where else to ask.

Daniel

Mark Smith on 2012-07-25 09:55:41
Thanks to René’s post on 30 April 2011. It seems even Matthias Radig (the creator of Download CLI) is unaware of it’s ability to handle nzb files.

Mark Smith on 2012-07-25 08:44:54
Hello,

I was very interested in the Download CLI section. Your article gives the impression it can handle nzb files which it can not.

Email received from Matthias Radig:



The script does not support NZB downloads and I do not plan to add this feature myself.

However, as the script is licensed under the GPL, someone else might in

the future.

Have I missed something?

Hugo on 2012-03-10 22:23:10
Problem was in solo, not in the redirection of stdout/stderr: I had some piece of HTML as program instead of the perl code… See also comment #13.

Gr. Hugo

Hugo on 2012-03-10 17:04:40
Great introduction to using your syno full speed. Thanks!

The redirect in the crontab file is not working (on my Syno). I changed this:
1& gt;& amp;2 & gt;
into:
2& gt;& amp;1 & gt;
(meaning stderr is redirected to the same file as stdout)

Regards,
Hugo

Valie on 2012-02-28 05:19:09
Marcello, Upload Image files using Synology Assistant. In this way, the PC on which Assistant runs, will do the computing for thumbnails generation

moydeliche on 2012-02-26 21:53:07
Hi Kevin,

Same issue as frey when lauching the script. Synthax errors are fournd…

Any idea?

marcello on 2012-01-27 13:44:13
Hello, thanks for your advice. I have a DS211j which creates miniatures for my 50000 pictures in a very very slow.
Do you know a way to create thumbnails on the PC and then upload them to the NAS?
Thanks for attention. By

ritonlajoie on 2011-12-10 20:52:08
Hi ! Thanks a lot for your script.
I’m having an issue with folders having multiple episodes in them. For example, a folder containing all the rar/par files for a whole season.

The script will actually process the first file, then remove all the remaining. Is there any way you could look into improving your script ?

Thanks !

espocartoons on 2011-12-10 11:38:30
Im really having difficulty editing or inputting anew line in Crontab.

I’ve gone into the files using vim /etc/crontab

but i cant enter a new line? I have messed around and managed to put a line between the two which are already in there but when i try to :quit it wont let me?

Michael on 2011-11-27 04:59:49
the section & quot;turning off media indexers& quot; really speed things up, thanks for the tips.

still waiting for Synology to fix HTTP downloads in Download Station, I can only get 30-150KB/s with HTTP while I can get 600-900KB/s with torrents.

wariyo on 2011-11-21 17:05:14
Is it possible to install FileBot CLI also on Synology NAS with PPC processor?

rednoah on 2011-11-09 16:14:15
FileBot as an awesome CLI, working very well for a few months now.

Sebastian on 2011-11-02 10:46:35
Just a small modification (I just had to make the adjustments again because of the DSM update): The URL for the curl command to download & quot;solo& quot; should be https://raw.github.com/timkay/solo/master/solo if you don’t want to download a redirected info page. Took me a moment to figure out why that can’t be executed. ;-)

Vince on 2011-10-19 10:50:41
Hi Kevin,

Great article, makes me want to buy a DS411. I already have fun_plugged a CH3MNAS to be an automatic RSS torrent scanner/downloader, but I now have the cash to take it a step further; I want my own PHP/MySQL server and do the download thingy at the same location.

So my question is: when I go berserk on inserting neat little ipkg files in that NAS, will I compromise the server functionality? I mean, you literally state: & quot;Here’s how I turned my budget NAS that’s mediocre at 8 things into a more powerful one that’s good at 3 things: downloading / file serving / backups.& quot;

Best,
Vince

Frank on 2011-10-12 19:50:16
re: & quot;After testdriving it for a while I was never able to get it to download above 3MB/s (2 average).& quot; and & quot;The j is just not powerful enough to do SABnzbd at these speeds& quot;

After installing sabnzbd and before it is restarted I can only get 3 Mbps and shutting down other services did not help. Only after restarting sabnzbd can it max out my bandwidth at 6 Mbps. Note that the Synology box may also have to be restarted too. Mine is restarted daily because I schedule it to power down on workdays during peak electricity rate.

I only found out about this several days after installation after re-starting sabnzbd in my attempt to start the queue. Restarting sabnzbd did not not fixed the queuing problem and it turned out to be caused by a schedule to pause all activities at that period. It did however, speed up my downloads.

Software used:
DSM 3.2-1922; Build Date: 2011/09/06.
SABnzbd 0.6.9 spk is from synoblog.superzebulon.org/2011/09/sabnzbd-0-6-9-1-spk/

Hardware:DS-211j

Settings:
Download threads: 12 SSL connections, from Astraweb.

Frey on 2011-10-01 02:07:39
I am getting the followring erro meassage when i try running the unpacker.sh script… any idea what i am doing wrong? :)



: command not found:

: invalid optione 5: set: +

set: usage: set [--abefhkmnptuvxBCHP] [-o option] [arg ...]

: command not found:

unpacker.sh: line 41: syntax error near unexpected token `fi'

'npacker.sh: line 41: ` fi

Sebastian on 2011-09-29 08:22:16
Thank you very much for the script.
I had the problem that after downloads finished, the whole volume1/downloads folder was disappearing. After a bit of poking around, I found out that the problem was some of the downloads ended on & quot;.par2& quot;, so the find command tried to repair those. The command & quot;-type f& quot; limits the search to files, and everything works just fine. Thanks again!

rednoah on 2011-09-10 04:33:14
FileBot will support CLI in with the next release. There is a test version available right now, check forums on sourceforge for details.

Tim on 2011-06-22 19:46:07
I followed your guide, but cannot seem to get my full connection speed. I have 120Mbit, but the speed is always around 6MB/s, which is not even half of what I have. Is this really the max? This is rather disappointing for a NAS marketed as a download box.

h0me5k1n on 2011-05-15 12:59:25
Excellent article… I’m using SABnzbd at the moment but I’m sorely tempted to switch usenet downloading to Synologys own Download Station after reading this. I’ve had problems updating SABnzbd to the newest versions on occasion.

I created a script to download torrents from RSS feeds (made before I was using a DiskStation). I modified it to use jroenes DownloadStation-CLI to load the torrents; it could easily be modified to download nzbs via rss. Using the comments from Rene they could be dropped into a directory on the internal webserver of the DiskStation and loaded from there…..

Just need to see if there’s a way to generate a custom RSS feed to select nzbs that I want now :D

René on 2011-04-30 00:24:23
@Kevin: I might have found kind of a workaround for loading NZB files.

It appears that Downloadstation CLI is expecting an URL to load the NZB file from. That’s why the link appears as ‘broken’ or ‘unknown’ because downloadstation cannot find the path to the NZB file.

What I have done now, is move the NZB file to a directory on the internal webserver of the diskstation. I then use the URL to this file, eg. http://diskstation/nzb/filename.nzb, as a parameter for Downloadstation CLI. So the commandline becomes:

/opt/bin/downloadstation add http://diskstation/nzb/filename.nzb

I don’t delete the NZB file after adding it to downloadstation, because it would then again turn into a broken link.

Instead, I let the unpacker script remove the NZB file when the download is completed and unpacked in the cleanup portion of the script.

Marc on 2011-04-20 13:59:25
The script is a really great idea for the auto check / unrar. Do know how I could do to only check the real par2 files (not the .vol files) (so file.par2 is taken, but file.vol000+01.par2 is not taken?
It interesting because sometimes there are more than one file in a nzb so there is more than one par2 file that is needed (file.par2, test.par2 and Newtest.par2). There will be 3 rar and 3 different par2 set. So instead of removing the complete par2 it could be great to remove only the files that are being tested. For example when file.par2 is done, only files starting by file.vol*.par2 (and file.par2) are deleted Not the par2 for test.par2)

Thank you

Kevin on 2011-04-17 16:48:44
@ uninb: Did you add the root user? Did you restart cron after making changes?

@ René: Yeah, turns out CLI has no support for nzb and never will get that either :/
Still looking for a good solution for that..

René on 2011-04-07 19:36:46
Hello,

Thank you for the tutorial. I have only one issue where you may be able to help me with.

When I use: downloadstation add & lt;nzbfile& gt;, the download appears in the download queue, but a few seconds later it is marked as ‘Broken link’ and the download failes.

When I add the same nzbfile to the queue through the Synology Download Station appp it does work as expected.

Did you run into that problem? I use a DS211 by the way.

kind regards

uninb on 2011-03-12 20:20:47
Hi,
thank you for your great tutorial, but i cannot get working the crontab part.
I am somewhat of newbie to synology and bad using command line, but is it possible that there is a mistake in the crontab edit and /video/ should be /downloads/?
anyways my crontab refuses to to do what im trying, I have to do it manually…
any ideas?
best regards

Install SABnzbd, Sickbeard, Couchpotato on your Synology DSM 3 NAS

2011-01-28T13:00:00+01:00

The Synology ships with a Download Station but it’s not remotely as advanced as SABnzbd. What I mostly miss is automatic par & unpacking of it’s downloads. Here’s how to fix that.

Warning

This article assumes you’re somewhat skilled in Linux. By applying these suggestions you could seriously mess up your Disk Station.

I’m doing this on a DS411j running DSM 3.0. Your mileage may vary.

Maybe you don’t want SABnzbd after all

In a later article I demonstrate how much higher downloads speeds can be achieved by synology’s own nzbget, and a way that you can still have automatic par & unpacking of it’s downloads. I

First of all

Turn off the Download Station (Config screen)
Turn on SSH acccess (Terminal)
Login as root (same password as admin) via SSH and type: sh
install ipkg, which is like your Synology’s secret AppStore.

Install SABnzbd & the family

The following method installs SABnzbd as root. There are some pretty serious risks involved with that so you may want to change it to someone with less permissions.

ipkg install bzip2 par2cmdline unrar unzip zlib git
ipkg install py26-cheetah py26-openssl python26
install sabnzbdplus

cd /opt/local
[ -d sickbeard/.git ] || git clone git://github.com/midgetspy/Sick-Beard.git sickbeard
cd sickbeard
git pull

cd /opt/local
[ -d couchpotato/.git ] || git clone git://github.com/RuudBurger/CouchPotato.git couchpotato
cd couchpotato
git pull

Config

Most can be done via the webinterfaces, except for the standard port of CouchPotato, which conflicts with the Synology interface (:5000).

Open the file /opt/local/couchpotato/config.ini, look for the [global] header and change the port key to: 9300

[global]
port = 9300

Startup

Create a startup file:

cat << EOT > /usr/syno/etc/rc.d/S99SABnzbd.sh
#!/usr/bin/env sh
if [ "start" = "\$1" ]; then
  /opt/bin/python2.6 /opt/share/SABnzbd/SABnzbd.py -f /root/.sabnzbd/sabnzbd.ini -s 0.0.0.0:9200 -d
  /opt/bin/python2.6 /opt/local/couchpotato/CouchPotato.py --config=/opt/local/couchpotato/config.ini --datadir=/volume1 -d
  /opt/bin/python2.6 /opt/local/sickbeard/SickBeard.py --quiet --port 9400 --config /opt/local/sickbeard/config.ini --datadir=/volume1 --daemon
elif [ "stop" = "\$1" ]; then
  /usr/bin/killall -9 python2.6
elif [ "restart" = "\$1" ]; then
  \$0 stop
  \$0 start
elif [ "" = "\$1" ]; then
  echo "Start, stop or restart service? Use a parameter..."
fi
EOT

Make it executable:

chmod +x /usr/syno/etc/rc.d/S99SABnzbd.sh

And run it:

/usr/syno/etc/rc.d/S99SABnzbd.sh start

Connect to:

http://[your_nas_ip]:9200/ for SABnzbd
http://[your_nas_ip]:9300/ for CouchPotate
http://[your_nas_ip]:9400/ for Sickbeard

Next time your Synology boots, SABnzbd and it’s family will boot as well.

Imported comments

These were imported from my old blog. Please use disqus below for new comments

Tom on 2012-09-01 01:41:08
Does the startup file work with the latest DSM release? I have it set as executable but nothing happens?

Shane on 2012-04-09 21:53:49
This was very helpful to me.

Thank you!!

Andre on 2012-02-13 10:56:51
Hi,

Thanks for the article its a great source of information but I wonder will this allow the synology to run sabnzbd automatically even though the computer is powered off.

Im searching for some information which allows the synology to do this.

Thanks again great article.

Doodlydood on 2012-01-26 16:23:09
Great article. A note that one can add Zebulon’s repository and install these applications through the package manager interface on the Synology as well.

Derek on 2011-11-02 07:35:12
To fix the error
/opt/libexec/git-core/git-pull: line 236: tr: not found

install textutils

ipkg install textutils

Tom on 2011-10-28 16:00:49
for the tr errors, just update busybox. ipkg install busybox

hughmc on 2011-08-26 06:14:27
had to add /opt/bin to the PATH in /etc/rc

not the best solution but it’s working now.

hughmc on 2011-08-26 03:50:03
now i can’t get par2 to work hah! any tips?

& quot;No PAR2 program found, repairs not possible& quot;

par2cmdline is installed and in the path.

Hughmc on 2011-08-24 11:29:57
@KeNMaSTeR

You are missing the tr binary from coreutils.

ipkg install coreutils

Will fix it! Also i have noted that to get the config.ini files you will need to run sickbeard and couch potato at least once. they will generate a whole bunch of files.

so just type:

/opt/bin/python2.6 /opt/local/couchpotato/CouchPotato.py

& amp;

/opt/bin/python2.6 /opt/local/sickbeard/SickBeard.py

Jayberz on 2011-08-21 21:25:47
How to i link both sab and sick to do the post processing? i have read i need to connect yo the synology.. please help!!! everything else works a treat.. just want it to move the files one downloaded..

KeNMaSTeR on 2011-08-01 00:20:32
Hi i got these problem when i try the command git pull ?

KeNMaSTeR& gt; cd couchpotato/
KeNMaSTeR& gt; git pull
/opt/libexec/git-core/git-pull: line 235: tr: not found
Your configuration specifies to merge with the ref ‘master’
from the remote, but no such ref was fetched.

Any solution, i’m very newby in linux :_/
And because of this error i cant see the config.ini in both sickbeard and CP.

Thx for the help ;)

JR on 2011-07-02 00:17:09
Why not just use Merty sabnzbd package that contains some nice tools, and upgrade build in.
Available for all synology models.
works great, and unpacking works (might have to install extra deccompressing apps), but unrar and zip works in the box (as far i remember)
Speed have never been a concent for me, used this since my first synology nas (a 407e)

http://www.mertymade.com/syno/

arta on 2011-06-14 14:28:30
this site is full of energy
& lt;a href=& quot;http://www.artadesign.ir& quot;& gt;http://www.artadesign.ir& lt;/a& gt;
& lt;/div& gt;

Senne on 2011-05-27 15:34:31
@Ryan
You can always choose to install sab manually, using the src.tar.gz from sourceforge. Something like this:

& gt; shut down sab
& gt; make a dir (I put it in /opt/share/sab/)
& gt; wget http://sourceforge.net/projects/sabnzbdplus/files/sabnzbdplus/sabnzbd-0.6.2/SABnzbd-0.6.2-src.tar.gz/download
& gt; tar -xvzf SABnzbd-0.6.2-src.tar.gz

I had the 0.5.6 ipkg installed first, so I dunno if there are more things that need to be done to get it working without it. You need to change the path in the start up file, obviously.

Ryan on 2011-05-26 06:16:39
The sab package in ipkg looks pretty old. Can we easily upgrade to 0.6.2?

stiri iasi on 2011-05-25 14:21:07
Oh man, i would like to have this Linux skills like you :)

alex on 2011-05-22 06:53:06
Well shit, I wished I had known that these boxes weren’t powerful enough to download at my full speed. Oh well, at least everything is automated now and I don’t have to worry about setting a bandwidth cap. No way I’m giving up using Sabnzbd+.

Senne on 2011-05-20 13:36:52
I decided to use a more friendly way to shutdown each program and prevent accidental killing other python scripts that are running

With login/pw:



#!/bin/sh

if [ &
quot;start&
quot; = &
quot;\$1&
quot; ]; then

  /opt/bin/python2.6 /opt/share/SABnzbd/SABnzbd.py -f /root/.sabnzbd/sabnzbd.ini -s 0.0.0.0:9200 -d

  /opt/bin/python2.6 /opt/local/couchpotato/CouchPotato.py --config=/opt/local/couchpotato/config.ini --datadir=/volume1 -d

  /opt/bin/python2.6 /opt/local/sickbeard/SickBeard.py --quiet --port 9400 --config /opt/local/sickbeard/config.ini --datadir=/volume1 --daemon

elif [ &
quot;stop&
quot; = &
quot;\$1&
quot; ]; then

  #shutdown sab

  /opt/bin/wget -q --delete-after &
quot;http://localhost:9200/sabnzbd/api?mode=shutdown&
amp;ma_username=SABUSER&
amp;ma_password=SABPASSW&
amp;apikey=YOURAPIKEY&
quot; 

  sleep 6s



  #shutdown sick beard

  /opt/bin/wget -q --http-user=YOURSBLOGIN --http-password=YOURSBPASSW &
quot;http://localhost:9400/home/shutdown/&
quot; 

  sleep 6s



  #shutdown coachpotato 

  /opt/bin/wget -q --http-user=YOURCPLOGIN --http-password=YOURCPPASSW &
quot;http://localhost:9300/config/exit/&
quot;

  sleep 6s

  

elif [ &
quot;restart&
quot; = &
quot;\$1&
quot; ]; then

  \$0 stop

  \$0 start

elif [ &
quot;&
quot; = &
quot;\$1&
quot; ]; then

  echo &
quot;Start, stop or restart service? Use a parameter...&
quot;

fi

without login/pw:



#!/bin/sh

if [ &
quot;start&
quot; = &
quot;\$1&
quot; ]; then

  /opt/bin/python2.6 /opt/share/SABnzbd/SABnzbd.py -f /root/.sabnzbd/sabnzbd.ini -s 0.0.0.0:9200 -d

  /opt/bin/python2.6 /opt/local/couchpotato/CouchPotato.py --config=/opt/local/couchpotato/config.ini --datadir=/volume1 -d

  /opt/bin/python2.6 /opt/local/sickbeard/SickBeard.py --quiet --port 9400 --config /opt/local/sickbeard/config.ini --datadir=/volume1 --daemon

elif [ &
quot;stop&
quot; = &
quot;\$1&
quot; ]; then

  #shutdown sab

  /opt/bin/wget -q --delete-after &
quot;http://localhost:9200/sabnzbd/api?mode=shutdown&
amp;apikey=YOURAPIKEY&
quot; 

  sleep 6s



  #shutdown sick beard

  /opt/bin/wget -q &
quot;http://localhost:9400/home/shutdown/&
quot; 

  sleep 6s



  #shutdown coachpotato 

  /opt/bin/wget -q &
quot;http://localhost:9300/config/exit/&
quot;

  sleep 6s

  

elif [ &
quot;restart&
quot; = &
quot;\$1&
quot; ]; then

  \$0 stop

  \$0 start

elif [ &
quot;&
quot; = &
quot;\$1&
quot; ]; then

  echo &
quot;Start, stop or restart service? Use a parameter...&
quot;

fi

Gaz on 2011-05-13 23:14:48
Hi. Cool guide, but I’m stuck. Am trying to install on a DS1511+ but I cannot locate a py26-yenc package anywhere… seems py25-yenc is the latest. Does anyone know where I can get the py26-yenc? Or should I just go with python25 for everything (removing python26) ?
Thanks

Holly on 2011-04-23 19:38:49
Thanks Kevin and Simpic. Edited the script with VI and works now.

Loopt als een tiet

Jacob Korf on 2011-04-20 19:55:56
It worked!! I found out that I forgot to add the $1 to the elif [ & quot;& quot; = & quot;$1& quot;.
Now it’s working properly!
Thanks a lot!

Kevin on 2011-04-19 15:57:55
@ Jacob Korf: ls -al will do the trick (to view them) checkout the chmod manpage for differences between 700 and e.g. a+x. But I still think you haven’t copied your startup script correctly cause it clearly says it tries to execute something on line 12 that it has no rights to. Like it would try to execute the ini file or something.

Jacob Korf on 2011-04-19 14:24:57
@Senne: I had this incident before too:), my problem was that my shared folder was called /korf/ on the synology, and the shared folder in Putty (SSH client) was /usbshare1/

@Senne & amp; Kevin:
How is it possible to see the CHMOD rights?

when I execute as following the rights should be allright?:
chmod +x /usr/syno/etc/rc.d/S99SABnzbd.sh

Or is it better to use chmod 700 /usr/syno/etc/rc.d/S99SABnzbd.sh

Thanks in advance

Senne on 2011-04-18 19:25:16
The wrong free space was caused by a typo in my settings :) ‘/Volume1/Usenet/’ but it should be ‘/volume1/Usenet/’

@Jacob Geen idee waar het aan kan liggen, ben zelf ook een ramp met linux :) Staan je (lees/schrijf) chmod rechten goed?

Jacob Korf on 2011-04-18 17:33:25
Thanks Senne & amp; Kevin!

I created the /root/.sabnzbd directory, and was able to create the .ini by executing the & quot;/opt/bin/python2.5 /opt/share/SABnzbd/SABnzbd.py -f /root/.sabnzbd/sabnzbd.ini -s 0.0.0.0:9200 -d& quot; script manually.
But… now when i execute the S99SABnzbd.sh start, i get the following message:
/usr/syno/etc/rc.d/S99SABnzbd.sh: line 12: /root/.sabnzbd/sabnzbd.ini: Permission denied

Seems that I don’t have the proper permission?

Alvast bedankt, (thanks in advance)

Jacob Korf

Senne on 2011-04-18 14:53:32
For some reason the parameter placeholders get stripped when the script is saved. I used the vi editor to add them again as simpic stated.

open the file using



vi /usr/syno/etc/rc.d/S99SABnzbd.sh

press ‘a’ to go into edit mode and add the $1 on the correct places in the script. Exit the edit mode by pressing esc and type ‘ZZ’ to safe and close.

Now it should work just fine.

Now my problem: I installed under root, so sabnzbd states that the disk space is & quot;1.85 GB Free Space& quot; which is obviously wrong since I have 175 GB of free space on my NAS. Is there I way to fix this? Otherwise the diskspace email is not send when needed.

Kevin on 2011-04-18 14:11:55
@ Jacob Korf: May try creating the parent directory of that ini configuration file first:

mkdir -p /root/.sabnzbd

Not sure why it says line 12. My example has the reference on line 3 so it might still be you have an error in your file.

Als not sure why you’re using either Vim or nano, when you can just copy-paste the heredoc.

Jacob Korf on 2011-04-18 14:03:00
You were right! I just changed the parameters using NANO (same as VI), because the $1 wasn’t filled yet. But now i get the following error (but it does execute):
/usr/syno/etc/rc.d/S99SABnzbd.sh: line 12: /root/.sabnzbd/sabnzbd.ini: not found

Thanks a lot! but still i can’t start the SABnzbd server

Kevin on 2011-04-17 16:42:55
@ Holly & amp; Jacob Korf: I agree with simpic, please make sure you pasted the file 100% correctly

TorRK on 2011-04-16 12:16:13
*EDIT

Was able to run it with Holly’s line & quot;opt/bin/python2.5 /opt/share/SABnzbd/SABnzbd.py -f /root/.sabnzbd/sabnzbd.ini -s 0.0.0.0:9200 -d& quot; and get SABnzbd running. Had to run the start command again after completing setup…

TorRK on 2011-04-16 11:29:48
Hi there.

NOOB alert
I get a & quot;-ash: usr/syno/etc/rc.d/S99SABnzbd.sh: not found& quot; error when running the last command… something went wrong, but have no clue where. My guess is that the startup script wasn’t created…

Please help.

simpic on 2011-04-15 21:24:41
Jacob & amp; Holly…

Double check the script to make sure it has the $1 or $0 in the parameter placeholders.

You might need to vi the script to add these if you just copied and pasted the lines in from the web page.

Jacob Korf on 2011-04-15 13:29:36
I have the same problem as Holly. Exactly the same. Are we doing something wrong? I already searched on several websites for this, but didn’t find anything that I can use.

Holly on 2011-03-26 23:25:15
Hi,

The auto start script does not seem to execute here.

When i run

 /usr/syno/etc/rc.d/S99SABnzbd.sh start

i get

Start, stop or restart service? Use a parameter...

running it with

/opt/bin/python2.5 /opt/share/SABnzbd/SABnzbd.py -f /root/.sabnzbd/sabnzbd.ini  -s 0.0.0.0:9200 -d

to start the script manually works OK.

I’m a linux noob, so perhaps you guys have some ideas.

Kevin on 2011-03-10 14:29:39
@ Ovidio & amp; ege: Yeah SABnzb is really neat. However I found that my NAS wasn’t really powerful enough for it. May wanna checkout this new article on higher Synology download speeds: http://kevin.vanzonneveld.net/techblog/article/optimize_your_synology_for_downloading/

ege on 2011-02-21 22:45:31
worked great! thank you very much for the clear instructions^^ hoping to discovering more useful programs that would run on these incredible pieces of hardware…

Ovidiu on 2011-02-01 10:33:37
Great setup you have there :)

Thanks for sharing, i am also using SABnzbd on my linux machine, works great!