Software Engineer's Notes

Systemd service for IP aliasing

2017-05-13T17:09:00.001+04:00

In this post I'll show how to make a Systemd service for easy addition and removal of local IP addresses.

Problem

The service may be useful when working with emulators such as Android Emulator, when one needs to configure network communication between the emulated environment and a local server. For example, the problem with Android Emulator is that it is isolated from the host's private network, as it has its own virtualised address space.

But it somehow manages to get access to addresses of the interfaces configured on the host. And we are going to exploit this feature.

I have tried different ways, and eventually came to making a Systemd service.

NetworkManager

We can easily overcome the problem by creating a virtual (VLAN) connection for an existing wired connection with the help of a popular network manager such as NetworkManager. For example, NetworkManager's interface allows to create a VLAN connection based on real wired or wireless connections and assign different IP addresses to it. However, for some reason it didn't work for my Wi-Fi connection, and wired connection was not available. So I was forced to search for another solution.

Systemd service

The following solution is based on the ip tool from the iproute2 package.

Actually we can add address to an interface with a single command without the need for a Systemd service or something, e.g.:

ip address add 10.100.100.10/24 dev wlp3s0 label wlp3s0:1

ip address change 10.100.100.10/24 dev wlp3s0 label wlp3s0:1

where 10.100.100.10/24 specifies IP address and the network (24-bit mask, particularly), wlp3s0 points to existing wireless interface, and wlp3s0:1 is a tag for the address being added (will appear as separate interface in the output of ifconfig). Removal is just as easy:

ip address del 10.100.100.10/24 dev wlp3s0

But I don't like the idea of typing the commands over and over. So I created the following script and saved it as /usr/local/sbin/ip-alias:

#!/bin/bash -
# Configures extra IP addresses for a network interface.
#
# Configuration files are located at /usr/local/etc/ip-alias.IFNAME.conf
# where IFNAME is the network interface name.
#
# Ruslan Osmanov 2017

# Prints an error message
err()
{
  printf 'Error: %s\n' "$1" >&2
}

# Prints an warning message
warn()
{
  printf 'Warning: %s\n' "$1" >&2
}

# Prints an error message $1 (if any), then exits
die()
{
  [ $# -gt 0 ] && err "$1"
  exit 1
}

# Prints usage info and exits with status $1
usage()
{
  printf 'Usage: %s IFNAME [add|remove]\n' "$0"
  exit $1
}

# Adds all addresses specified in the configuration file for device $1
add()
{
  local device="$1"
  local address
  local new_device

  # Use the command group '(...)' in order to prevent modifying variables in
  # the global scope when 'source'ing the configuration file
  (
    load_config "$device" || die "failed to read configuration file"

    [ ! -v addresses -o ${#addresses[*]} -eq 0 ] && \
      die "required configuration 'addresses' is invalid/missing"

    for index in "${!addresses[@]}" ; do
      address="${addresses[$index]}"
      label="${device}:${index}"

      printf 'adding %s for %s as %s\n' "$address" "$device" "$label"
      # We might used ip address add instead, but it complicates consecutive calls to the script
      /bin/ip address change "$address" dev "$device" label "$label"
      [ $? -eq 0 ] || die "failed to add $address for $label"
    done
  )
}

# Removes address $1 for device $2
remove_address()
{
  local address="$1"
  local device="$2"

  printf 'removing address %s for device %s\n' "$address" "$device"
  /bin/ip address del "$address" dev "$device"

  if [ $? -ne 0 ]; then
    err "Failed to remove device $device"
    return 1
  fi
}

# Removes all addresses specified in the configuration file for device $1
remove()
{
  local device="$1"
  local address

  # Use the command group '(...)' in order to prevent modifying variables in
  # the global scope when 'source'ing the configuration file
  (
    load_config "$device" || die "failed to read configuration file"

    [ ! -v addresses -o ${#addresses[*]} -eq 0 ] && \
      die "required configuration 'addresses' is invalid/missing"

    for index in "${!addresses[@]}" ; do
      address="${addresses[$index]}"
      remove_address "$address" "$device"
    done
  )
}

# Loads configuration file for device $1
load_config()
{
  local filename="/usr/local/etc/ip-alias.${1}.conf"

  printf 'loading configuration from %s\n' "$filename"
  source "$filename"
}


if [ $# -lt 1 ]; then
  err "Invalid number of arguments"
  usage 1
fi

device="$1"
[ $# -gt 1 ] && action="$2"
: ${action:='add'}


case "$action" in
  add)
    add "$device"
    ;;
  remove)
    remove "$device"
    ;;
  *)
    die "action $action didn't match anything"
esac

Configuration file is just a shell script with declaration of addresses array variable. For example, /usr/local/etc/ip-alias.wlp3s0.conf file might look like the following:

addresses=("10.100.100.10/24" "10.100.101.10/24")

Having this script we don't need to remember the syntax of the ip command. All we need is to pass an interface name and an action name to the command:

/usr/local/sbin/ip-alias wlp3s0 add
/usr/local/sbin/ip-alias wlp3s0 remove

But we can do better with the help of an init system. For example, we can create /etc/systemd/system/ip-alias@.service Systemd service file with the following content:

[Unit]
Description=Setup IP aliases for %i network interface
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/sbin/ip-alias %i add
ExecStop=/usr/local/sbin/ip-alias %i remove

[Install]
WantedBy=multi-user.target

If you are using NetworkManager, you may want to add it as dependency:

[Unit]
Description=Setup IP aliases for %i network interface
Wants=NetworkManager.service NetworkManager-wait-online.service
After=network.target NetworkManager.service NetworkManager-wait-online.service
BindsTo=NetworkManager.service NetworkManager-wait-online.service
PartOf=NetworkManager.service NetworkManager-wait-online.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/sbin/ip-alias %i add
ExecStop=/usr/local/sbin/ip-alias %i remove
ExecReload=/usr/local/sbin/ip-alias %i remove
ExecReload=/usr/local/sbin/ip-alias %i add

[Install]
WantedBy=multi-user.target

# systemctl enable NetworkManager-wait-online.service

Then we can enable, start and stop the service for specific network interface(s), e.g.:

# systemctl enable ip-alias@wlp3s0
# systemctl start ip-alias@wlp3s0
# systemctl stop ip-alias@wlp3s0

With this service we don't need to remember anything, except the use of systemctl.

Sample ifconfig output when the service is running:

wlp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.0.102  netmask 255.255.255.0  broadcast 10.10.0.255
        inet6 fe80::2ac2:ddff:fec7:16d5  prefixlen 64  scopeid 0x20<link>
        ether 28:c2:dd:c7:16:d5  txqueuelen 1000  (Ethernet)
        RX packets 173600  bytes 166406125 (158.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 96319  bytes 13015624 (12.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlp3s0:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.100.100.10  netmask 255.255.255.0  broadcast 0.0.0.0
        ether 28:c2:dd:c7:16:d5  txqueuelen 1000  (Ethernet)

wlp3s0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.100.101.10  netmask 255.255.255.0  broadcast 0.0.0.0
        ether 28:c2:dd:c7:16:d5  txqueuelen 1000  (Ethernet)

Making charts from iostat output

2016-03-30T15:34:00.000+04:00

I've decided to make some charts from the output of the iostat command. Just for fun =)

The idea is simple: a script named `report.sh` runs periodically in background(via `cron`) collecting `iostat` data into $REPORT_DIR/$param/$device files, where

$REPORT_DIR is a directory with the reports;
$param is the `iostat` parameter name;
$device is a device name from the same `iostat` output.

When we need to update the charts, we call `plot.sh`.

Required components

Bash
iostat utility
Gnuplot

Scripts

util.sh

#!/bin/bash -

# $1 - Optional error message.
function die
{
  [ $# -gt 0 ] && echo >&2 $1
  exit 1
}

function warning
{
  echo >&2 "Warning: $1"
}


function getSafeFilename
{
  r=${1//\//_}
  echo ${r/\%/}
}

#vim: ts=2 sts=2 sw=2 et

report.sh

#!/bin/bash -
# Generates iostat data files for processing with the GNU plot utility.

HEADER=
TIME_FORMAT="%Y/%m/%d/%H:%M"
TIME=$(date "+$TIME_FORMAT")
DIR=$(cd $(dirname "$0"); pwd)
REPORT_DIR="${DIR}/r"


#####################################################################
# Functions

source $DIR/util.sh


# $1 - exit code
function usage
{
  echo "
Generates iostat data files for processing with the GNU plot utility(plot.sh).

Usage: $0 OPTIONS

OPTIONS:
-h, --help            Display help message
-t, --time-format     Time format. Default: ${TIME_FORMAT}.
-i, --report-dir      Directory with reports. Default: ${REPORT_DIR}.
"
  exit $1
}

#####################################################################
# Parsing CLI options

OPTS=$(getopt -o t:i:h -l time-format:,report-dir:,help -- "$@")
[ $? -eq 0 ] || usage 1
eval set -- "$OPTS"

while true
do
  case "$1" in
    -t|--time-format)
      TIME_FORMAT="$2"
      shift 2;;
    -i|--report-dir)
      REPORT_DIR="$2"
      shift 2;;
    -h|--help)
      usage 0
      shift;;
    --)
      shift
      break;;
    ?)
      usage 1;;
    *)
      die "Internal error: failed to parse CLI args"
  esac
done

echo TIME_FORMAT: $TIME_FORMAT REPORT_DIR: $REPORT_DIR

#####################################################################

mkdir -p $REPORT_DIR
[ $? -eq 0 ] || die "Report directory '$REPORT_DIR' is inaccessible"

iostat -dpxk | while read -r line
do
  if [[ -z $HEADER && $line == Device* ]]; then
    HEADER=( $line )
  elif [[ -n $HEADER ]]; then
    [[ -z $line ]] && continue

    columns=( $line )

    [[ ${#HEADER[@]} != ${#columns[@]} ]] && continue


    i=1
    for h in ${HEADER[@]:1}
    do
      param_report_dir="${REPORT_DIR}/"$(getSafeFilename $h)
      mkdir -p $param_report_dir
      [ $? -eq 0 ] || die "Parameter directory '$param_report_dir' is inaccessible"

      # $REPORT_DIR/param/device
      echo $TIME$'\t'${columns[$i]} >> $param_report_dir'/'${columns[0]}
      (( ++i ))
    done
  fi
done

echo 'Done'

plot.sh

#!/bin/bash -
# Creates visual representation of data files generated by report.sh

TIME_FORMAT="%Y/%m/%d/%H:%M"
# Report files location $REPORT_DIR/param/device
DIR=$(cd $(dirname "$0"); pwd)
REPORT_DIR="${DIR}/r"
OUTPUT_DIR="${DIR}/o"
XRANGE=
PLOT_TERMINAL="png"

#####################################################################
# Functions

source $DIR/util.sh

function getPlots
{
  param_path="$1"

  find $param_path/ -maxdepth 1 -mindepth 1 -type f | while read -r device_path; do
    # $device_path = /*/*/.../param/device
    device=$(basename $device_path)

    printf "'%s' using 1:2 t '%s' with lp pt 5, " \
      "$device_path" $device
    printf "'%s' using 1:2 notitle with impulses, " \
      "$device_path"
  done
}


# Returns default value for GNU plot xrange
function getXRange
{
  y=$(date +%Y)
  m=$(date +%m)

  (( m -= 1 ))

  if [ $m -lt 1 ]; then
    m=1
    (( y -= 1 ))
  fi

  d=$(date "+$TIME_FORMAT")

  echo "[ '$y/$m/1/00:00' : '$d' ]"
}


# $1 - exit code
function usage
{
  echo "
Creates charts from files generated by report.sh

Usage: $0 OPTIONS

OPTIONS:
-h, --help            Display help message
-x, --xrange          GNU Plot xrange value. Default: $(getXRange).
-t, --time-format     Time format. Default: ${TIME_FORMAT}.
-i, --report-dir      Directory with reports. Default: ${REPORT_DIR}.
-o, --output-dir      Output directory. Default: ${OUTPUT_DIR}.
-f, --term            GNU Plot terminal. Default: ${PLOT_TERMINAL}. See 'gnuplot set terminal'.
"
  exit $1
}

#####################################################################
# Parsing CLI options

OPTS=$(getopt -o x:t:i:o:f:h -l xrange:,time-format:,report-dir:,output-dir:,term:,help -- "$@")
[ $? -eq 0 ] || usage 1
eval set -- "$OPTS"

while true
do
  case "$1" in
    -x|--xrange)
        XRANGE="$2"
        shift 2;;
    -t|--time-format)
      TIME_FORMAT="$2"
      shift 2;;
    -i|--report-dir)
      REPORT_DIR="$2"
      shift 2;;
    -o|--output-dir)
      OUTPUT_DIR="$2"
      shift 2;;
    -f|--term)
      PLOT_TERMINAL="$2"
      shift 2;;
    -h|--help)
      usage 0
      shift;;
    --)
      shift
      break;;
    ?)
      usage 1;;
    *)
      die "Internal error: failed to parse CLI args"
  esac
done

echo XRANGE: $XRANGE TIME_FORMAT: $TIME_FORMAT OUTPUT_DIR: $OUTPUT_DIR REPORT_DIR: $REPORT_DIR

#####################################################################

[ -d $REPORT_DIR ] || die "$REPORT_DIR doesn't exist"

find $REPORT_DIR/ -maxdepth 1 -mindepth 1 -type d | while read -r param_path
do
  # $param_path = /*/*/.../param
  param=$(basename $param_path)
  plots=$(getPlots "$param_path")
  plots=${plots/%, /}

  mkdir -p $OUTPUT_DIR
  [ $? -eq 0 ] || die "Output directory '$OUTPUT_DIR' is inaccessible"

  outfile="${OUTPUT_DIR}/${param}.${PLOT_TERMINAL}"

  [[ -z $XRANGE ]] && XRANGE="$(getXRange)"

  gnuplot <<EOS
TIME_FORMAT = "$TIME_FORMAT"
set terminal $PLOT_TERMINAL
set grid nopolar
set xtics rotate
set xlabel "Date/Time"
set ylabel "Value"
set yrange [ 0 : ]
set format x TIME_FORMAT timedate
set timefmt TIME_FORMAT
set xdata time
set xrange $XRANGE
set output "$outfile"

plot $plots
EOS
done

echo 'Done'

Sample chart

Arch Linux ARMv8 VM on Gentoo AMD64

2016-03-23T12:15:00.000+04:00

The post is about configuring Arch Linux ARMv8 guest virtual machine(VM) on Gentoo AMD64 host using QEMU.

VirtualBox

I'm used to making VMs in VirtualBox. It can handle a large variety of disk image formats. We can easily convert the RAW image to VDI:

$ VBoxManage convertfromraw disk.img arch.vdi --format VDI 
Converting from raw image file="disk.img" to file="arch.vdi"...
Creating dynamic image with size 4294967296 bytes (4096MB)...

However, at the time of writing I found no way to run the AARCH64 guest on AMD64 host. The boot process stuck on UEFI interactive shell which failed to detect the drive contents. All my attempts to boot from the shell had failed. I had also tried to fix it through VBoxInternal2/EfiBootArgs without success. Finally decided to give QEMU a try.

QEMU

QEMU proved to be more flexible. I had managed to boot the system by specifying initial ramdisk, firmware, kernel and kernel arguments. The following steps describe the whole process.

Install required packages

Prepare USE-flags:

$ sudo euse -p app-emulation/qemu -E qemu_softmmu_targets_aarch64 qemu_user_targets_aarch64 spice

Install QEMU as described in Gentoo Wiki.

Download firmware

Download QEMU_EFI.fd from snapshots.linaro.org.

Create disk image

$ qemu-img create -f raw disk.img 4G

Create partition table

$ parted disk.img mklabel msdos

Mount the disk image as a loopback device

$ sudo losetup -f disk.img
losetup -a
/dev/loop0: []: (/home/ruslan/aarch64-qemu/disk.img)

Create a partition

$ sudo fdisk /dev/loop0
fdisk> n
(choose defaults)
fdisk> a
fdisk> w

Detach the loop device

$ sudo losetup -d /dev/loop0

Create a loop device for the new partition

$ sudo losetup -f -P disk.img

$ losetup -a
/dev/loop0: []: (/home/ruslan/aarch64-qemu/disk.img)

Format the partition to ext4

$ sudo mke2fs -t ext4 /dev/loop0p1

Mount the new partition

$ mkdir -p mnt
$ sudo mount /dev/loop0p1 mnt

Extract the Arch files

Extract the Arch ARM Linux files from archive according to instructions:

$ sudo su -
# bsdtar -xpf ArchLinuxARM-aarch64-latest.tar.gz -C mnt
# exit

Boot in fallback mode

The system(d) failed to detect the hard drive. So I had to fix initial ramdisk hooks order through fallback mode.

$ filename="disk.img"

$ qemu-system-aarch64 -m 2048 -cpu cortex-a57 \
  -smp 1 -M virt -bios QEMU_EFI.fd -serial stdio \
  -drive if=none,file=$filename,id=hd0  \
  -device virtio-blk-device,drive=hd0 \
  -kernel mnt/boot/Image -initrd mnt/boot/initramfs-linux-fallback.img \
  -append "root=/dev/vda1"

Within the guest system change HOOKS in /etc/mkinitcpio.conf from

HOOKS="base udev block autodetect modconf filesystems keyboard fsck"

to:

HOOKS="base udev autodetect modconf block filesystems keyboard fsck"

Remount / for read-write:

# mount -o remount,rw /

Then regenerate initramfs and shutdown the guest:

# mkinitcpio -p /etc/mkinitcpio.d/linux-aarch64.preset
# shutdown now

Remount the loop device

$ sudo umount /dev/loop0p1
$ sudo losetup -d /dev/loop0
$ sudo losetup -f -P disk.img
$ sudo mount /dev/loop0p1 mnt

Finally

Run the machine in normal mode:

$ filename="disk.img"
$ qemu-system-aarch64 \
  -m 512 \
  -cpu cortex-a57 -M virt \
  -drive if=none,format=raw,file=$filename,id=hd0  \
  -device virtio-blk-device,drive=hd0 \
  -smp 1 \
  -bios QEMU_EFI.fd \
  -serial stdio \
  -kernel mnt/boot/Image \
  -initrd mnt/boot/initramfs-linux.img \
  -append "root=/dev/vda1 rootfstype=ext4 rw" \
  -netdev user,id=unet -device virtio-net-device,netdev=unet

The only drawback is that ICMP packets(ping) will not work in this mode. The network settings can be tweaked, though.

Helper scripts

To make further usage easier I've created scripts to mount and unmount the disk image, and to launch the VM.

util.sh

#!/bin/bash -

function usage()
{
  echo -e "Usage:\n$0 disk-image mount-point"
}

function error()
{
  [ $# -gt 0 ] && echo >&2 "!! $1"
}

function die()
{
  error $1
  exit 1
}

function notice()
{
  echo "* $1"
}

function checkimg()
{
  [ -e "$1" ] || die "$1 doesn't exist"
}

function checkmnt()
{
  [ -e "$1" ] || die "$1 doesn't exist"
}

mount.sh

#!/bin/bash -

source util.sh

if [ ! $# -gt 1 ]; then
  error 'Missing required arg(s)'
  usage
  die
fi

imgfile="$1"
mntdir="$2"

checkimg $imgfile
checkmnt $mntdir

free_loop_part=$(losetup --show --find --partscan "$imgfile")"p1"
[ $? -eq 0 ] || die 'Failed to create loop device'
notice "Created loop partition $free_loop_part"

mount -o loop,rw "$free_loop_part" "$mntdir"
[ $? -eq 0 ] || die "Failed to mount $imgfile to $mntdir"
notice "Associated $free_loop_part with $imgfile and mounted to $mntdir"

unmount.sh

#!/bin/bash -
# Unmounts and detaches loop devices associated with a disk image

source util.sh

if [ ! $# -gt 1 ]; then
  error 'Missing required arg(s)'
  usage
  die
fi

imgfile="$1"
mntdir="$2"

checkimg $imgfile
checkmnt $mntdir

notice "Unmounting $mntdir"
while true
do
  umount --detach-loop "$mntdir" >/dev/null 2>&1
  [ $? -eq 0 ] || break
done

losetup -j "$imgfile" -l --raw -O NAME -n | while read -r ld
do
  losetup -d "$ld"
  [ $? -eq 0 ] && notice "Detached loop device $ld"
done

run.sh


#!/bin/bash -

source util.sh

[ $# -gt 0 ] && filename="$1"
[ $# -gt 1 ] && mntdir="$2"

: ${filename:="disk.img"}
: ${mntdir:="mnt"}

checkimg $filename
checkmnt $mntdir

notice "Launching Arch aarch64, img:$filename mntdir $mntdir"

qemu-system-aarch64 \
  -m 512 \
  -cpu cortex-a57 -M virt \
  -drive if=none,format=raw,file=$filename,id=hd0  \
  -device virtio-blk-device,drive=hd0 \
  -smp 1 \
  -bios QEMU_EFI.fd \
  -serial stdio \
  -kernel $mntdir/boot/Image \
  -initrd $mntdir/boot/initramfs-linux.img \
  -append "root=/dev/vda1 rootfstype=ext4 rw" \
  -netdev user,id=unet -device virtio-net-device,netdev=unet

Usage

With the help of these scripts we can start and stop the VM easily:

$ sudo ./mount.sh disk.img mnt
* Created loop partition /dev/loop2p1
* Associated /dev/loop2p1 with disk.img and mounted to mnt
$ df
...
/dev/loop3       4061888   1128524   2707316  30% /home/ruslan/aarch64-qemu/mnt
$ losetup -l
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE
...
/dev/loop2         0      0         0  0 /home/ruslan/aarch64-qemu/disk.img
/dev/loop3         0      0         1  0 /dev/loop2p1

$ ./run.sh
* Launching Arch aarch64, img:disk.img mntdir mnt
...
Arch Linux 4.5.0-1-ARCH (ttyAMA0)

alarm login: root
Password: 
Last login: Tue Mar 22 17:23:48 on ttyAMA0
[root@alarm ~]# arch
-bash: arch: command not found
[root@alarm ~]# uname -a
Linux alarm 4.5.0-1-ARCH #1 SMP Mon Mar 14 18:47:45 MDT 2016 aarch64 GNU/Linux
[root@alarm ~]# shutdown now
...
$ sudo ./unmount.sh disk.img mnt
* Unmounting mnt
* Detached loop device /dev/loop2

PostgreSQL function for parsing JSON into Sphinx keywords

2015-11-02T17:22:00.001+04:00

This post describes how to write a basic PostgreSQL function for parsing a JSON into Sphinx keywords(space-separated values of the JSON object).

Requirements

Perl core
Perl JSON module
PostgreSQL built with JSON support

Sample Problem

We have workers table:

CREATE TABLE workers(
  id integer,
  props text
);

where props keeps arbitrary worker properties in JSON format.

Let's insert a worker:

INSERT INTO workers VALUES(1, '{"x":"x test", "d":104, "o":{"o1":"o1 name"}, "a":["a1", "a2"]}');

We want to make the worker properties searchable through Sphinx. So we add corresponding PostgreSQL source and an index in our sphinx.conf:

source src_workers: pgsql
{
  sql_query_pre = set work_mem = '100MB'
  sql_query = select id, props from workers
  sql_field_string = props
}

index idx_workers
{
  source = src_testing
  path = /home/ruslan/sphinxsearch/idx_workers
  min_prefix_len = 3
  docinfo = extern
}

With the current sql_query the props field will be cluttered with JSON keys and specific characters. We'll write a PostgreSQL function which will give us a single line of space-separated values:

$ createlang plperlu t # < "t" is a database name
$ psql -d t
create or replace function json2sphkw(text,text) returns text as $$
use JSON qw( decode_json );

my ($in) = @_;
$_SHARED{read_sub} = sub {
  my ($in) = @_[0];

  if (ref($in) eq "HASH") {
    my @res, $read_sub = $_SHARED{read_sub};
    foreach $v (values %$in) {
      push @res, &$read_sub($v);
    }
    return join(' ', @res);
  } elsif (ref($in) eq "ARRAY") {
    my @res, $read_sub = $_SHARED{read_sub};
    foreach $v (values @$in) { push @res, &$read_sub($v); }
    return join(' ', @res);
  }
  return $in;
};

my $h = decode_json $in;
my $read_sub = $_SHARED{read_sub};

return &$read_sub($h);
$$ LANGUAGE plperlu;

Now we can use it:

SELECT json2sphkw(props, ' ') FROM workers WHERE id = 1;
        json2sphkw
--------------------------
 o1 name a1 a2 x test 104
(1 row)

And the sphinx.conf can be changed as follows:

@@ -1,7 +1,7 @@
 source src_workers: pgsql
 {
   sql_query_pre = set work_mem = '100MB'
-  sql_query = select id, props from workers
+  sql_query = select id, json2sphkw(props, ' ') from workers
   sql_field_string = props
 }

References

PostgreSQL: PL/Perl - Perl Procedural Language

Randomly changing the desktop wallpaper via crontab

2014-12-18T14:04:00.000+04:00

Here is a simple way to organize a random desktop wallpaper by means of crontab.

We'll use feh application here. Let's put the following code into ~/bin/wallpaper-change file:

#!/bin/bash -
# Pick and set a random wallpaper

: ${DISPLAY:=":0"}

if [ $# -lt 1 ]; then
  echo >&2 "No source directory specified"
  exit 1
fi

dir="$1"
if [[ ! -d "$dir" ]]; then
  echo >&2 "Directory $dir doesn't exist"
  exit 1
fi

find $dir/ -maxdepth 1 -type f | sort -R | tail -1 | while read f
do
  feh --bg-fill "$f" &
done

(Don't forget to make it executable.)

Now we can adjust the crontab, e.g.:

DISPLAY=:0
@hourly ~/bin/wallpaper-change ~/Pictures/wallpapers

Note, setting the DISPLAY environment variable is crucial.

Changing keyboard layout in system console

2014-11-26T09:47:00.000+04:00

The post is about configuring a Unicode keyboard layout for the system console on Gentoo. However, the steps should be similar on other systems.

Enable Unicode in /etc/rc.conf:

unicode="yes"

Install a font for UTF-8 consoles:

# emerge -av media-fonts/terminus-font

Change /etc/conf.d/consolefont:

consolefont=LatArCyrHeb-16
# consolefont="ter-v16b"
consoletranslation=""

Make sure that /etc/init.d/consolefont is enabled:

# rc-update add consolefont boot

Set default console keymap in /etc/conf.d/keymaps:

keymap="us"

Add /etc/init.d/keymaps to the boot run level:

# rc-update add keymaps boot

Configure the shell to send switch-to-unicode escape sequence at each login. For Bash add the following to your ~/.bash_profile:

if test -t 1 -a -t 2 ; then
        echo -n -e '\033%G'
fi

To make the switch to Unicode global for all users the above-mentioned snippet should be added to /etc/profile.

Reboot the system. Now Unicode should be available on all system console logins.

To load different keyboard layout use the loadkeys command (which is shipped with sys-apps/kbd package):

# loadkeys /usr/share/keymaps/{YOUR_ARCH}/{PATH_TO_KEYMAP}

For instance, to load Russian Qwerty layout for i386 (will work on amd64):

# loadkeys /usr/share/keymaps/i386/qwerty/ru.map.gz

One might want to make a shortcut for this. Use Ctrl-Shift shortcut for switching between the keymaps.

Gentoo configure error: C compiler cannot create executables

2014-06-28T16:40:00.001+04:00

If configure phase failes with this message, then most likely the dev-libs/mpc package is removed. I removed it accidentally. Afterwards I was unable to emerge anything. Here is a simple solution for the issue.

Download stage3 tar ball from a Gentoo mirror, unpack it and copy usr/lib/libmpc.so* over /usr/lib. Then run ldconfig and try to build something.

Hopefully, the post will help another gentoer like me who will also wonder what's going wrong with the C compiler.

How to capture audio stream using Pulseaudio and VLC

2014-06-21T09:57:00.000+04:00

Some Internet radio stations offer URLs for streaming with popular software, some don't. I'll show a way to capture currently runnig audio stream to file using Pulseaudio and VLC.

Step 1. Pick an output

$ pactl list short | grep RUNNING | awk '{print $2}'
alsa_output.pci-0000_00_1b.0.analog-stereo
alsa_output.pci-0000_00_1b.0.analog-stereo.monitor

(The list of devices also be obtained via VLC GUI: View - Playlist - Audio capture.) Let's pick the second one.

Step 2. Write a Bash script

In the following script replace the value of pulseaudio_stream variable with 'pulse://YOUR_OUTPUT', where YOUR_OUTPUT is the output chosen in the first step.

#!/bin/bash -
set -o nounset

pulseaudio_stream='pulse://alsa_output.pci-0000_00_1b.0.analog-stereo.monitor'

usage() {
  IFS= read -r errstr <<-errstr
  Captures current pulseaudio stream $pulseaudio_stream and saves it to a file
  Usage: $0 <output-path>
  Example: $0 ~/Music/radio.mp3
errstr

  echo >&2 "$errstr"
  exit 1
}

if [[ $# < 1 ]]; then
  usage
fi

output="$1"

nvlc "$pulseaudio_stream" ':sout=#transcode{vcodec=none,acodec=mp3,ab=128,channels=2,samplerate=44100}:duplicate{dst=std{access=file,mux=raw,dst='$output'}}'

Save it to ~/scripts/bash/rec-audio.sh and make it executable.

Step 3. Use it

~/scripts/bash/rec-audio.sh ~/Music/radio.mp3

Currently running audio will be recorded to the file until you quit VLC.

Slim and Systemd

2014-03-31T17:16:00.001+04:00

I just want to drop in a warning from Arch linux documentation:

Warning: Currently SLiM is not fully compatible with Systemd, resulting in
various problems on a second login. See Display manager#Incompatibility with
systemd for an outline of these problems.

I had really experienced such kind of issues after switching from OpenRC to Systemd! Text artifacts in GTK-based applications, SSH keyring issues, system stall on reboot or shutdown is not full list of bugs I got with Slim running under Systemd. All of a sudden I see the Arch docs. Replacing Slim with LightDM fixed all of these issues at a time!

An alternative to gksudo/kdesudo

2014-03-22T12:35:00.001+04:00

The gksudo and kdesudo commands are okay, if we whether work in GNOME or KDE environments, or are satisfied with the number of required dependencies. However, these pacakges are pretty cumbersome, if we want just a lightweight GUI version of sudo, especially on non-GTK/non-KDE environments like Xfce. In this post I'll show a simple GUI front-end I use on my Xfce setup. Steps:

Install x11-ssh-askpass or similar.
Create ~/bin/xsudo executable with the following contents:
```
#!/bin/bash -
sudo -A $@
  
```
Set the $SUDO_ASKPASS environment variable in some initialization script (e.g. ~/.xinitrc, ~/.bash_profile, ~/.bashrc etc.):
```
export SUDO_ASKPASS="/usr/bin/x11-ssh-askpass"
  
```
Alternatively, put the following into /etc/sudo.conf file:
```
Path askpass /usr/bin/x11-ssh-askpass
  
```
(Make sure to adjust the path.)

Now we can use xsudo command just as if it were gksudo or kdesudo.

Extracting unescaped part from Nginx location in UTF-8

2014-02-21T11:07:00.000+04:00

The Nginx location matches unescaped URI. However, to capture a Unicode part of URI is somewhat tricky. First of all, we have to prepend our regular expression with (*UTF8), e.g:

location ~* "(*UTF8)^/img/(.+)\.(jpg|jpeg|gif|png))$" {
  # ...
}

Now we can use ordinary $1, $2, ..., $N variables for corresponding regular expression groups. In most cases they work well. For instance, we can successfully pass them to a proxy URI. The parts are getting url-encoded, and everything works fine..., except some cases when we use them as file/directory names, e.g.:

proxy_store $my_variable;

The problem is that a UTF-8 character can be converted to 3..12 characters! A 3-character sequence in Chinese like '艾弗吉' gets converted to 27-character long string '%E8%89%BE%E5%BC%97%E5%90%89'. Obviously, we can reach Ext4's maximum filename length limit of 255 characters with just 28 Chinese symbols. The following captures basename as unescaped UTF-8 string:

location ~* "(*UTF8)^/img/(?<basename>(.+)\.(jpg|jpeg|gif|png)))$" {
  # $basename is unescaped string in Unicode
}

Now we can pass $basename to directives accepting file/directory names. Hope this helps someone.

Fixing PHP start-up error: «Unable to load dynamic library»

2013-07-25T17:50:00.000+04:00

Suppose you have a PHP extension compiled as ext.so shared library. Let's assume that it depends on sockets extension(which is compiled as sockets.so library file). It means that PHP should load sockets.so before ext.so. Sometimes it doesn't, however. In this case running any PHP script you'd catch a startup error similar to the following:

/usr/lib64/php/modules/event.so: undefined symbol: php_sockets_le_socket in Unknown on line 0

Current version of PHP(at the time of writing 5.5.0 was the latest stable version) loads extensions from php.ini downwards line by line. So the order of "extensions" in php.ini specifies the order of loading of the shared libraries. Most GNU/Linux distributions use the --with-config-file-scan-dir= configure option, which specifies a directory where PHP looks for additional configuration. Usually this directory contains a file per extension. These files are loaded in alpha-numeric order. It means that our ext.ini will be loaded before sockets.ini, as well as ext.so will be loaded before sockets.so, which is not what we wanted!

People in the PHP Group promised to work on this issue. To fix this issue pro tem just rename the ini files in the config file scan directory. Thus our ext.ini has to be renamed to, z-ext.ini, for instance.

How to factory reset Samsung Galaxy Note 2

2013-06-01T23:41:00.000+04:00

I spent a day scraping my head in attempts to reset my phone to factory settings. Common instructions didn't help. So I'd like to share a very simple way to reset a Samsung Galaxy Note 2. It should work for other similar headsets, too.

The problem

Samsung Galaxy Note 2 GT-N7100 headset has been locked. Neither SIM card, nor Internet are available. There is no way to access menu/options. The only available options are: emergency calls and an invitation to enter Google account.

So the only way to factory reset is to use combinations mentioned in all these instructions like "hold power and volume up buttons until you see menu with 'factory reset' option" etc. But the boot-up menu has just "self-test" options without a tip on factory reset.

The fix

If you have no 'factory reset' option on the boot-up menu, and fully aware that the reset will remove all user data(contacts, custom apps, settings etc.); if the only thing you want is a working phone, then do the following:

Download and install fastboot utility. The simplest way is to download Zip file from this forum thread.

Move to the directory with fastboot and run:

$ ./fastboot -w reboot
< waiting for device >

Hold down the MENU button
Hold down the POWER button for 10..20 seconds until you see the Samsung logo with a red text: "fastboot mode". After you entered fastboot mode, release the phone buttons.

Hold down the VOLUME UP key while connecting the phone with computer using USB cable. Fastboot utility should detect the phone now. It resets the phone and reboots it:

$ ./fastboot -w reboot
< waiting for device >

erasing 'userdata'...
OKAY [  1.610s]
erasing 'cache'...
OKAY [  0.605s]
rebooting...

finished. total time: 2.219s

I did it on Gentoo x86_64. I'm not sure if my udev rules helped. To be on the safe side, I'd put the following to /etc/udev/rules.d/51-android.rules file:

SUBSYSTEM=="usb", ATTR{idVendor}=="1782", MODE="0666", GROUP="plugdev" 
SUBSYSTEM=="usb", ATTR{idVendor}=="22b8", MODE="0666", GROUP="plugdev"
SUBSYSTEM=="usb", ATTR{idVendor}=="0bb4", MODE="0666", GROUP="plugdev"
SUBSYSTEM=="usb", ATTR{idVendor}=="04e8", MODE="0666", GROUP="plugdev"
SUBSYSTEM=="usb", ATTR{idVendor}=="18d1", MODE="0666", GROUP="plugdev"

Don't forget to restart the udev daemon after creating/modifying the rules.

Done! :-) I hope this post saves someone's time.

Example of XML parsing in Perl

2013-05-28T21:53:00.000+04:00

The following is just a one-shot Perl script I used to convert one's XML to a CSV file. For me(Perl beginner) it took a while to figure out a fast and simple way to parse XML in Perl. So it might be useful for someone else.

#!/usr/bin/perl

use strict; use warnings;
use XML::Twig;
use Text::CSV_XS;

my @headers = (
    'Picture',
    'StateID',
    'ItemTypeID',
    'Name',
    'Descript',
    'Article',
    'Price',
    'Qty',
    'ItemAvailabilityID',
    'SerialNumber',
    'Weight',
);

my %twig_handlers = (
    'Item' => \&handle_item,
);
foreach (@headers) {
    $twig_handlers{$_} = \&handle_tag;
}

my $twig = XML::Twig->new(twig_handlers => \%twig_handlers);
my $csv = Text::CSV_XS->new({sep_char => ';'});
my @columns;
my $have_picture = 0;

print join(';', @headers), "\n";
$twig->parsefile(shift @ARGV);

sub handle_tag() {
       if ($_->gi eq 'Picture') {
           $have_picture = 1;
           push @columns, $_->{'att'}->{'url'};
       } else {
           if ($_->gi eq 'StateID' && !$have_picture) {
               push @columns, "";
           }
           push @columns, $_->trimmed_text;
    }
}

sub handle_item() {
    $csv->print(\*STDOUT, \@columns); 

    # handlers are called when elements closed
    print "\n";

    # reset
    @columns = ();
    $have_picture = 0;
}

The XML was similar to the following:

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE Items SYSTEM "http://domain.com/xml/MerchantItems.dtd">
<Items>
  <Item>
    <Images>
      <Picture url="http://domain.com/775/7757d248aeb782e916a9a51670171ec1.jpeg"/>
    </Images>
    <StateID>1735000</StateID>
    <ItemTypeID>29490209</ItemTypeID>
    <Name><![CDATA[Lamp 2541/8]]></Name>
    <Descript><![CDATA[ ... HTML code ... ]]></Descript>
    <Article>2541/8</Article>
    <Price>7267.50</Price>
    <Qty>100</Qty>
    <ItemAvailabilityID>3719000</ItemAvailabilityID>
    <SerialNumber>92645</SerialNumber>
    <Weight>0</Weight>
  </Item>
  <!-- ... -->
  </Item>
</Items>

Fixing Sphinx wordforms

2013-05-26T23:49:00.000+04:00

Sphinx wordforms feature has a drawback: the stemmer skips "destination" words. In this post I'll show a quick and dirty way to fix a large wordforms file.

Let's assume we have a wordforms file with the following contents:

noisy > noise
noisyyy > noise

Any document having word noisy will be found by searching for noisyyy. And vice versa. Bad news is that the word noise is out of stemming. It means that if we try to search for noise, we'll find only those documents that contain exactly "noise".

Sphinx 2.1.1-beta introduced indextool --morph INDEXNAME option, which applies morphology to the characters given on the standard input, e.g.:

echo 'confidence
> presence' | ~/bin/indextool --morph s3_all
Sphinx 2.1.1-beta (rel21-r3701)
Copyright (c) 2001-2013, Andrew Aksyonoff
Copyright (c) 2008-2013, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file './sphinx.conf'...
dumping stemmed results...
confid presenc

Yes, the output might be more friendly(parsable). It even doesn't support --quiet option at the time. Now we are going to replace the destination words with their stemmed variants. Here is a simple one-shot Perl script for this.

#!/usr/bin/perl
# File: stem-wordforms.pl
use strict;
use warnings;
use Fcntl; # sysopen
use POSIX 'tmpnam';
use IO::Seekable; # seek

sub usage () {
    print "Usage: $0 <filename> [<out_filename>]
<filename>         Path to SphinxSearch wordforms file
<out_filename>   Path to output filename 
";
}

if ($#ARGV < 0) {
    usage;
    exit;
}
my $in_filename  = $ARGV[0];
my $out_filename = $#ARGV > 0 ? $ARGV[1] : $in_filename .".out";
my $tmp_filename = tmpnam();
my $line;
my $n_in_lines = 0;
my $n_out_lines = 0;

sysopen FH_IN, $in_filename, O_RDONLY
    or die("Failed to open `$in_filename`");
sysopen FH_RES, $out_filename, O_CREAT | O_TRUNC | O_WRONLY, 0640
    or die("Failed to open `$out_filename`");
# Open pipe for indextool. We'll write destination wordforms here line by line.
# By means of `tr` command we place a word per line.
open my $fh_indextool, "|-",
"~/bin/indextool -c sphinx.conf --morph s3_all | tr ' ' '\n' > $tmp_filename"
    or (print "Failed\n" and die());

print ">> Stemming to `$tmp_filename`...";
while (<FH_IN>) {
    ++$n_in_lines;

    # Get destination word
    s/^[^\>]+\> // ;
    # Patch: for some reason Sphinx 2.1.1-beta strips `ё` character even if it
    # is declared in charset_table
    s/ё/e/g;
    $line = $_;

    print $fh_indextool $line;
}
close $fh_indextool;
print " OK\n";

print ">> Joining temporary results with the source wordforms... ";
my $got_results = 0;
my $line2;
sysopen(FH_TMP, $tmp_filename, O_RDWR, 0640)
    or die("Failed to open `$tmp_filename`");
seek FH_IN, 0, SEEK_SET;
while (<FH_TMP>) {
    $line = $_;

    if ($got_results == 0 && m/^results\.\.\./) {
        print "Got results... ";
        $got_results = 1;
        next;
    }
    $got_results or next;

    last if !($line2 = <FH_IN>);
    $line2 =~ s/[^\>]+$//;
    print FH_RES $line2 , " ", $line;
    ++$n_out_lines;
}
unlink $tmp_filename or print "Failed to unlink $tmp_filename\n";

$n_in_lines != $n_out_lines and die("* Failed! Number of lines of `$in_filename` and `$out_filename` doesn't match!\n");

print "OK\nDone\n";

close FH_IN;
close FH_RES;
close FH_TMP;

Not very nice script, but hopefully will help someone. You may want to adapt it for your configuration. Usage:

$ ./stem-wordforms.pl wordforms_myspell_ru_RU_UTF-8.txt out
>> Stemming to `/tmp/file7AawtG`... OK
>> Joining temporary results with the source wordforms... Got results... OK
Done

Accessing server vhosts on the VirtualBox guests

2012-12-10T22:57:00.002+04:00

This post will probably help you, if you are searching for way to access an Apache/Nginx vhost on VirtualBox's guest. Let's assume we have vhost mega8080 listening on port 8080. The following makes it available for Windows XP guest.

Edit vhost configuration

Vhost configuration file like /etc/apache2/sites-available/localhost should listen to port 8080:

ServerName server.megahost
NameVirtualHost 127.0.0.3:8080


DocumentRoot /var/www/megahost8080
ServerName megahost8080

Register IP

In our sample /etc/hosts should have the following like:

127.0.0.3	megahost8080

Relay port 8080 connections to IP 127.0.0.3

# iptables -t nat -I PREROUTING -p tcp --dport 8080 -j DNAT --to-destination 127.0.0.3
# iptables -t nat -A OUTPUT -p tcp --dport 8080 -j DNAT --to-destination 127.0.0.3

We need this because in Windows XP ipconfig outputs:

IPv4Address: 10.0.2.15
Default Gateway: 10.0.2.2

We need port forwarding because it won't work if configure our vhost to listen to 10.0.2.15 directly.

Simplest way to configure network for guest is to keep it's default NAT configuration :)

Network adapter: 
 - attached to: NAT 
 - type: PCnet-FAST III (NAT)

Forward port 8080 connections through TCP in VBox

$ VBoxManage setextradata "winxp" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/http/Protocol" TCP
$ VBoxManage setextradata "winxp" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/http/GuestPort" 8080
$ VBoxManage setextradata "winxp" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/http/HostPort" 8080

Or, in new versions, just set it up via GUI: Settings - Network - Port Forwarding.

Bind guest IP to the hostname

File C:\windows\system32\drivers\etc\hosts(or similar) should contain line like the following:

10.0.2.2	megahost8080

Test it

Go to http://megahost8080:8080

Upgrading to Newest Version of an App Which Still Is Not Available In the Gentoo Portage

2012-11-07T15:02:00.000+04:00

Sometimes you need new version of an app, but it is not yet availlable via Gentoo's portage(or overlays). If you want to keep things organized, you have to make package yourself. In some distros it is done easily with `checkinstall` utility, which helps to produce Debian, RPM or Slackware packages. In Gentoo we have make a ebuild ourselves.

However, ebuild tool and existing ebuilds in PORTDIR(usually /usr/portage) make it pretty easy too. I'll show it with example.

Let's say, I have to upgrade app-misc/sphinx to version 2.0.6. But even after emerge --sync the latest ebuild available is /usr/portage/app-misc/sphinx/sphinx-2.0.5.ebuild. For this, we'll do the following: 1) Add our custom portage directory 2) Copy and modify sphinx-2.0.5.ebuild in our brand new portage dir 3) emerge the new version, pulling the code from the official application maintainer's site

Add new portage overlay dir:

# mkdir -p /var/lib/layman/my_ebuilds
# vim /etc/portage/make.conf
# ...
PORTDIR_OVERLAY="${PORTDIR_OVERLAY} /var/lib/layman/my_ebuilds"
# ...

Copy existing ebuild:

# cp /usr/portage/app-misc/sphinx/sphinx-2.0.5.ebuild /var/lib/layman/my_ebuilds/app-misc/sphinx/sphinx-2.0.6.ebuild
# vim /var/lib/layman/my_ebuilds/app-misc/sphinx/sphinx-2.0.6.ebuild
... change version, URIs etc. if needed (in my case it detects version automatically)

Build ebuild manifest(this will fetch file from the Web):

# cd /var/lib/layman/my_ebuilds/app-misc/sphinx
# ebuild sphinx-2.0.6.ebuild manifest

Finally upgrade to the new version:

# emerge -Duva app-misc/sphinx

That's it! Enjoy:)

References

1. Gentoo WIKI: Writing Ebuilds

Speeding Up Copy Process Between Two Hard Drives

2012-11-04T22:24:00.000+04:00

cp command is not suitable for large data transfers between two hard drives. It makes about 10..100KB/sec. Obviously, it will take 29 hours to copy 10GB with 100KB/sec.

There are many ways to speed up copying process -- using USB, Firewire, Ethernet etc. But if everything is done on a local machine, rsync is very effective and simple to use. For example, the following shows how to copy files over the virtual loopback device:

$ mkdir -p /mnt/hdd
$ sudo mount /dev/sdbX /mnt/hdd
$ rsync -az user@localhost:/some/folder/* /mnt/dest-hdd/

a("archive mode") option takes care of permissions, ownership, group, symlinks and so. z instructs rsync to use compression.

/var/run/mysqld is removed after each reboot?

2012-05-10T23:32:00.001+04:00

After recent upgrade on my Gentoo box I've accidently got /var/run/mysqld directory removed. The socket path pointed there, and, of course, mysqld denied to launch. The same thing happened some time ago on Ubuntu and openSUSE. I've no clue what makes it(mysqld? rc script?) remove the directory. But I've got at least two simple fixes for that and wanna share it in this post.

Fix #1

Just change path to the socket in /etc/mysql/my.cnf and get rid of these upgrade issues:

socket = /tmp/mysqld.sock

I have it in client and mysqld sections.

Fix #2

Change the runlevel script. In Gentoo it looks like the following:

checkconfig() {
    if [ ! -d /var/run/mysqld ] ; then
        checkpath -d -m 755 -o mysql:mysql /var/run/mysqld
    fi
    if [ -z "${MYSQL_PERMS_OK}" ] && [ "$(stat -c %a /var/run/mysqld)" != "755" ] ; then
        ewarn "mysqld run dir is not world readable, you should reset the perms:"
        ewarn "chmod 755 /var/run/mysqld"
        ewarn "To disable this warning, set 'MYSQL_PERMS_OK' in /etc/conf.d/mysql"
    fi
}
start() {
    checkconfig
    # ...
}

You should have got it. checkconfig is called on at the very beginning of /etc/init.d/mysql start handler.

That's it. I hope this saves someone's time.

Installing ImageMagick with DejaVu support

2011-12-15T08:53:00.000+04:00

I've had troubles with DejaVu fonts in ImageMagick. Almost everything with -font 'DejaVu-Sans-Condensed' failed. Well, it was searching for the font, consuming much of CPU resources, and used default one afterwards. In this post I'll show how to compile ImageMagick from source with DejaVu support. Althouth it may seem trivial, the process may sometimes be tricky.

First off, install FreeType dev package. It should be called libfreetype*-dev or so. For instance, in Debian:

$ sudo apt-get install libfreetype6-dev

Make sure you have DejaVu font installed. Package name should be named somewhat like 'ttf-dejavu'. In Debian you can install it as follows:

$ sudo apt-get install ttf-dejavu-core ttf-dejavu ttf-dejavu-extra

Download the source

Download tarball from http://www.imagemagick.org/script/download.php and unpack it.

Find font directories

Figure out the main directory with fonts and a directory with DejaVu fonts, and specify them in configuration:

$ ./configure --with-freetype=yes --with-djvu=yes \
--with-fontpath=/usr/share/fonts/truetype \
--with-dejavu-font-dir=/usr/share/fonts/truetype/ttf-dejavu

Compile and install:

$ make
$ sudo make install 
# or 
# sudo make checkinstall
# whatever preferred/appropriate.

ldconfig

I had to make the follownig for my /usr/local prefix:

$ sudo ldconfig

Likely, you should do it also.

Verify

Verify that ImageMagick recognizes DejaVu:

$ identify -list font | grep -i dejavu

This should output a list with installed DejaVu fonts.

I hope this will be helpful for someone.

eio - a new PHP extension for asynchronous POSIX I/O

2011-10-11T11:17:00.002+04:00

Eio is a new PECL extension for asynchronous POSIX I/O.

Eio provides interface for the libeio library. It means that each POSIX call runs in a separate thread. It is important to aware that the order of ungrouped requests is not precisely known. However, libeio, as well, as eio, supports complex call creation to accumulate a set of regular eio_* requests in a single group request. This allows to create a group request ex. to open, read and close a file.

Currently Windows platforms are not supported.

Any extension-specific suggestions are welcome :)

Creating new Vim filetype for server logs

2011-09-01T17:59:00.000+04:00

In this post I'll show how to create a simple Vim syntax script to highlight source of common error and access logs.

First make sure you have the following folders and files:

~/.vimrc		- the main config
~/.vim/ftdetect		- directory with filetype scripts
~/.vim/syntax		- directory with specific syntax scripts

If something missing, you can copy it from $VIMRUNTIME directory. Find block starting with

if has("autocmd")

and add the following line there:

au BufRead,BufNewFile *.log set filetype=error_log

Create filetype script in ~/.vim/ftdetect/error_log.vim:

if did_filetype() " filetype already set.. finish " ..don't do these checks endif

au BufRead,BufNewFile *.error.log set filetype=error_log au BufRead,BufNewFile *.access.log set filetype=error_log

Finally you need syntax script itself, ~/.vim/syntax/error_log.vim:

" Vim syntax file
" Language:    error_log
" Maintainer:  Ruslan Osmanov <rrosmanov at gmail dot com>
" Last Change: Thu Sep  1 15:30:52 MSD 2011
" Version:     1.0

if version < 600
  syntax clear
elseif exists("b:current_syntax")
  finish
endif

" Always ignore case
syn case ignore

" General keywords which don't fall into other categories
syn keyword error_logKeyword 			error warning notice 

" Special values
syn keyword error_logSpecial         	referer client

" Strings (single- and double-quote)
syn region error_logString           start=+"+  skip=+\\\\\|\\"+  end=+"+
syn region error_logString           start=+'+  skip=+\\\\\|\\'+  end=+'+

" Numbers and hexidecimal values
syn match error_logNumber            "-\=\<[0-9]*\>"
syn match error_logNumber            "-\=\<[0-9]*\.[0-9]*\>"
syn match error_logNumber            "-\=\<[0-9][0-9]*e[+-]\=[0-9]*\>"
syn match error_logNumber            "-\=\<[0-9]*\.[0-9]*e[+-]\=[0-9]*\>"
syn match error_logNumber            "\<0x[abcdefABCDEF0-9]*\>"

" IP addresses
syn match error_logIP            	"-\=\<[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*\>"

syn region error_logOperator         start="referrer\:" end="" contains=ALL

" URI
syn match error_logURI            	"http\:\/\/[^ ]*"

" Date 
syn match error_logDate 			"\[[0-9]*\/[a-zA-Z]\{3\}\/\d\{4\}:\d\{2\}:\d\{2\}:\d\{2\} +\d\{4\}\]"
syn match error_logDate				"\[[a-zA-Z]\{3\} [a-zA-Z]\{3\} \d\{2\} \d\{2\}:\d\{2\}:\d\{2\} \d\{4\}\]"

" Domain
syn match error_logDomain            "\(www\.\)\?[a-z0-9\-\_]*\.\(ru\|com\|by\|uz\|ua\|kz\|su\|net\|org\)"

" Define the default highlighting.
" For version 5.7 and earlier: only when not done already
" For version 5.8 and later: only when an item doesn't have highlighting yet
if version >= 508 || !exists("did_error_log_syn_inits")
	if version < 508
		let did_error_log_syn_inits = 1
		command -nargs=+ HiLink hi link <args>
	else
		command -nargs=+ HiLink hi def link <args>
	endif

	HiLink error_logKeyword         Statement
	HiLink error_logIP				Type
	HiLink error_logSpecial  	    Special
	HiLink error_logString          String
	HiLink error_logNumber          Number
	HiLink error_logDate			Comment 
	HiLink error_logURI				Underlined 
	HiLink error_logDomain			Underlined 

	delcommand HiLink
endif

let b:current_syntax = "error_log"
" vim:sw=4:

Colorized svn status

2011-08-03T16:29:00.002+04:00

I post some shell scripts here to output svn status command output colorized.

Bash Script

#!/bin/bash - 

set -o nounset                              # Treat unset variables as an error

txtblk='\e[0;30m' # Black - Regular
txtred='\e[0;31m' # Red
txtgrn='\e[0;32m' # Green
txtylw='\e[0;33m' # Yellow
txtblu='\e[0;34m' # Blue
txtpur='\e[0;35m' # Purple
txtcyn='\e[0;36m' # Cyan
txtwht='\e[0;37m' # White
bldblk='\e[1;30m' # Black - Bold
bldred='\e[1;31m' # Red
bldgrn='\e[1;32m' # Green
bldylw='\e[1;33m' # Yellow
bldblu='\e[1;34m' # Blue
bldpur='\e[1;35m' # Purple
bldcyn='\e[1;36m' # Cyan
bldwht='\e[1;37m' # White
unkblk='\e[4;30m' # Black - Underline
undred='\e[4;31m' # Red
undgrn='\e[4;32m' # Green
undylw='\e[4;33m' # Yellow
undblu='\e[4;34m' # Blue
undpur='\e[4;35m' # Purple
undcyn='\e[4;36m' # Cyan
undwht='\e[4;37m' # White
bakblk='\e[40m'   # Black - Background
bakred='\e[41m'   # Red
badgrn='\e[42m'   # Green
bakylw='\e[43m'   # Yellow
bakblu='\e[44m'   # Blue
bakpur='\e[45m'   # Purple
bakcyn='\e[46m'   # Cyan
bakwht='\e[47m'   # White
txtrst='\e[0m'    # Text Reset

RESULT="`svn st`"
echo "$RESULT" | while read LINE
do
 case "${LINE:0:1}" in
  'M')
   echo -e "$txtylw$LINE$txtrst"
   ;;
  'X')
   echo -e "$bakpur$LINE$txtrst"
   ;;
  '?')
   echo -e "$txtgrn$LINE$txtrst"
   ;;
  'D')
   echo -e "$bakylw$LINE$txtrst"
   ;;
  'I')
   echo -e "$txtblu$LINE$txtrst"
   ;;
  *)
   echo -e "$LINE";
   ;;
 esac
done

AWK Script

#!/usr/bin/awk -f
BEGIN {
  cmd = "svn st";
  while (cmd | getline) {
    char = substr($1, 0, 1);
    if (char == "M") {
      print "\033[36m(", $1, ")    ", $2, $3, "\033[0m"; 
    } else if (char == "X") {  
    print "\033[35m(", $1, ")    ", $2, "\033[0m"; 
    } else if (char  == "A") {  
      print "\033[32m(", $1, ")    ",  $2, $3, "\033[0m"; 
    } else if (char == "?") { 
      print "\033[1;31m(", $1, ")    ", $2, $3, "\033[0m"; 
    } else if (length($1) == 1 ) {
      print $1,"      " $2; 
    } else if ($1 == "---") { # changelists etc.
      print "\033[0;40m", $1, $2, $3, $4, "\033[0m"; 
    } else {
      print $0;
    }
  }
}

Mapping special characters in Sphinx configuration

2011-07-29T17:16:00.006+04:00

Sphinx sometimes assumes some characters as word separators. For instance, letters 'e' and 'ё' are kinda similar in Russian. Some print issues even replace the latter with the former. However, Sphinx assumes 'ё' a word separator. To prevent it, one should search for Unicode code points and append mapping to charset_table index config:

charset_type  = utf-8
charset_table  = 0..9, A..Z->a..z, _, a..z, U+A8->U+E5, U+B8->U+E5, U+410..U+42F->U+430..U+44F, U+430..U+44F, \
U+0451->U+0435

Here U+0451->U+0435 maps 'ё'(U+0451) to 'e'(U+0435). The Unicode code points could be found here.

References

How to create a Sphinx wordform dictionary

2011-06-16T10:22:00.001+04:00

To build a wordform file for Sphinx you may use some kind of spellchecker dictionary like myspell, ispell, pspell, aspell. Let's make a wordform file for Russian language from myspell-russian package in openSUSE.

To install myspell-russian:

$ sudo zypper in myspell-russian
$ rpm -ql myspell-russian
/usr/share/doc/packages/myspell-russian
/usr/share/doc/packages/myspell-russian/descr_en.txt
/usr/share/doc/packages/myspell-russian/descr_ru.txt
/usr/share/doc/packages/myspell-russian/description.xml
/usr/share/doc/packages/myspell-russian/dictionaries.xcu
/usr/share/doc/packages/myspell-russian/icon.png
/usr/share/doc/packages/myspell-russian/licence.txt
/usr/share/myspell
/usr/share/myspell/ru_RU.aff
/usr/share/myspell/ru_RU.dic

Build wordforms:

$ spelldump /usr/share/myspell/ru_RU.dic /usr/share/myspell/ru_RU.aff wordforms_myspell_ru_RU.txt
spelldump, an ispell dictionary dumper

Loading dictionary...
Loading affix file...
Using MySpell affix file format
Dictionary words processed: 146265

Detect encoding:

$ cat wordforms_myspell_ru_RU.txt | enca -L ru 
KOI8-R Cyrillic
  LF line terminators

Конвертируем в UTF-8:

$ iconv -f KOI8-R -t UTF-8 -o wordforms_myspell_ru_RU_UTF8.txt wordforms_myspell_ru_RU.txt

Finally, in sphinx.conf set:

wordforms     = /path/to/wordforms_ru_RU_UTF8.txt

Software Engineer's Notes

Systemd service for IP aliasing

Problem

NetworkManager

Systemd service

Making charts from iostat output

Required components

Scripts

Sample chart

Arch Linux ARMv8 VM on Gentoo AMD64

VirtualBox

QEMU

Install required packages

Download firmware

Create disk image

Create partition table

Mount the disk image as a loopback device

Create a partition

Detach the loop device

Create a loop device for the new partition

Format the partition to ext4

Mount the new partition

Extract the Arch files

Boot in fallback mode

Remount the loop device

Finally

Helper scripts

util.sh

mount.sh

unmount.sh

run.sh

Usage

PostgreSQL function for parsing JSON into Sphinx keywords

Requirements

Sample Problem

References

Randomly changing the desktop wallpaper via crontab

Changing keyboard layout in system console

Gentoo configure error: C compiler cannot create executables

How to capture audio stream using Pulseaudio and VLC

Step 1. Pick an output

Step 2. Write a Bash script

Step 3. Use it

Slim and Systemd

An alternative to gksudo/kdesudo

Extracting unescaped part from Nginx location in UTF-8

Fixing PHP start-up error: «Unable to load dynamic library»

How to factory reset Samsung Galaxy Note 2

The problem

The fix

Example of XML parsing in Perl

Fixing Sphinx wordforms

Accessing server vhosts on the VirtualBox guests

Edit vhost configuration

Register IP

Relay port 8080 connections to IP 127.0.0.3

Forward port 8080 connections through TCP in VBox

Bind guest IP to the hostname

Test it

Upgrading to Newest Version of an App Which Still Is Not Available In the Gentoo Portage

References

Speeding Up Copy Process Between Two Hard Drives

/var/run/mysqld is removed after each reboot?

Fix #1

Fix #2

Installing ImageMagick with DejaVu support

Download the source

Find font directories

Compile and install:

ldconfig

Verify

eio - a new PHP extension for asynchronous POSIX I/O

Creating new Vim filetype for server logs

Colorized svn status

Bash Script

AWK Script

Mapping special characters in Sphinx configuration

References

How to create a Sphinx wordform dictionary

See also