gosub

git remote repository url using ssh on nonstandard port

Fri, 04 Dec 2015 00:00:00 +0100

git remote repository url using ssh on nonstandard port

why would one need this

i run gogs on a server (dockerized of course ;) in my home network to share repositories between my laptop and desktop. since i still want to be able to ssh in to the server on port 22, i mapped the gogs containers port 22 to 10022 on the docker host.

To make this work, the repo url that gogs tells you to use, has to be tweaked a little bit.

this url works

For example if this is the url gogs tells you to use:

git@thegithost:lumue/myrepo.git

then this would be the repo url to use for gogs listening on port 10022:

ssh://git@thegithost:10022/lumue/myrepo.git

There might be a way to get gogs to display the correct url to begin with, but for the time being i now have at least this cheatsheet and wont have to go hunting on the internet everytime i clone a new repo :)

Installing kodi on a raspberry pi

Wed, 02 Dec 2015 00:00:00 +0100

Installing kodi on a raspberry pi

Howto install an image of the osmc kodi distribution on a raspberry pi using linux.

download image

Download the osmc distribution image for raspberry pi and unzip it:

curl http://download.osmc.tv/installers/diskimages/OSMC_TGT_rbp1_20151129.img.gz
gunzip OSMC_TGT_vero1_20151129.img.gz

The exact filename for the current image can be found here

copy image to sd card

This is the short version. More detailed instructions on howto create a bootable sd card from a disk image can be found here. Find out the device name of your sdcard (usually /dev/mmcblk0) and unmount it (if mounted)

sudo umount /dev/mmcblk0p1

Copy the downloaded image to your sdcard and sync it.

sudo dd bs=4M if=OSMC_TGT_rbp1_20151129.img of=/dev/mmcblk0 && sync

done

insert the sdcard in your raspberry, and boot into osmc.

noticed by google

Sun, 22 Nov 2015 00:00:00 +0100

Noticed by Google

Yay! The google bot came by for the first time! Over the last days, i did the following things to lure it here:

added a sitemap
added rss feed
registered the site with google analytics
registered the rss feed with feedburner
some modifications to page headers

I will likely never find out, what send google’s spider my way. But obviously some of the above worked:

Using a Logitech G13 gaming keypad on debian sid

Tue, 03 Nov 2015 00:00:00 +0100

Using a Logitech G13 gaming keypad on debian sid

This is a short HOWTO on setting up g13d as a systemd controlled service on debian linux.
If you just want to use your g13 and dont care about how it is done, fair enough: I compiled what is described here into a bunch of shell scripts ready to checkout and execute for you from here.

checkout and build

go to https://github.com/ecraven/g13, checkout their repo and follow the installation instructions.

    git clone https://github.com/ecraven/g13.git
 cd g13
 make

installation

copy the g13d binary to /usr/bin/g13d or make it otherwise accesible on your path.

sudo cp g13d /usr/bin/g13d && sudo chmod a+x /usr/bin/g13d

create a start script g13d-run to init the bindings and start g13d and put it in /usr/bin

sudo touch /usr/bin/g13d-run && sudo chmod a+x/usr/bing13d-run

open the file and paste the following script:

#!/usr/bin/env bash
G13D_WORK="/tmp/g13d"

if [ -d G13D_WORK ]
then
    echo "using existing $G13D_WORK as working directory ..."
else
    echo "creating $G13D_WORK as working directory ..."
    mkdir "$G13D_WORK"
fi

cd "$G13D_WORK"
cat /etc/g13d.d/default.bind >> /tmp/g13-0
/usr/bin/g13d

setup systemd service

create a start script for the service named g13d-service and save it in /etc/init.d

sudo touch /etc/init.d/g13d-service && sudo chmod a+x /etc/init.d/g13d-service

open the file and paste the following script:

#!/bin/bash
G13D_HOME="/usr/lib/g13d"
PID=""

function get_pid {
   PID=`pidof g13d`
}

function stop {
   get_pid
   if [ -z $PID ]; then
      echo "server is not running."
      exit 1
   else
      echo -n "Stopping server.."
      kill -9 $PID
      sleep 1
      echo ".. Done."
   fi
}


function start {
   get_pid
   if [ -z $PID ]; then
      echo  "Starting server.."
        (/usr/bin/g13d-run &> /dev/null  &) &
      get_pid
      echo "Done. PID=$PID"
   else
      echo "server is already running, PID=$PID"
   fi
}

function restart {
   echo  "Restarting server.."
   get_pid
   if [ -z $PID ]; then
      start
   else
      stop
      sleep 5
      start
   fi
}


function status {
   get_pid
   if [ -z  $PID ]; then
      echo "Server is not running."
      exit 1
   else
      echo "Server is running, PID=$PID"
   fi
}

case "$1" in
   start)
      start
   ;;
   stop)
      stop
   ;;
   restart)
      restart
   ;;
   status)
      status
   ;;
   *)
      echo "Usage: $0 {start|stop|restart|status}"
esac

create a systemd .service script for the service named g13d-service.service and save it in /lib/systemd/g13d-service.service

sudo touch /lib/systemd/g13d-service.service && sudo chmod a+x /lib/systemd/g13d-service.service

open the file and paste the following script:

[Unit]
Description=Start g13d-service

[Service]
WorkingDirectory=/tmp/g13d
Type=forking
ExecStart=/bin/bash /etc/init.d/g13d-service start
KillMode=process

[Install]
WantedBy=multi-user.target

setup default bindings

create a directory /etc/g13d.d and put your bindings in a file named /etc/g13d.d/default.bind

my bindings file looks like this:

bind G1 KEY_F1
bind G2 KEY_F2
bind G3 KEY_F3
bind G4 KEY_F4
bind G5 KEY_F5
bind G6 KEY_F6
bind G7 KEY_F7
bind G8 KEY_1 
bind G9 KEY_2
bind G10 KEY_3
bind G11 KEY_4
bind G12 KEY_5
bind G13 KEY_6
bind G14 KEY_7
bind G15 KEY_8
bind G16 KEY_9
bind G17 KEY_0
bind G18 KEY_F8
bind G19 KEY_F9
bind G20 KEY_F10
bind G21 KEY_F11
bind G22 KEY_TAB
bind DOWN KEY_LEFTALT

Now execute

sudo systemctl daemon-reload
sudo systemctl enable g13d-service.service

to install the service, and we are done!

Running

Now you can control g13d like any other service on your box. just issue

sudo systemctl status g13d-service

to start the service.

Deploying jee webapplication and rdbms with docker

Fri, 30 Oct 2015 00:00:00 +0100

Deploying jee webapplication and rdbms with docker

I plan to migrate a testing environment comprised of a few jee webapps running on glassfish and connecting to a firebird database to a set of docker containers. I tried this for one of the application, and here is how it can be done:

What will be deployed

Deployment consists of

war packaged java webapp
glassfish jee container
firebird database server
datacontainer with prefilled firebird db volume (for testing)
datacontainer with prefilled application working directory (for testing)

Application state in a container

To setup testing and demo environments it will be useful to have a set of known application states versioned and instantly deployable. The docker solution to this seems to be setting up a datacontainer as described here. Since the application i try to deploy here stores its state in the database and in a working directory, we want to fit both in the same container image.

It is recommended to use the same base image for data containers that is used in the other containers which make up your deployment. This will save you some diskspace, but other than that its irrelevant what you use.

The Dockerfile for jacobalberty/firebird:2.5-ss reveals the base image used:

FROM debian:jessie

We want our image to expose two volumes. one for database file(s), and one for the work directory. The Dockerfile below can be used to build this image.

It shall be named appdata, for it stores appdata! to build and run this image execute

docker build -t <your_repo>/appdata .
docker -d --name appdata <your_repo>/appdata

check the result by running docker ps. if all went well you should see output like this:

CONTAINER ID        IMAGE                      COMMAND             CREATED             STATUS              PORTS               NAMES
55c6523c4307        18384fe4a822....e558e295   "/bin/bash"         21 minutes ago      Up 18 minutes                           appdata

which means our datacontainer is up and running.

Databaseserver

getting the image

We will use the firebird 2.5 superserver docker image from jacobalberty/firebird.

docker pull jacobalberty/firebird:2.5-ss

running the container

For a production deployment we would map the exposed database volume to a path in the hosts filesystem (in this case /data/firebird/databases). The image also exposes firebirds default port 3050. We will not expose it on the host machine, but instead build a connection between webapp and database images later.

docker run -d --name firebird -v /data/firebird/databases:/databases jacobalberty/firebird:2.5-ss

But what we want to do is use our datacontainer “appdata” to store the database files. To tell docker about it we have to use the –volumes-from flag when starting the firebird image:

docker run -d --name firebird --volumes-from appdata jacobalberty/firebird:2.5-ss

you can verify that the firebird container uses volumes from appdata by attaching to both containers and writing to the /databases directory. if everything is working as planned, you should see the same content in both containers.

Application and Applicationserver

creating the image

To setup the image for glassfish4 plus webapp, we use a custom shell script and a slightly modified version of the official glassfish Dockerfile from docker hub. both are displayed below.

Lets see whats happening:

The build.sh script

template domain is packed into a .jar file
docker build is executed

In the Dockerfile

glassfish is downloaded and installed (same as in the official image)
.jar file with template domain is added to the image
password.txt file containing the admin password is added
glassfish domain named webapp is created from the template-domain
the domain is started to execute the enable-security-admin command and stopped again

Posting to slack from shell

Mon, 01 Jun 2015 00:00:00 +0200

Posting to slack from shell (using the webapi)

We are trying out slack as a communication tool at work, so i decided to give its webapi a little try.
What i want to do, is send a message from the commandline into slack.

Getting api access to slack

Access to slacks web api is authorized by an access token. so we need to obtaion one. to do this, log in to slack and point the browser to slack webapi. scroll down, and hit the issue token button.

you can always go back to the same url to view your token.

Using the slack api to post

incoming webhooks

To post a message to a slack channel, we need to setup an incoming webhook.

creating the webhook

In order to do so, navigate to new incoming webhook, select in which channel you want the messages to appear, and click the big green button which says “Add incoming WebHooks Integration”. The selected channel is just a default and can be overriden when sending the actual message.

The URL of the created webhook is shown on th resulting page:

testing the webhook

Let’s see if this works:

curl -X POST --data-urlencode 'payload={"token":"<your-token-here>","channel": "#korona-resource","username": "lumue","text": "This is posted to #korona-resource from lm","icon_emoji": ":ghost:"}' https://<your-incoming-webhook-url-here>

if everything works as it should, you should see this testmessage displayed in the selected channel:

the username does not have to exist in your slack team.

roll everything into a friendly bash script

because i know i will never be able to memorize this call properly we will put this into a shell script i will call “slack_send.sh”. also, it will look cleaner this way when used from the commit-hook script later on.

#!/bin/bash
#these have to be setup via slack
api_token="<your-token-here>"
url="https://<your-webhook-url-here>"

#commandline parameter
message=$1
channel=$2
username=$3

#assemble json payload
jsonstring="'payload={\"token\":\"$api_token\",\"channel\":\"$channel\",\"username\":\"$username\",\"text\":\"$message\",\"icon_emoji\":\":ghost:\"}'"

#call curl
curl -X POST --data-urlencode $jsonstring $url

posting svn commit notifications to slack

Mon, 01 Jun 2015 00:00:00 +0200

Posting svn commit notifications to slack

We are trying out slack as a communication tool at work, and i thought it would be nice to post notifications about commits to our svn repository to the channel i set up for project stuff.
So the plan is to setup a post-commit hook on the svn host which calls into slacks webapi.

Posting to the slack webapi

I have already described howto post to slack channels from the shell in another post.
To make it easier for to send the commit information to slack, i modified the script from the above mentioned post to accept a svn repo location and a revision number (which is what we get passed into the svn hook handler) instead of the message. Also the user and channel name are fix.

Now it looks like this:

#!/bin/bash

#these have to be obtained from slack
api_token="your slack webapi token"
url="https://your webhook url"

username="svnbot"
channel="#korona-resource"

#commandline parameter
repo=$1
revision=$2

#build message
message=`svn log --revision $revision $repo`
message=`echo $message | iconv -f iso8859-1 -t ascii//translit`
message=`echo $message | sed "s/'/\\\\\'/g"`
message=${message//--/}
message=${message//|/\\n}

#build json
jsonstring="{\"token\":\"$api_token\",\"channel\":\"$channel\",\"username\":\"$username\",\"text\":\"$message\",\"icon_emoji\":\":ghost:\"}"
echo $jsonstring > /tmp/slack_git_msg

#call curl
curl -i -X POST -H 'Accept: application/json' -H "Content-Type: application/json; charset=ascii"  --data @/tmp/slack_git_msg $url

As you can see, there is some converting and replacing magic involved, to end up with a message which curl can hand over as valid json.

The SVN side

Place the above script in a location where it is callable from svn (e.g. /usr/bin) and give it the appropriate execution rights.

Next we have to tell svn to call the script and pass it the correct parameters after each commit to our repository(-branch).
To do this, edit the post-commit script of your repository. Its usually located in <path-to-repo>/hooks/post-commit.

Add the following lines:

if [ -n "$(svnlook changed ${REPOS} -r ${REV} | grep <branchname>)" ]
then
  /usr/bin/slack_send_svnlog.sh ${REPOS} ${REV}
fi

And we are done here. A commit should now produce a post displaying the commit message in the selected slack channel:

Packaging a spring-boot application for distribution with docker

Fri, 27 Mar 2015 00:00:00 +0100

Packaging a spring-boot application for distribution

But how

Over the last couple of weeks i built a little web application which allows for downloading files to a server. think headless version of jdownloader. because i wanted to try out some stuff like spring-boot,reactor,webcomponents, papyrus, websockets. and because i had use for a headless version of jdownloader.
Anyway, it’s ready now for installation, and i want to package the application jar along with a script which allows it to run as a service and an external config file. But it turns out, imho, there is no really clean and portable way of doing so.

Docker!

So i am going to do this with docker. which seems to be the hip way to do it nowadays anyway.
And it turns out, there are even docker plugins for maven and gradle which let you build an docker image from a Dockerfile. Yay!
As you see, this is really easy to accomplish, and as a bonus you get to control the environment your app is deployed in.
which translates for me to: Java 8 anywhere :)
Deploying to any major PaaS provider should theoretically work too.

What it looked like

build.gradle

add this to build.gradle

buildscript {
    dependencies {
    	classpath('se.transmode.gradle:gradle-docker:1.2')
    }
}

apply plugin: 'docker'

group = 'somehost:5000' //your docker registry location

task buildDocker(type: Docker, dependsOn: build) {
	
	push = true
	applicationName = jar.baseName
	
	dockerfile = file('src/main/docker/Dockerfile')
	
	doFirst {
	  copy {
		from jar
		into stageDir
	  }
	}
	
  }

Dockerfile

and the Dockerfile for my application lives ins src/main/docker/ and looks like this :

FROM jeanblanchard/busybox-java:8
MAINTAINER lm "lm@combase.de"

ADD getdown-app-springboot.jar app.jar
RUN touch /app.jar
CMD mkdir /media
ENTRYPOINT ["java","-jar","/app.jar"]

Hello World! with KODI

Fri, 02 Jan 2015 00:00:00 +0100

“Hello World!” with KODI

I wanted to use the last couple of days before its back to work to take a brief look into the world of xbmc (now KODI!) addon development.
This post describes how one goes about writing and running the obligatory “hello world!” program with KODI.
The addon language of choice in KODI world is python, and it brings its own interpreter. I will use pycharm as an IDE.

What you need

KODI - A media center software which will act as the addons host. howto install KODI in your linux distro of choice is described here.
A Texteditor or IDE. Popular ones for Python are Pycharm, Jetbrains take at a Python IDE, or the Eclipse based PyDev.
xbmcstubs - a set of python files which provide method stubs for the libraries included in KODI’s python interpreter.

Other useful resources

HOW-TO:HelloWorld addon on KODI wiki.
Audio/video add-on tutorial on KODI wiki.

How to do it

Project structure

The addon directory must have the same name as the addon itself, which will be named io.github.lumue.xbmc.hello.
to be available to KODI it must be placed in $HOME/.kodi/addons and contain at least these two files:

addon.xml addon metadata
addon.py the addon python script

addon.xml

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<addon id="io.github.lumue.xbmc.hello" name="Hello World" version="1.0.0" provider-name="lumue">
  <requires>
    <import addon="xbmc.python" version="2.14.0"/>
  </requires>
  <extension point="xbmc.python.pluginsource" library="addon.py">
    <provides>executable</provides>
  </extension>
  <extension point="xbmc.addon.metadata">
    <summary lang="en">Hello World</summary>
    <description lang="en">a "Hello World" addon</description>
    <disclaimer lang="en"></disclaimer>
    <language></language>
    <platform>all</platform>
    <license></license>
    <forum></forum>
    <website></website>
    <email>mueller.lutz@gmail.com</email>
    <source></source>
  </extension>
</addon>

addon.py

import xbmcaddon
import xbmcgui

xbmcgui.Dialog().ok("Hello World!","Hello World!")

A redirect page with fallbacks

Sun, 12 Oct 2014 00:00:00 +0200

A Template HTML5 redirect page

<html>
  <head>
  <title>Page Title</title>
  
  <meta http-equiv="Refresh" content="0; url=target.html">
  
  <!-- If the meta tag doesn't work, try JavaScript to redirect. -->
  
  <script type="text/javascript">
        window.location.href = "target.html"
      </script>
  </head>
  
  <body>
    <!-- If JavaScript doesn't work, give a link to click on to redirect. -->
    <p>
      If you are not redirected automatically, click <a
        href="target.html">here</a>.
    </p>
  </body>
</html>

running ntopng on ubuntu 14.04

Sun, 31 Aug 2014 00:00:00 +0200

running ntopng on ubuntu 14.04

starting nprobe

sudo nprobe --zmq tcp://*:5556 -i none -n none -b 2 -3 9996 -G

starting ntopng

sudo ntopng -i tcp://127.0.0.1:5556 -e -F

LAN Traffic monitoring with dd-wrt

Fri, 22 Aug 2014 00:00:00 +0200

LAN Traffic monitoring with dd-wrt

My setup

I use an ancient wrt54 running dd-wrt to keep our home devices connected. it has no fancy gbit interfaces or super fast wlan, but aside from the occasional dhcpd crash, it is really a reliable little box. i think i have used it for over 9 years now.
first with external antennas to connect to the residential community wlan, later when i switched to a dsl land line, i started to use it as my home router and wifi accesspoint..

The firmware i am using is dd-wrt.
connected to its lan interfaces is a dsl router, my media center, PCs and network storage.
the wlan interface is used as an AP for mobile devices.

##The Plan! Whats missing is a tool that lets me “see” what is going on inside my home lan and wlan. who is talking to whom, which machine uses how much bandwidth, etc.
since the wrt54 is the only device which is connected to lan and wlan, this is where the capturing tool will run. for data collection and crunching i plan to use ntop on another box.

How to do it

Setting up the RFlow probe

The probing can be activated in dd-wrt’s web console. The RFlow section is located under services and looks like this:

Server ip refers to the ntop host.
Port can be any available port on the ntop box.AFAIR there are a few standard ports, and 9996 is apparently one of them.

Save the settings, and the router setup should be complete. easy!

ntop

Installation

lm@voyager3:~$ sudo apt-get install ntop*

The installation routine asks for a network interface to listen on. “none” since the network traffic will be probed somewhere else. next you have to choose a password for ntops webinterface.

The designated ntop host is a box running ubuntu 14.04. i use apt-get to install all packages ntop. and again no errors, faults or exception. not even a lowly warning. almost to easy to blog :D but its not done until its in production as they say.
next is setting up ntop.

Setup

ntop configuration is done in a webui. note that we are doing linux sysadmin stuff, and yet no config file has been touched and no kernel module has been compiled!
personally, i dont like all this userfriendliness. kids these days.

the url is http://<ntop-host>:3000.

to listen for data from the router on port 9996, ntop requires the configuration of a NetFlow Device.

the NetFlow Device configuration is located under Plugins/NetFlow/Configure.

The first three attributes were all i had to change:

NetFlow Device is a name for the device, i used the routers hostname.
Local Collector UDP Port the port we want ntop to listen on. it has to be the same as setup for the rflow probe.
Virtual NetFlow Interface Network Address ip and netmask of the box hosting the rflow probe.

The last step is switching the nic ntop collects data on to the netflow device.It should be selectable in the admin menu:

And its done! Data is streaming in and the statistics are filling up.

Load balancing insert and update requests to a http api using spring-integration

Mon, 18 Aug 2014 00:00:00 +0200

Load balancing insert and update requests to a http api using spring-integration

Overview

We will use a message queue to achieve asynchronous processing of insert and update requests to a webapi.

Why is it important

Imagine what happens in your application if suddenly a lot of requests start coming in, which demand synchronous transaction execution against a database. The webserver will start to spawn more threads. Each of these threads has to acquire a database connection. This will not only drain your http-threadpool, but also your applications connection pool. after a while, most threads will be waiting for a database connection. which in turn leads to the webserver spawning more and more threads. eventually your application becomes unresponsive.

So http-threads are a resource you do not want to hog. this is even more important if your application is accessible on the public internet.

it is also a good thing to decouple your backend workload from the http-request frequency. remember that every thread that executes a transaction against a database likely also occupies a db connection.

How does it work (the short version)

When invoking a service-method directly from a http-request-processing thread, it will be occupied until the transaction completes. To avoid this, the request is handed off to a message queue and delivered to a service-activator endpoint.

this way, updates can be queued while your backend-threads and database eat them up at a steady load.

if guaranteed delivery is needed, the message queue can be made persistent with a message store. a http ok response is sent only if the message has been added to the queue.

When (not) to use this

This pattern is useful if you have to handle a large amount of non trivial create or update requests to your backend in a short amount of time. Consider it especially if you have a public facing api or a spiky usage pattern.

Its not applicable if the clients requires a response containing processing results. In this case consider to redesign your endpoint api to allow for “fire and forget” style requests:

Does your client really need to know if the backend transaction completed successfully, or is it enough to acknowledge the delivery?

Is it possible to pull the backend for results later?

Obviously, this adds another level of complexity and also some overhead to your application. So only use it, if you need it. Fortunately, if you design your service api with this pattern in mind, its easy to add the implementation later.

An example

You have built an application which receives and processes orders from thousands of clients around the country. you even stress tested it with the expected workload of 500.000 orders a day and everything works fine.

###There is always a little detail missing from the spec!

but after it is deployed to production, complaints start coming in. apparently each order has to be sent several times until it is accepted and also the userinterface seems to become slow and unresponsive every night. So you check the logs and notice a lot of exceptions which look like this:

turns out there is a little detail about that specified load which you haven’t been told: all the clients start sending their orders at the same time every night!

The Application

The Domainmodel

Is a really simple one (after all it is an example application!).

But there is a small gotcha with serious implications on the processing of orders. As you can see, the product quantity on stock is a direct property of Product. Since the processing of an order involves the update of stock, the corresponding row in the product table has to be locked, which in turn requires the transaction isolation level to be serializable.

####Object Persistence

The domain objects are persisted using JPA and hibernate. Mysql is used as a database. Nothing special here.