Claus Witt dot Com

How to be an idiot at coding part infinite + 1

2023-11-20T08:54:35+00:00

We had some code read partial responses from a server. This code was tested thoroughly and worked as intended. We got the amount of bytes we needed from the head of a file, and then were able to parse that data as a binary blob.

However; when we put this into production - containers started crashing randomly.

Or rather the code has been in production for years without a crash, but now we were utilizing this code path many times an hour instead of a couple of times per day.

The culprit was hard to find - but it all releates to how ruby works, how networking works, and how I though it worked more like in c.

The old code did something like this:

bytes = nil

uri = URI(url)
begin
  http = Net::HTTP.new(uri.host, uri.port)
  if url.start_with?('https')
    http.use_ssl = true
    http.verify_mode = OpenSSL::SSL::VERIFY_NONE
  end
  http.start do |h|
    request = Net::HTTP::Get.new(uri.request_uri)
    h.request(request) do |response|
      bytes = response.socket.read(count)
    end
  end
rescue IOError => e
  # ignore
end

bytes

and the new code does something far more simple; uses the range header

headers = {'Range' => "bytes=0-#{limit}"}
uri = URI(url)
response = Net::HTTP.get_response(uri, headers)

The former code retrieved the complete file response from the server, and then read the amount of bytes into a variable; the latter only requests the bytes from the server that is needed.

For small files the difference is negligible; but the larger the files get the larger the problems becomes. Response times of the former code goes up, memory usage also goes up. And since video files can be more than 100mb in size in our usage; the slow downloads combined with the large memory usage causes the OOMkiller to destroy the container before the process finishes; further re-enqueing the same job - which increases the possibility that multiple jobs of this type gets handled by the same container in "the same time" - further increasing the possibility of the container crashing again.

One of those "I just needed to deploy"-days

2023-09-25T09:38:20+00:00

It all began when I had to deploy a change, that needs to go live today.

The software to build the docker image failed because of the ruby version not being supported... and so began the "let's update this outdated rails app with obscure dependencies"-dance that took an hour or so. (Updating a rails app from ruby 2.x to ruby 3.x where many dependencies are locked to 2.x)

And then I ended up at the "this ruby version is not supported".. because the one I have installed locally on my machine has be succeeded by a newer version - and the build...

Trying to install this version with rbenv also failed, because the install of ruby build is out of date, and we need to run a brew update...

Finally did that (while writing this) and now I can run the rbenv install command....

Hopefully this is the final step, or I will start having a tantrum.

Pretty Print json in vim

2023-06-29T07:08:16+00:00

I hand edit json many times a week. For that reason the vim command for doing a quick pretty printing of json in current buffer usually is muscle memory for me; however I mistyped it twice today, and decided to make a command and a keybinding for it instead. (Even though I will probably forget it in a couple of days, and revert to typing the command by hand).

Nevertheless; here it is if you need it:

function! JsonPrettyPrint()
  %!python -m json.tool
endfunction


noremap <Leader><Leader>jp :call JsonPrettyPrint()<cr>
command! -nargs=0 JsonPretty :call JsonPrettyPrint()

First we define the function, and it is really just a function name wrapping the command that I mistyped today.

Then we bind a key-binding to it; in my case ,,jp

And finally we also make a command that can be called by typing :JsonPretty in normal mode.

Our most simple transcoding service

2022-04-27T06:30:00+00:00

At Fliva we had an issue; customers kept uploading iPhone footage in HDR formats with varying framerates, and our software could not handle it. The colors looked washed out, and just plain wrong.

Our first stab at a solution was, of course, to use ffmpeg. However ffmpeg produced the same result when converting hdr to sdr. We did find some posts (you know that thing called stackoverflow?) about how to set - a lot of - parameters to make it work. But we never found the magic incantation to make it work with the same settings every time.

We tried out a bunch of transcoding service providers; but none of them supported converting hdr footage from an iPhone to sdr footage that looked right (one was close; but then the sound started being out of sync in some cases)

We have our own iPhone app that most users upload through, and in an earlier version this app could transcode the files correctly locally on the phone before uploading. However, this feature was buggy in other ways - causing the app to sometimes crash at this step and other times to upload the hdr version anyway. And it was sloooow for the endusers.

But this made us think; we know that apple hardware/software supports this; and quickly found the avconvert tool installed on all macs. (If you right click a video file, there is an entry in the Service submenu called "Encode Selected Video Files" which uses this exact command behind the scenes)

We started using this for manually converting hevc/hdr footage from customers, and it just worked every time.

Next step; make a service that does this for us. And we allready had most of the plumbing ready for the other tries with various providers, so instead of our api wrapper project which receives a http call from us, and uses a callback to notify when the file is done transcoding - we move the enqueing part to sqs.

Then we made a very small go program, that consists of theses steps:

poll the sqs queue for new entries; and for each:
use ffprobe to find out if this requires transcoding; and if it does:
download the file locally
convert it using avconvert
upload the resulting file to s3
delete the files locally
and always (whether we transcode or not) callback with a valid video url

That is it.

The go program really delegates all heavy lifting to ffmpeg, curl and avconvert, and is only responsible for polling the sqs queue and sending http requests as callbacks when it is done.

If we need more throughput the current server can probably handle running a couple of these instances without problems; and if we need even more - we will just start a new server with the same software.

If we need to handle multiple priorities; we will just use multiple sqs queues and push the job to the priority we need; and let the services go through high priority queues before low priority ones.

I suspect this will scale very well without much change needed; and will provide us with some time to make sure our own software can handle this video format.

Writing a Progress Bar Overlay Shader

2022-03-11T07:00:00+00:00

Some of our customers in Fliva want to show a simple progress bar inside their video. You see this a lot nowadays in social media videos, where the progress of the video is embedded inside it so that you, as the consumer of the video, know how much more you need to watch.

Our rendering engine has two ways of implementing (visual) effects: either you write a complete plugin, which in turn requires you to write a custom class adhering to our plugin interface, or you just need to write a custom shader.

I chose the latter for this simple plugin, created the shader, and named it "progress," and it is now generally available on our platform.

#version 330 core
uniform sampler2D sceneTex;
out vec4 outColor;
uniform float progress;
uniform vec2 resolution;
uniform float bar_size = 2.0;
uniform float bar_padding = 10.0;
uniform vec4 bar_color = vec4(1.0, 1.0, 1.0, 1.0);

void main()
{
  vec4 color;
  float y = gl_FragCoord.y;
  float x = gl_FragCoord.x;
      if(y > bar_padding && y < bar_padding + bar_size && x < progress * resolution.x) {
      outColor = bar_color;
  } else {
      vec2 uv = gl_FragCoord.xy / resolution.xy;
      outColor = texture(sceneTex, uv);
  }
}

The effect can be added to any scene (or a complete video, which would be the most common use case and our platform exposes the uniforms as variables on the effect.

Writing a Hubot Adapter for Bitrix

2022-03-10T10:00:00+00:00

Back in the day, we installed hubot on our Hipchat developer chat. When hipchat got sold to Atlassian we jumped ship and chose Slack instead. Years later the company chose to invest heavily in having a shared system across all division handling tasks and CRM primarily - but since this tool also had a chat function, we dropped the slack instance, losing access to all the nice Hubot integrations we wrote over the years. Photo by Rock'n Roll Monkey / Unsplash As you may have already guessed the system we chose was Bitrix.

Today we added yet more developers into our shared chat; I got to think about dead old Grottebot (the name for our Hubot instance).

Inspecting requests from Bitrix

I started by setting up a chatbot in our system and sent its requests to a requestbin so that I could inspect them.

Then I added the bot to a chat, send it some messages, and inspected the data, to get acquainted with which data we needed to manage.

Looking into existing adapters

I then took a look at the existing adapters to try to commonalities and differences. And quickly found that this might actually be quite simple.

We need a way to get messages from the bitrix chats; this just requires we set up a webhook in our hubot instance, that can read the webhook calls from Bitrix; and then we need to implement the reply and send methods of the adapter to send stuff back to the chats.

Initial implementation

Since this is a simple one-off adapter for use in an internal tool, I found it overkill to create a separate package; however making a local package which conforms to the hubot adapter naming conventions proved to take a little time.

I ended up with the followinging entry in package.json

{
  "dependencies": {
    "hubot": "^3.3.2",
    "hubot-bitrix": "file:hubot-bitrix",
    .... other dependencies
  }
}

I then implemented the required methods in the adapter

{Robot,Adapter,TextMessage,User} = require 'hubot'

BitrixBotClient = require './bitrix_bot_client'

class BitrixAdapter extends Adapter

  constructor: ->
    super
    @robot.logger.info "Constructor"
    @client = new BitrixBotClient(@robot)

  send: (envelope, strings...) ->
    @client.send(envelope, strings...)

  reply: (envelope, strings...) ->
    @client.reply(envelope, strings...)

  run: ->
    @robot.logger.info "Run"
    @emit "connected"
    @robot.router.post '/some/secret/path/to/the/webhook/endpoint', (req, res) =>
      @client.parseMessage req

      res.status(200).send ok: true

exports.use = (robot) ->
  new BitrixAdapter robot

And I just needed to implement the BitrixBotClient class, which does all of the heavy lifting.

{Robot, Adapter, TextMessage} = require "hubot"

class BitrixBotClient
  constructor: (robot) ->
    @robot = robot

  parseMessage: (request) ->
    if request.body.event == 'ONIMBOTJOINCHAT'
      return @parseJoinRequest(request)
    if request.body.event == 'ONIMBOTMESSAGEADD'
      return @parseMessageRequest(request)

  parseJoinRequest: (request) ->
    @robot.brain.set 'bitrix', {
      base_url: process.env.BITRIX_REST_URL, # This will be linked to some user; so beware
      client_id: request.body.auth.application_token,
      bot_id: request.body.data.PARAMS.BOT_ID
    }
    @sendMessage(request.body.data.PARAMS.DIALOG_ID, "Bot entered the building")
    return undefined

  parseMessageRequest: (request) ->
    text = request.body.data.PARAMS.MESSAGE
    messageId = request.body.ts
    @createUser request, (user) =>
      message = new TextMessage user, text.trim(), messageId
      @robot.receive(message) if message?

  createUser: (request, cb) ->
    userId = request.body.data.USER.ID
    userName = request.body.data.USER.NAME
    roomId = request.body.data.PARAMS.DIALOG_ID

    cb @robot.brain.userForId(userId, name: userName, room: roomId)


  send: (envelope, strings...) ->
    for message in strings
      @sendMessage envelope.room, message

  reply: (envelope, strings...) ->
    for message in strings
      @sendMessage envelope.room, "<@#{envelope.user.name}> #{message}"

  sendMessage: (chatId, message) ->
    bitrix = @robot.brain.get('bitrix')
    url = bitrix.base_url + '/imbot.message.add'
    data = JSON.stringify({
      'BOT_ID': bitrix.bot_id,
      'DIALOG_ID': chatId,
      'MESSAGE': message, # Message text
      "CLIENT_ID": bitrix.client_id,
    })

    console.log("SEND MESSAGE", data)

    @robot.http(url)
      .header('Content-Type', 'application/json')
        .post(data) (err, res, body) ->
          result = JSON.parse(body)
          console.log(result)


module.exports = BitrixBotClient

That is the complete solution currently.

Missing features

Currently this sollution has some minor problems which make it work differently than hubot normally does.

The chatbot api in Bitrix does not allow the chatbot to get all messages in a groupchat, only the ones specifically aimed at it. So (assuming the botname in bitrix is called Hubot)

When want a mention in hubot terms (the respond function in plugins) you need to first mention the bot so that the bot gets the message, and then mention it again, so that the bot knows it is a respond-function that should be called.

We have solved this by making all apis use the hear-method instead.

That also means that the bot can never "hear" anything without getting mentioned. Sadly; because it has it's uses. Even though most of the uses we had for it in our internal chats was to annoy the hell out of people using specific phrases.

Corona Beats from Last Year

2022-03-10T07:30:00+00:00

Last year I spent some time one afternoon making a short corona-beat. I never really shared it... Now it's here...

SoundCloud · Corona Beatz #1

Decoding Video Frames with ffmpeg/libav

2022-01-04T06:13:00+00:00

Opening the file

AVFormatContext *formatContext;

if (avformat_open_input(&formatContext, "path/to/file.ext", nullptr, nullptr) != 0)
    exit(1);  // Couldn't open file

/// Retrieve stream information
if (avformat_find_stream_info(formatContext, nullptr) < 0)
    exit(2);  // Couldn't find stream information

Find the first video stream

        int videoStreamIdx = -1;
        for (int i = 0; i < (int)formatContext->nb_streams; ++i)
          if (formatContext->streams[i]->codecpar->codec_type ==
              AVMEDIA_TYPE_VIDEO) {
            videoStreamIdx = i;
            break;
          }
        if (videoStreamIdx == -1) exit(3);  // Didn't find a video stream

Find the correct codec for decoding

AVCodecContext *codecContext;
AVCodec *codec;

codecContext = avcodec_alloc_context3(codec);
/// Get a pointer to the codec context for the video stream
avcodec_parameters_to_context( codecContext, formatContext->streams[videoStreamIdx]->codecpar);
/// Find the decoder for the video stream
codec = avcodec_find_decoder(codecContext->codec_id);
if (codec == nullptr) {
    fprintf(stderr, "Unsupported codec!\n");
    exit(4);  // Codec not found
}

Initialize buffers and conversion context

AVFrame *inputFrame;
AVFrame *outputFrame;

inputFrame = av_frame_alloc();
outputFrame = av_frame_alloc();

int w = codecContext->width;
int h = codecContext->height;
int numBytes = av_image_get_buffer_size (AV_PIX_FMT_RGBA, w, h, 16);
buffer = (uint8_t *)av_malloc(numBytes * sizeof(uint8_t));

av_image_fill_arrays (outputFrame->data, outputFrame->linesize, buffer, AV_PIX_FMT_RGBA, codecContext->width, codecContext->height, 1);


conversionContext =
    sws_getContext(w, h, codecContext->pix_fmt, w, h, AV_PIX_FMT_RGBA,
                   SWS_FAST_BILINEAR, nullptr, nullptr, nullptr);

Decode a frame

AVPacket packet;

while ((av_read_frame(formatContext, &packet) >= 0)) {
    if (packet.stream_index == videoStreamIdx) {
        avcodec_send_packet(codecContext, &packet);
        int frameFinished = avcodec_receive_frame(codecContext, inputFrame);
        // Did we get a video frame?
        if (frameFinished == 0) {
            sws_scale(conversionContext, (const uint8_t *const *)inputFrame->data,
                      inputFrame->linesize, 0, codecContext->height, outputFrame->data,
                      outputFrame->linesize);
            av_packet_unref(&packet);

            // DO SOMETHING WITH outputFrame
        }
    }
}

Drain frames

avcodec_send_packet(codecContext, nullptr);
while(avcodec_receive_frame(codecContext, inputFrame) == 0)
    sws_scale(conversionContext, (const uint8_t *const *)inputFrame->data,
              inputFrame->linesize, 0, codecContext->height, outputFrame->data,
              outputFrame->linesize);
    // DO SOMETHING WITH outputFrame
}

Clean up

av_free(buffer);
av_free(outputFrame);
av_free(inputFrame);
avcodec_close(codecContext);
avformat_close_input(&formatContext);

Install Datadog in Kubernetes

2021-06-04T05:33:00+00:00

First of all, we use helm. So we start here: https://docs.datadoghq.com/agent/kubernetes/?tab=helm

Add the repository

helm repo add datadog https://helm.datadoghq.com
helm repo add stable https://charts.helm.sh/stable
helm repo update

Install the chart

helm install datadog -f values.yaml  --set datadog.apiKey=<DATADOG_API_KEY> datadog/datadog --set targetSystem=linux

Special case: AWS

And then because we use EKS and thus Amazon Linux 2 we need to do this (source: https://artifacthub.io/packages/helm/datadog/datadog#configuration-required-for-amazon-linux-2-based-nodes)

agents:
  # (...)
  podSecurity:
    # (...)
    apparmor:
      # (...)
      enabled: false

# (...)

Service discovery

Then we need to tell the pods where the datadog api is (it is no longer on localhost inside a container, as we were used to from the datadog buildpack, but on the node it is scheduled on). This can be pushed to us on pod creation (existing pods wont get this value though - but we deploy changes to all applications anyway and change a hardcoded localhost to a read of the environment variable DD_AGENT_HOST)

  admissionController:
    # clusterAgent.admissionController.enabled -- Enable the admissionController to be able to inject APM/Dogstatsd config and standard tags (env, service, version) automatically into your pods
    enabled: true

    # clusterAgent.admissionController.mutateUnlabelled -- Enable injecting config without having the pod label 'admission.datadoghq.com/enabled="true"'
    mutateUnlabelled: true

Findin all nodes: taint and tolerations

Finally we ran into a minor issue; which took forever to figure out. Our cluster have two autoscaling groups, one for long running pods, and one for pods that we allow to get rescheduled (most of our apps are ok with that). But this is set up as a taint in kubernetes; and we need to tell datadog about the toleration, otherwise only the nodes without the taint will get a datadog agent.

  tolerations:
    - key: "removable"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"

A quick test

echo "some.new.to.send.to:1|c" |nc -w0 -u $DD_AGENT_HOST 8125

on a newly scheduled pod, and everything worked!

Just Create a Relation

2021-05-18T10:13:00+00:00

I recently read You might as well timestamp it and today I came across an equal truism

Just create a relation.

Many a time have i seen tables of a given object where the table has a status field and a status_changed timestamp.

But guess what; knowing when a status changed, what caused it AND WHAT THE PREVIOUS STATUS WAS happens quite a lot.

So whenever I write a field called status, state or something similar I always default to making it a relation.

So instead of post table like idtitlestatusstatus_changed_atbodysome idsome titlesome statustimestampbody I opt for two tables idtitlebodysome idsome titlebody and post_idstatuscreated_atcreated_bysome idsome statustimestampuser_id The current status is always the last entry (ordered by created_at) in that table for that given id.

Basic Breakout Game with Raylib

2021-05-17T09:11:00+00:00

I have always wanted to write games. Back in the day that was the whole idea why I wanted to be a programmer. And I have worked myself into a new job at Fliva exactly because I wanted to become a game programmer.

Only problem was that I found a way to make a much more lucrative business out of our video company by applying what little knowledge I had aquired about game programming to video programming... And suddenly I was doing low level C++ programming as a day job; and game programming went a little to the background in my mind.

And that have not really changed; however I like to do little experiments with game programming now and again.

Why write a game now?

This time it was because I was looking into teaching my son to make games. He wants to be a game programmer when he grows up; but he is still in the unrealistic phase of "I want to make the next GTA meets Fortnite on my own, and become a bazzilionaire"thinking, where this game programming thing is just get an idea, tell the computer what the idea is and make a ton of money!

I want to both teach him that this is hard, but also that it is doable.

Well; first of all I need to know whether to go the "learn to code first"-route with him or the "learn a game engine first" route. This is my attempt to find a simple starting point for the learn to code first route, where it is not overly complex, but also not overly limiting in what he will be able to do.

Raylib

RayLib is in my opinion a great learning tool; so I tried to make breakout in Raylib - and c++. I am in no way done; and I have some things that are definitely not working correctly. But the game can be started, played and has two levels... Whoah!

Entrypoint

Currently the game has the simplest main function:

int main(void)
{
    auto game = setup();
    run(game);
    shutdown();
    return 0;
}

Setup returns a Game struct, which is sent to run (which is the main game loop). When the game loop is over, the shutdown method will be called (shutdown just closes the window, which is opened in the setup method)

Immutable state

The run method is slightly more interesting.

void run(Game game) {
    while (!WindowShouldClose())
    {
        game = simulate(game);
        draw(game);
    }
}

I opted - mostly as an experiment - to have the Game state be immutable, which means that simulate will take the current state of the world, and return a completely new Game object for each frame. This works quite well (for this tiny game) actually.

The draw game then cannot mutate the Game state at all; and those two methods just run forever, until something makes WindowShouldClose be true.

Project Structure

All that code (and some boilerplate) was placed in a main.cpp file, and the rest of the game has been split into three files.

game_objects.hpp containing all the structs of the game. Game, Scene, Ball, Player, Brick.
draw.hpp containg everything to do with drawing the game
simulation.hpp containing everything that is mutating the game (e.g. player input)

Game Setup

Game Objects

The main state of a game is called Game. This contains data that should "persist" duiring the game; it has a currentScene which is just the current level being played (I have been programming video stuff so long, that Scene was a more natural word for me than Level). The game object also owns the score, number of lives and other data that persists across levels.

The level Scene has a collection of bricks Brick and a collection of balls Ball (this game allows more than one ball being present at a time, even though one is the default). And finally the level has a Player

Player consists of a Rect (position and size) and a speed.

Ball has a radius, position, direction, speed and a texture.

Bricks have a position, texture and score (how much you are awarded in score for destroying the brick)

Loading Levels

Levels are loaded from Tiled data; we parse the tilesets (tsx files) and levels (tmx files); extracting the parts we need.

We create a texture per brick based on the data from the tileset; and set its position and score from the level file.

A future blog post will explore the details of this a bit further ; but it really just is using rapidxml to read the data we need from the xml files.

Simulation

User Input

The only user input we support is KEY_LEFT, KEY_RIGHT for moving the player, SPACE for shooting another ball (if extra balls are present) and finally ESC to exit the game.

Raylib has a single function for testing user input called IsKeyPressed which we use to check for each key in turn.

Then we mutate our (copy of) the game state.

    if(IsKeyDown(KEY_LEFT)) {
        game.currentScene.player.rect.x -= game.currentScene.player.speed * elapsedTime;
    }

    if(IsKeyDown(KEY_RIGHT)) {
        game.currentScene.player.rect.x += game.currentScene.player.speed * elapsedTime;
    }

    if(IsKeyPressed(KEY_SPACE)) {
        if(game.extraBalls > 0) {
            auto ball = defaultBall();
            ball.position.x = game.currentScene.player.rect.x + 20;
            ball.position.y = game.currentScene.player.rect.y - 20;
            game.currentScene.balls.push_back(ball);
            game.extraBalls--;
        }
    }

Collisions

There are three types of collisions in the game.

Check ball against outer walls

There are no rendered outer walls in this game; so we just check to see if the ball is heading outside the window.

Special handling is required for the bottom part of the screen, since this will cause a life to be lost (and potentially end the game)

        if(ball.position.x + 0.5 * ball.radius > game.width || ball.position.x - 0.5 * ball.radius < 0) ball.direction.x *= -1;
        if(ball.position.y - 0.5 * ball.radius < 0) ball.direction.y *= -1;
        if(ball.position.y + 0.5 * ball.radius > game.height) {
            if(ballCount == 1) {
                if(game.lives == 1) game.state = GAMEOVER;
                game.lives -= 1;
                ball.direction.y *= -1;
                ball.position.y = game.currentScene.player.rect.y - 50;
                ball.position.x = game.currentScene.player.rect.x + 50;
            } else {
                game.currentScene.balls.erase(game.currentScene.balls.begin() + ballId);
                ballId--;
            }
        }

Check ball against bricks

Raylib has a collision function we can use called CheckCollisionCircleRec which does the heavy lifting. We run through all the bricks to check if there is a collision with any of them; and then we run through the collision list handling all collisions by changing the balls direction, adding the score and removing the brick.

        std::vector<int> collisionList;
        {
            int index = 0;
            for(auto brick : game.currentScene.bricks) {
                Rectangle rect;
                rect.x = brick.position.x;
                rect.y = brick.position.y;
                rect.width = brick.texture.width;
                rect.height = brick.texture.height;
                if(CheckCollisionCircleRec(ball.position, ball.radius, rect)) {
                    collisionList.push_back(index);
                    break;
                }
                index++;
            }
        }

        for(auto index : collisionList) {
            auto brick = game.currentScene.bricks[index];
            if(ball.position.y < brick.position.y) {
                ball.direction.y *= -1;
            } else if(ball.position.y > brick.position.y + brick.texture.height) {
                ball.direction.y *= -1;
            } else {
                ball.direction.x *= -1;
            }
            if(game.currentScene.bricks.size() < 20) {
                ball.speed += 10;
            }
            game.score += brick.score;
            game.currentScene.bricks.erase(game.currentScene.bricks.begin() + index);
        }

Check ball against player

The first naive version of this game only has the ball moving in 45 degree angles; which makes this test very simple.

        if(CheckCollisionCircleRec(ball.position, ball.radius, game.currentScene.player.rect)) {
            ball.direction.y *= -1;
        }

Drawing

The draw function is relatively simple. BeginDrawing and EndDrawing are functions Raylib requires you to call before and after drawing. ClearBackground is called before any drawing commands, and the switch statement is there to only draw the parts of the app, that matches the state the game is in.

This also means that we draw every single element to screen for every frame of the game.

void draw(Game game) {
    BeginDrawing();
    ClearBackground(RAYWHITE);

    drawGameUI(game);

    switch(game.state)
    {
        case START  : drawStartScreen(game);   break;
        case PLAYING: drawScene(game); break;
        case NEXTSCENE: drawScene(game); break;
        case GAMEOVER : drawGameOver(game);  break;
    }

    EndDrawing();
}

Drawing UI

UI Elements like score, lives and extra balls are really just text. Drawing them is dead easy with Raylibs DrawText function.

    std::string score = "Score: " + std::to_string(game.score);
    DrawText(score.c_str(), 20, 20, 20, BLACK);

    std::string lives = "Lives: " + std::to_string(game.lives);
    DrawText(lives.c_str(), 520, 20, 20, BLACK);

    std::string balls = "Extra Balls: " + std::to_string(game.extraBalls);
    DrawText(balls.c_str(), 720, 20, 20, BLACK);

Drawing Player Bat

The player bat is really just a rectangle with a position, size and a color.

    DrawRectangle(scene.player.rect.x, scene.player.rect.y, scene.player.rect.width, scene.player.rect.height, BLACK);

Drawing Ball

The balls are all just circles with a color.

    for(auto ball : scene.balls) {
        DrawCircle(ball.position.x, ball.position.y, ball.radius, RED);
    }

Drawing Bricks

The bricks are drawn using textures (meanig they have an image, instead of just a color).

    for(auto brick : scene.bricks) {
        DrawTexture(brick.texture, brick.position.x, brick.position.y, WHITE);
    }

Know your assumptions, and test them

2020-02-04T12:31:00+00:00

In my day to day I very often stumble upon yet another issue in one of our software systems that needs fixing. And most often, the issue is rather apparent and easy to fix.

This past couple of days (ok, almost a week) I have been stuck on a problem that I only really got solved when I talked to a colleague about a problem (or actually two problems) they had.

My issue had to do with something in one of three threads not doing what it was supposed to in very rare cases (about 5% of all runs), but attaching a debugger or inserting debugging print statements made the problem go away. At least it seemed that way.

My colleague was stuck on especially one tricky issue - which got me telling him: "Ok, you have tried poking at this for a day - what is it that you assume to be true?" And we got talking - and me being the old guy, having spent days and weeks tracking down weird issues, only to find that it was what I thought I knew to be true, that was the problem all along - I decided to try to be wise...

Ok, I told him (and another colleague in attendance) - you need to know what the symptoms of the problem is - what is it that we know to be true about the problem - and we listed up what was experienced by the users.

And next - try to come up with every possible thing - weird as they may be - that could cause this behavior... And then try to disprove as many as them as quickly as possible.

Next list up what you absolutely know to be true about the state of the system - and then try to disprove that.

... what you end up with should - hopefully - be a manageable shortlist of ideas to what the problem could be, and how to solve it...

It took me a night at home relaxing and a good nights sleep (and a good dose of coffee this morning) to have the epiphany: I was actually not doing that!

I decided to put debug statements outside the threads in places where I was pretty sure it would not cause problems, and began working on a completely different feature - I figured since this problem only appears sometimes I could at least make the test runs test something else as well... And lo and behold - within 30 minutes the error appeared the log message told me something interesting: "the ring buffer had 375 written entries of sample data - but no reads yet".

OH! I knew exactly what line of code checked whether we should read samples or not... And that single if-statement read a sample_count variable (how many samples we had written to the output video), and if that was lower than the available samples it should read samples from the ring buffer and write to the output video and increment sample_count variable.

And since we now knew that we never wrote any data to the stream (since we had read zero times from the ring buffer) and that the ring buffer was full (375 being the max entries of 128 bytes samples it could hold) - the initial value of sample_count would have to be more than zero the first time that if statement was run... (at least in 5% of the runs we had)

This sounded like the good old "uninitialized integer assumed to be zero"-problem. I cannot understate how many times I have made this mistake and let it consume multiple days of debugging before getting to the eureka moment of realizing: yep, I was being stupid again.

Much more than ten albums that shaped my taste in music

2020-01-29T13:10:00+00:00

My wife began this challenge on Facebook. Each day post an image of an album cover of music that shaped your taste in music... The rules states that you should say why, just post the image (and the rules) and tag someone new each day. (My wife as a good Facebook-citizen of course denied to do that last part)

But her challenge got me thinking. Ten albums? I don't think I could keep it to ten albums per genre I like. So i decided to make a great big list of lists.

Rock Music

As a genre rock may be one of the broadest, but a lot of releases in this genre has shaped my tastes. It is also the first genre I really listened to - that moved me - so it had to come first.

Backman Turner Overdrive - "Not Fragile"

The first times I heard the first track on this album - it meant "Daddy is angry - and probably drunk!". I hated that song for years. But at some point - as a pre-teen - I began listening to it myself. And especially the first two songs on the album was what I thought as Hard Rock for years. (I got into a fight with classmates whether or not it was as good as Metallica).

Eric Clapton - "Journeyman"

When I began playing guitar, Clapton was god to me. My parents recorded his Royal Albert Hall concert from the Journeyman tour on VHS for me ; and I have watched that tape soo many times. I really liked the newer songs from the new albums from that concert - many of them are still among my favorite songs. Not long after, though, my focus shifted toward other rock acts.

Guns n Roses - "Use Your Illusion II"

This is the first cd I ever bought. (I bought Metallicas Black Album the same day). I had heard a lot of gnr before that - but I heard this album (and the black album) on repeat for weeks and weeks.

Guns n Roses - "Appetite for Destruction"

I don't really know when I got this album (it was after "Use your Illusion"). But I love the songs on this album much more today - I honestly only think that "Civil War" and "Pretty Tied Up" gets any airplay at home these days - but I can listen to all of Appetite without pause...

Metal

Metallica - "Metallica"

The Black Album was the first metal album I bought. (See above). I listened to it daily for weeks (maybe even months). I still love each and every song on it... Sadly I hate everything after this release (maybe I am biased by being into metal for years, and we all just cargo-culted the "Metallica sold out" - I don't know...)

Pantera - "Far Beyond Driven"

On a soccer camp one year, I borrowed a walkman with this tape in it, because I couldn't sleep... I wore out his battery - because I listened to it all night... I bought it as soon as I got home. I love that album. To this day! I can listen to it multiple times back to back, and still not get tired of it... And when listening to their cover of "Planet Caravan" my mind wanders back to lying on in a classroom in some school far away from home.. I can remember everything about that trip from that one song... Powerful stuff...

Sepultura - "Chaos A.D."

I bought this album the same day as I bought the Pantera album. I don't remember if I had heard about them before or not (I probably had). I actually think I quite quickly got tired of them - but they were so popular - so I guess I just pretended to like them...

Our first band actually was called Sepulcultura when we started it. Named after - of course - Sepultura, and a danish yoghurt called Cultura... It was very bad naming.

Machine Head - "Burn My Eyes"

I fucking love "Davidian" from this album. I bought the album on a whim (I guess the cover looked cool?) It was my last thrash/groove metal album - once everybody began listening to the band, I hated it. (You know, I was a hipster before hipsters were a thing!)

Deicide "Once Upon the Cross" & Morbid Angel "Domination"

I don't know which of these was most important to me at the time. (I know that I have listened more to the Morbid Angel album in recent years). I bought both albums in a record store in Prague in 9th grade. I found the guitar sound on the Morbid Angel album very weird, and the Deicide album just sounded brutal as fuck...! (And the overt anti-christian lyrics and imagery was very appealing to a young angry self-proclaimed satanist - of course!)

Edge of Sanity - "Purgatory Afterglow"

I borrowed a tape from my good friend Lars with this album on it... I am pretty sure he never got the tape back. The song writing on the album was refreshing to me, and I loved the lyrics.

In Flames - "Lunar Strain"

Guess what was on the other side of that tape? Yes! The first album by In Flames. It shaped much of my own music writing for the first couple of years.

Dissection - "Storm of the Lights Bane"

To this day one of my favorite albums. I will never grow tired of this album! I heard an earlier version of "Nights Blood" on a compilation a couple of months before the album was released - and I decided to learn it. (And I must admit - I have written many riffs that either sounded like Dissection without me trying - or that I tried to make sound like Dissection, and failed)

I love all of the eight tracks (yes, the intro and outro are awesome in my ears as well!)

I cannot even tell you my favorite track on the album - it has changed a lot. It started being Nights Blood - since I heard that one first. Then it was the hit "Where the Dead Angels Lie" - like everybody else! Later it was Soulreaper... Now I just love the whole album...! Come to think of it ... let's listen to it now!

Emperor - "In the Nightside Eclipse"

The only black metal I really liked when it was released - in those years. (Unless you call Cradle and/or Dissection black metal, that is). Everything else was just noise to me at first. (Within a couple of years I loved all of it - exposure seems to change your tastes)

I can still get lost in the worlds this music builds in my mind. The album is the only Emperor album that is best as a whole - meaning I need to listen to the whole album from start to finish... The other albums have good songs, and less good songs... I don't know the song titles from each other on the eclipse-album... It is all one song to me!

Cradle of Filth - "The Principles of Evil Made Flesh", "Dusk... and Her Embrace" & "Cruelty and the Beast"

I listened to "To Eve The Art Of Witchcraft" on a compilation album I borrowed from a friend. I fell absolutely in love with the song - and needed to borrow the complete album - which I also fell in love with! I have listened to every Cradle release since then - as soon as I could get a hand on them (in later years, Spotify has helped with that). The first three especially was very important to me. A lot of good memories have been embedded into that music as well... (The first time I slept - in a bed anyways - with my now wife "Cruelty" was playing on repeat ... so the sentence "Hear me now, all crime should be treasured if they bring you pleasure somehow" - has been our mantra ever since)

HipHop

Eminem - "The Marshall Mathers L.P."

I quit my band just before going of to highschool (Højskole in danish - it is not the same as an American highschool). And I had a period of my life where I was trying to put a distance to metal music. I needed something new in my life. I had been listening to hiphop for years. I love great lyrics, and to this day I still love the wordplays of good rap-music.

This album was on repeat on our school - a couple of us listened to it daily (I am pretty sure a lot of people on the school hated us for listening to it all the time - but they had shitty taste)

Dr Dre - "2001"

Party music for me when I got home from that school..!

L.O.C. - "Dominologi" & U$O - "Mr Mista"

A couple of friends of mine and I started a company (web-development agency) and these two albums were our work-playlist for a while. (Ok, maybe only for the two of us)

To this day I know every sentence to all songs on both albums by heart...!

Tudsegammelt - "Tudsegammelt"

My wife and I was out on the town one night - in our old home-town Ribe - and this band was playing. Some of my old friends/acquaintances were in the band - so I just had to see them... AND I FELL IN LOVE!

I listen to their debut album (which was released 6 months later or so) all the time. The day my daughter was born, we listened to the album on our way to the hospital. I still listen to it weekly.

The band has also been a gateway drug to underground danish rap... And there are a lot of great lyrical mc's out there...!

Madvillain (M.F. Doom) - "Madvillainy"

Your favorite rappers favorite rapper? I fell over his name when I saw a video of Mos Def praising the album on youtube... I was hooked when Mos Def quoted some of his favorite lines... When I heard the album I think I heard it on repeat for a week.

A ring buffer for audio samples

2020-01-21T10:56:00+00:00

At Fliva we work with audio (we have a c++ library that essentially works as a audio sequencer), video (we have a rendering engine which basically works as a scriptable After Effects clone) and we do transcoding, muxing and all that jazz...

When we began this journey we generated video and audio in separate processes and used ffmpeg to mux the files to a video file that could be used by our customer - or uploaded to our player platform.

Now though we do it all in one process, which then requires that the various parts of the engine does not slow down the other parts. I.e. we run the whole thing in multiple threads. At least one for generating video, at least one for generating audio and at least one for muxing the resulting video file.

We use ring buffers for this, which offsets some of the fluctuation in generation speeds by using extra memory.

Our audio data is flowing through such a ring buffer, with a rather simple interface.

class AudioRingBuffer {
  public:
    AudioRingBuffer(int size = 128, int count = 375 , int channels = 2);
    ~AudioRingBuffer();

    bool empty() const { return writes == reads; }
    bool full() const { return size() == bufferCount; }
    uint64_t size() const { return (writes - reads); }

    void write(const float* const* data, int numSamples);

    uint64_t availableSamples() const { return size() * bufferSize; }

    uint64_t read(float *outData);

    uint64_t nextWritePosition();
    uint64_t nextReadPosition();


  private:
    uint64_t writes;
    uint64_t reads;
    uint16_t bufferSize;
    uint16_t  bufferCount;
    int channelCount;
    float* buffer;

};

The sizes should make some sense. Our internal buffersize is set to 128 samples (per channel). The reason is that we run events in the sequencer after each write to the buffer, and doing this for every 128 samples, means there is max of approximately 3ms delay between events (we run at 44100 samples per second internally). In other words, an audio event (play file, change effect param, change pan, change gain, stop file and so forth) will never be more than 3ms late.

The number of buffer places is set to 375, which means we could max have a second of audio in the buffer. (I know what you are thinking, 375 times 128 is not 44100 - and thus it is not a second of audio.... No you are correct dear pedantic reader - but 44100/128 is not an integer - so I decided to use 48000 as the basis for this calculation instead)

Finally almost all our videos are 2 channel audio (stereo) - so we default to that. However both our audio engine library and our muxer could potentially emit any amount of channels...

The implementation of the file is equally simple

AudioRingBuffer::AudioRingBuffer(int size, int count, int channels) : writes(0), reads(0), bufferSize(size), bufferCount(count), channelCount(channels), buffer(new float[bufferSize*bufferCount*channelCount])  {
}
AudioRingBuffer::~AudioRingBuffer() {}

uint64_t AudioRingBuffer::nextWritePosition() {
  auto position = writes % bufferCount;
  return position * bufferSize * channelCount;
}

void AudioRingBuffer::write(const float* const* data, int numSamples) {
  while(full()) {
    std::this_thread::sleep_for(std::chrono::milliseconds(1));
  }
  if(!full()) {
    for(int i = 0;i < channelCount;++i) {
      std::memcpy((buffer + (nextWritePosition() + (i*bufferSize))), data[i], bufferSize * sizeof(float));
    }
    ++writes;
  }
}

uint64_t AudioRingBuffer::nextReadPosition() {
  auto position = reads % bufferCount;
  return position * bufferSize * channelCount;
}

uint64_t AudioRingBuffer::read(float *outData) {
  if(!empty()) {
    std::memcpy(outData, (buffer + nextReadPosition()), bufferSize * channelCount * sizeof(float));
    ++reads;
    return bufferSize;
  }
  return 0;
}

The method nextReadPosition and nextWritePosition could have been private, however we use these in tests.

The write method takes an array of channels of samples. If the ring buffer is full, it blocks the thread (by looping over a sleep call, until the buffer is no longer full), then it copies the audio data into the ringbuffer into a single contiguous area of memory (each channel comes directly after the previous one).

The read method does almost the same thing, just in reverse. The read method never blocks, it either returns samples or nothing. And the samples are written in the incoming pointer contiguously - it is the callers responsibility to map this to channels again, if needed.

This design makes this buffer easy to develop, easy to maintain and relatively easy to test as well...

The test file has a crazy amount of boilerplate, but it tests the most common issues that would crop up in such a ring buffer.

#define CATCH_CONFIG_MAIN  // This tells Catch to provide a main() - only do
// this in one cpp file
#include "audio_ring_buffer.hpp"
#include "catch.hpp"

TEST_CASE("Defaults to not be full", "[audio ring buffer]") {
  AudioRingBuffer subject;
  REQUIRE(subject.full() == false);
}

TEST_CASE("Defaults to be empty", "[audio ring buffer]") {
  AudioRingBuffer subject;
  REQUIRE(subject.full() == false);
}

TEST_CASE("Default read position is the start pointer", "[audio ring buffer]") {
  AudioRingBuffer subject;
  REQUIRE(subject.nextReadPosition() == 0);
}

TEST_CASE("Default write position is the start pointer", "[audio ring buffer]") {
  AudioRingBuffer subject;
  REQUIRE(subject.nextWritePosition() == 0);
}


TEST_CASE("Writing changes size", "[audio ring buffer]") {
  AudioRingBuffer subject;
  REQUIRE(subject.full() == false);

  auto data = new float*[2];
  data[0] = new float[128];
  data[1] = new float[128];
  for(int channel = 0;channel < 2;++channel) {
    for(int sample = 0;sample < 128;++sample) {
      data[channel][sample] = 0;
    }
  }

  subject.write(data, 128);

  REQUIRE(subject.size() == 1);
}


TEST_CASE("Writing changes write poisition by 256 samples", "[audio ring buffer]") {
  AudioRingBuffer subject;
  REQUIRE(subject.full() == false);

  auto data = new float*[2];
  data[0] = new float[128];
  data[1] = new float[128];
  for(int channel = 0;channel < 2;++channel) {
    for(int sample = 0;sample < 128;++sample) {
      data[channel][sample] = 0;
    }
  }

  subject.write(data, 128);

  REQUIRE(subject.nextWritePosition() == 256);
}


TEST_CASE("Reading changes size", "[audio ring buffer]") {
  AudioRingBuffer subject;
  REQUIRE(subject.full() == false);

  auto data = new float*[2];
  data[0] = new float[128];
  data[1] = new float[128];
  for(int channel = 0;channel < 2;++channel) {
    for(int sample = 0;sample < 128;++sample) {
      data[channel][sample] = 0;
    }
  }

  subject.write(data, 128);

  auto outData = new float[256];

  auto samples = subject.read(outData);

  REQUIRE(samples == 128);
  REQUIRE(subject.size() == 0);
}


TEST_CASE("Reading changes read poisition by 256 samples", "[audio ring buffer]") {
  AudioRingBuffer subject;
  REQUIRE(subject.full() == false);

  auto data = new float*[2];
  data[0] = new float[128];
  data[1] = new float[128];
  for(int channel = 0;channel < 2;++channel) {
    for(int sample = 0;sample < 128;++sample) {
      data[channel][sample] = 0;
    }
  }

  subject.write(data, 128);

  auto outData = new float[256];

  auto samples = subject.read(outData);

  REQUIRE(samples == 128);
  REQUIRE(subject.nextReadPosition() == 256);
}


TEST_CASE("Data is read in order", "[audio ring buffer]") {
  AudioRingBuffer subject;
  REQUIRE(subject.full() == false);

  auto data1 = new float*[2];
  data1[0] = new float[128];
  data1[1] = new float[128];
  for(int channel = 0;channel < 2;++channel) {
    for(int sample = 0;sample < 128;++sample) {
      data1[channel][sample] = 0;
    }
  }

  auto data2 = new float*[2];
  data2[0] = new float[128];
  data2[1] = new float[128];
  for(int channel = 0;channel < 2;++channel) {
    for(int sample = 0;sample < 128;++sample) {
      data2[channel][sample] = channel * 128 + sample + 1;
    }
  }

  auto data3 = new float*[2];
  data3[0] = new float[128];
  data3[1] = new float[128];
  for(int channel = 0;channel < 2;++channel) {
    for(int sample = 0;sample < 128;++sample) {
      data3[channel][sample] = 257;
    }
  }
  REQUIRE(subject.nextWritePosition() == 0);
  subject.write(data1, 128);
  REQUIRE(subject.nextWritePosition() == 256);
  subject.write(data2, 128);
  REQUIRE(subject.nextWritePosition() == 512);
  subject.write(data3, 128);
  REQUIRE(subject.nextWritePosition() == 768);

  auto outData = new float[256];

  REQUIRE(subject.nextReadPosition() == 0);
  auto samples = subject.read(outData);
  REQUIRE((int) outData[0] == 0); // LOWER BOUND FIRST READ
  REQUIRE((int) outData[255] == 0); // UPPER BOUND FIRST READ

  REQUIRE(subject.nextReadPosition() == 256);
  samples = subject.read(outData);
  REQUIRE((int) outData[0] == 1); // LOWER BOUND SECOND READ
  REQUIRE((int) outData[255] == 256); // UPPER BOUND SECOND READ

  REQUIRE(subject.nextReadPosition() == 512);
  samples = subject.read(outData);
  REQUIRE((int) outData[0] == 257); // LOWER BOUND THIRD READ
  REQUIRE((int) outData[255] == 257); // UPPER BOUND THIRD READ

  REQUIRE(subject.nextReadPosition() == 768);
}

TEST_CASE("Continous writing and reading does not exhaust the buffer (i.e. does not block)", "[audio ring buffer]") {
  AudioRingBuffer subject(128, 3);
  REQUIRE(subject.full() == false);

  auto data1 = new float*[2];
  data1[0] = new float[128];
  data1[1] = new float[128];
  for(int channel = 0;channel < 2;++channel) {
    for(int sample = 0;sample < 128;++sample) {
      data1[channel][sample] = 0;
    }
  }

  auto outData = new float[256];
  for(int i=0;i<100;++i) {
    subject.write(data1, 128);
    subject.read(outData);
  }
}


TEST_CASE("Data pointers get reset when wrapping", "[audio ring buffer]") {
  AudioRingBuffer subject(128, 3);
  REQUIRE(subject.full() == false);

  auto data1 = new float*[2];
  data1[0] = new float[128];
  data1[1] = new float[128];
  for(int channel = 0;channel < 2;++channel) {
    for(int sample = 0;sample < 128;++sample) {
      data1[channel][sample] = 0;
    }
  }

  auto data2 = new float*[2];
  data2[0] = new float[128];
  data2[1] = new float[128];
  for(int channel = 0;channel < 2;++channel) {
    for(int sample = 0;sample < 128;++sample) {
      data2[channel][sample] = channel * 128 + sample + 1;
    }
  }

  auto data3 = new float*[2];
  data3[0] = new float[128];
  data3[1] = new float[128];
  for(int channel = 0;channel < 2;++channel) {
    for(int sample = 0;sample < 128;++sample) {
      data3[channel][sample] = 257;
    }
  }
  REQUIRE(subject.nextWritePosition() == 0);
  subject.write(data1, 128);
  REQUIRE(subject.nextWritePosition() == 256);
  subject.write(data2, 128);
  REQUIRE(subject.nextWritePosition() == 512);
  subject.write(data3, 128);
  REQUIRE(subject.nextWritePosition() == 0); // Reset back to zero here!!!

  auto outData = new float[256];

  REQUIRE(subject.nextReadPosition() == 0);
  auto samples = subject.read(outData);
  REQUIRE((int) outData[0] == 0); // LOWER BOUND FIRST READ
  REQUIRE((int) outData[255] == 0); // UPPER BOUND FIRST READ

  REQUIRE(subject.nextReadPosition() == 256);
  samples = subject.read(outData);
  REQUIRE((int) outData[0] == 1); // LOWER BOUND SECOND READ
  REQUIRE((int) outData[255] == 256); // UPPER BOUND SECOND READ

  REQUIRE(subject.nextReadPosition() == 512);
  samples = subject.read(outData);
  REQUIRE((int) outData[0] == 257); // LOWER BOUND THIRD READ
  REQUIRE((int) outData[255] == 257); // UPPER BOUND THIRD READ

  REQUIRE(subject.nextReadPosition() == 0); // Reset back to zero here
}

Formatting Xml Files with xmllint

2017-05-10T06:31:00+00:00

I am currently working on a project where I need to parse Open Xml files comming from the Office Suite. As you may already know this involves opening what is essentially a zip file full of xml files.

When creating a parser though, having source files for referencing is very useful, and the saved files in these zip files are unformatted, i.e. they have no newlines or indenting - which making navigating them very painful.

After having opening a file I usually ran this vim command %!xmllint --format % which would run the file through xmllint, format it, and then replace the buffers content with the formatted version of the file.

This also worked well - however I keep opening new files, and keep forgetting to save after having formatted them... So that command got run a lot!

First try to solve it

I decided to just parse all xml files in one go after having opened the zip. I thought this to be as easy as:

find . -name "*.xml*" -exec xmllint --format {} \;

However this just output each file to stdout. And xmllint has no option to save the file in place.

Second try

Well - maybe we could just pipe the output back into the file.

find . -name "*.xml*" -exec xmllint --format {} > {} \;

But, alas, this did not work. The command now just did nothing. Maybe because you cannot reference the filename multiple times (turns out, you can...)

Third try

Ok, so I created a script to wrap the command in instaed.

#!/bin/bash
xmllint $1 > $1

and calling that

find . -name "*.xml*" -exec format.sh {} \;

But now all files gives us this error in xmllint:

parser error : Document is empty

So we are destroying the file, before xml lint is parsing.

Final try

Instaed of finding an elegant way to do this, I thought: Well if this happened to me, I would just format to a new file, the remove the original file, and rename the new file to the old name...

So that's what I did:

#!/bin/bash
xmllint --format $1 > $1.tmp
rm $1
mv $1.tmp $1

I then extended it a bit, and placed in my scripts folder:

#!/bin/bash
if  $1 == "all"  
then
  find . -name "*.xml*" -exec $0 {} \;
else
  xmllint --format $1 > $1.tmp
  rm $1
  mv $1.tmp $1
fi

and now I can just be in a unzipped folder of xml files and run

formatxml all

Using tcpdump to Spy on Webservers

2016-11-10T07:31:00+00:00

I recently had to debug an error where stuff that should have been returned through a server to server call did not work correctly. Both servers - in development mode at least - was on my local machine. But since I had to hack either an external dependency or the sending server to find out what was actually being sent between the servers - I realized this was a great time to utilize the tcpdump tool.

As always when I need to remember something about some of the unix tools I use too seldom, I googled for "Julia Evans", found her blog, and found a post about exactly what I had to do.

I had a oauth server running on port 3000, which should send some data on a route to my consumer app running on port 3001. For some reason the published gem (yes we are in ruby here) referenced data that was always nil.

I started tcpdump like so

sudo tcpdump -A port 3000 -i any

and went through the oauth login process. And sure enough I soon found the answer.

The data was in JSONAPI format, and the client expected it to be a raw json object. Quickly fixing the client gem with this new format, bumping the version, and releasing it - in less time than finding the bug by trying to do some puts-statement stuff instead.

So the next time you find yourself wondering what a given service sends/recieves via tcp, try out tcpdump - or visit Julia Evans blog for more tools you need to learn about.

How to Read a File in C#

2016-11-08T08:31:11+00:00

Csharp was probably the first compiled language I really loved. When the first version of .net came out I wrote a small CMS using C# and the new runtime. (A big step up from old school asp)

Reading a file - here as a stream - is easy. Setup a streamreader and call ReadLine on it, until it returns null.

string line;
int linenum;
System.IO.StreamReader file = new System.IO.StreamReader("test.csharp");
while((line = file.ReadLine()) != null) {
  linenum++;
  Console.WriteLine (linenum + " : " + line);
}

We prefix each line with a linenumber.

How to Read a File in Lua

2016-10-28T08:31:11+00:00

Reading a file in lua is pretty straight forward.

lines = {}

for line in io.lines(arg[1]) do 
  lines[#lines + 1] = line
end

for k,v in pairs(lines) do
  print(k .. ' ' .. v)
end

First create a map for the lines to be placed in. Then for iterate over the files lines using io.lines (we have no error checking here, we just assume it works). We use what we know about reading program arguments to get the filename from the command line.

We place each line in the map using the line number as key.

Finally we print out the pairs in lines having a space between the line number and the actual contents.

How to Read Program Arguments in Lua

2016-10-25T08:31:11+00:00

Lua is among the programming languages - that I write about here - I have used the least.

Reading program arguments is as easy as in most languages.

for i=1,#arg,1
do
  print(arg[i])
end

First we loop from 1 to the count of elements in arg (#arg).

In the do block we just print each argument.

I Wrote a Pong Game with My Son using Cocos Creator

2016-10-21T06:31:00+00:00

My son (Toke, 7 years old) and I have long talked about making our own computer game.

I have four reasons for doing so.

I would like him to know a bit about what I do all day
I would like him to be interested in games from a makers standpoint (learning instead of just consuming)
To spend quality time with my favourite son (yeah, he is the only son... but still)
I have always wanted to make a game

Inital work

I started looking at a few different ways we could make the game. If I had done it myself I would probably just have programmed it from scratch. But when having a 7-year-old on the project as well, I needed something that was a bit more visual.

I looked at different game engines and decided on the rather new Cocos Creator. (Mostly because I like the cocos2d-x library). I looked at a lot of different engines. But on a features/stability matrix only Cocos Creator seemed good enough (except maybe Unity and Unreal Engine - however they seemed like overkill)

First import the graphics

I cheated a bit and googled for paddles, balls and backgrounds. (And must admit we have not checked the copyright of the images we "stole" - but since this is never published anywhere I think we are safe)

The first thing we did together was import the graphics into the engines editor, and began placing and naming the parts.

Make something move

Then we decided to make the ball move.

I introduced my son to code writing - and created a script, and attached it to the ball as a component.

cc.Class({
    extends: cc.Component,

    properties: {
       speed: 200,
       x_direction: 1,
       y_direction: 1
    },

    onLoad: function () {
    },

    update: function (dt) {
        this.node.x += dt * this.speed * this.x_direction;
        this.node.y += dt * this.speed * this.y_direction;
    },
});

Bounce the ball of the walls

Next up we introduced collisions. Otherwise the ball would just go off the screen and go the same way forever.

We created a 1px box collider on all walls, and a circle collider on the ball. We tagged the top and bottom wall with one tag, the left wall with another and the right wall with yet another tag.

Then we changed the script for the ball.

cc.Class({
    extends: cc.Component,

    properties: {
       speed: 200,
       x_direction: 1,
       y_direction: 1
    },

    onLoad: function () {
        cc.director.getCollisionManager().enabled = true;
    },

    onCollisionEnter: function(other) {
        this.speed += 20;
        // Top or bottom wall
        if(other.tag == 1) {
            this.y_direction *= -1;
        }
        // Left or right wall
        if(other.tag == 2 || other.tag == 3) {
            this.x_direction *= -1;
        }
        // Anything else (in this case, just the paddles)
        if(other.tag == 0) {
            this.x_direction *= -1;
        }
    },

    // called every frame, uncomment this function to activate update callback
    update: function (dt) {
        this.node.x += dt * this.speed * this.x_direction;
        this.node.y += dt * this.speed * this.y_direction;
    },
});

Now we have ball bouncing around, and increasing it's speed each time it hits something

Next up: Make it a game

Next up we added scores for each player - they are just labels initialized to zero. We created a new script and attached it to the root node.

This script has links to almost everything else in the game, and is now also responsible for instancing a new ball.

Instancing the ball

We changed the ball, with it's attached script to a prefab, which is instanced each time the game starts, and each time a wall is hit.

cc.Class({
    extends: cc.Component,

    properties: {
        ball: {
            default: null,
            type: cc.Prefab
        },
        root: {
            default: null,
            type: cc.Node
        },
        player1Score: {
            default: null,
            type: cc.Label
        },
        player2Score: {
            default: null,
            type: cc.Label
        },
        player1ScoreValue: 0,
        player2ScoreValue: 0
    },

    getRandomPosition: function() {
        return cc.p(cc.randomMinus1To1() * this.randomRange.x, cc.randomMinus1To1() * this.randomRange.y);
    },

    createBall: function () {
        this.current_ball = cc.instantiate(this.ball);
        this.current_ball.parent = this.root;
        var movementComponent = this.current_ball.getComponent('Ball');
        movementComponent.game = this;
        movementComponent.speed = cc.p(250, 500).x;
        movementComponent.x_direction = cc.randomMinus1To1();
        movementComponent.y_direction = cc.randomMinus1To1();
        this.current_ball.position = this.getRandomPosition();
    },

    destroyBall: function() {
      this.current_ball.destroy();
    },

    updateScore: function() {
        this.player1Score.string = this.player1ScoreValue;
        this.player2Score.string = this.player2ScoreValue;
    },

    hitLeftWall: function() {
        this.player2PointValue++;
        this.destroyBall();
        this.createBall();
        this.updateScore();
    },
    hitRightWall: function() {
        this.player1PointValue++;
        this.destroyBall();
        this.createBall();
        this.updateScore();
    },

    onLoad: function () {
        this.randomRange = cc.p(300, 200);
        this.createBall();
    },
});

and then we updated the balls script to call the game script whenever we hit a wall.

    onCollisionEnter: function(other) {
        this.fart += 20;
        if(other.tag == 1) {
            this.y_retning *= -1;
        }
        if(other.tag == 2) {
            this.game.hitRightWall();
        }
        if(other.tag == 3) {
            this.game.hitLeftWall();
        }
        if(other.tag == 0) {
            this.x_retning *= -1;
        }
    },

and finally we added a script to each of the paddles to allow them to move.

Left paddle

cc.Class({
    extends: cc.Component,

    properties: {
        direction: 0,
        speed: 250
    },

    // use this for initialization
    onLoad: function () {
        cc.eventManager.addListener({
            event: cc.EventListener.KEYBOARD, 
            onKeyPressed: this.onKeyPressed.bind(this),
            onKeyReleased: this.onKeyReleased.bind(this),
        }, this.node);

    },

    onKeyPressed: function(keyCode, event) {
        switch(keyCode) {
          case cc.KEY.w:
            this.direction = 1;
            break;
          case cc.KEY.s:
            this.direction = -1;
            break;
        }
    },

    onKeyReleased: function(keyCode, event) {
         switch(keyCode) {
          case cc.KEY.w:
          case cc.KEY.s:
            this.direction = 0;
            break;
        }
    },

    update: function (dt) {
        this.node.y += this.direction * this.speed * dt;
    },
});

The right paddle is exactly the same - except that the keys used there is "up" and "down" instead of "w" and "s".

Conclusion

We had fun. But maybe my son had most fun trying to beat me at pong when we actually playtested the game.

Finally: A screenshot.

How to Read a File in C++

2016-10-04T08:31:11+00:00

Reading a file in C++ is almost identical to how it's done C.

#include <iostream>
#include <fstream>
int main()
{
  std::ifstream in("test.cpp");

  std::string line;
  auto num = 0;
  while (std::getline(in, line)) {
    num++;
    std::cout << num << ": " << line << std::endl;
  }

  in.close();

  return 0;
}

First we load the content of a file into a filestream. Next we iterate over each line, putting that into a string variable called line.

For each line in the file we print out the line prefixed with a line number to stdout.

And finally we close the filestream, and exit the program

How to create a plugin system in C++

2016-10-04T06:31:00+00:00

To create a plugin system in C++ you need three things.

An interface for the plugin
A way for the plugin to inject itself into your program
A way for the program to load the plugins

The interface

The first two parts are handled together in this case. By having an C++ interface for the plugin you are going to write, and a simple macro you need to call with the type, the name and the version of your plugin to create the correct plumbing for your program to load the plugin correctly.

#include <string>
class Plugin {
  public:
    Plugin() {};
    virtual ~Plugin() {};
    virtual std::string command(std::string command, std::string options) {return "";}
};


#define DEFINE_PLUGIN(classType, pluginName, pluginVersion)     \
  extern "C" {                                                  \
    std::shared_ptr<Plugin> load()                              \
    {                                                           \
      return std::make_shared<classType>();                     \
    }                                                           \
                                                                \
    const char* name()                                          \
    {                                                           \
      return pluginName;                                        \
    }                                                           \
                                                                \
    const char* version()                                       \
    {                                                           \
      return pluginVersion;                                     \
    }                                                           \
  }

Header guards have been omitted. But other than that this would be a complete header file for a plugin class with only one public method called command.

We have used std::string here, that should work fine as long as you use the same compiler for the main program and the plugins.

The macro, when called, will create three functions in a extern "C" block. One for loading a plugin, which just calles it's default constructor and returns a shared pointer, one for getting the name, and one for getting the version.

Define Plugins

Next up we'll create a plugin implementing this interface, and calling the macro.

#include <plugin.h>

class MyPlugin : public Plugin {
  public:
    virtual std::string command(std::string command, std::string options) {
      return command + " " + options;
    }
};

DEFINE_PLUGIN(MyPlugin, "Simple Plugin", "0.0.1")

This defines a simple plugin which just echoes it's command and options back. But you really could do anything in the plugin.

You could, for instance, create extra methods on the plugin. They would only be callable by your code - but you are not confined to only using the methods in the interface.

#include <plugin.h>

class MyPlugin : public Plugin {
  public:
    std::string get_value() {
      return "THIS IS INTERNAL TO THE PLUGIN";
    }

    virtual std::string command(std::string command, std::string options) {
      return get_value();
    }


};

DEFINE_PLUGIN(MyPlugin, "Plugin with extra methods", "0.0.1")

You could also write a plugin which calls out to other classes, and that would still work, as long as your application only knows about the interface.

Loading Plugins

To load plugins you - again - need the interface for the plugin, as well as the dlfcn header (for unix like systems). To get this working on windows you need to make some changes to both the plugin.h file, and the loader.

#include <dlfcn.h>
#include <plugin.h>
class PluginHandler {
  std::shared_ptr<Plugin> (*_load)();
  void* handle;
  char* (*_get_name)();
  char* (*_get_version)();

  std::shared_ptr<Plugin> instance;
  public:

  PluginHandler(std::string name) {
    handle = dlopen(name.c_str(), RTLD_LAZY);
    _load = (std::shared_ptr<Plugin> (*)())dlsym(handle, "load");
    _get_name = (char* (*)())dlsym(handle, "name");
    _get_version = (char* (*)())dlsym(handle, "version");
  }

  std::string get_name() {
    return std::string(_get_name());
  }

  std::string get_version() {
    return std::string(_get_version());
  }

  std::shared_ptr<Plugin> load() {
    if(!instance)
      instance = _load();
    return instance;
  }

};

First we define the functions that we want to load from the plugin, and then we make a constructor that opens the plugin using dlopen, which returns a handle to the load plugin. Using this handle we can load the addresses of the functions we want to import into our program, and map them to our defined functions.

The rest of the class is just delegating calls to the imported functions, and here the load method is the most intersting, since it is the one that actually instantiates a plugin object, and returns the shared pointer to the caller.

Using it

If you want to use this you only need to create a new plugin handler with one param: the path of the plugin.

PluginHandler ph("path/to/a/plugin.dylib");

And then use that to load the actual object.

std::shared_ptr<Plugin> plugin = ph.load();

Then you will be able to call the plugins command method.

Example

Following is an example of a program that loads plugins dynamically. The code that loads the plugins is rather naive. It assumes you have a plugins/bin directory in the working directory, and the only contents of that directory is valid plugins. (Otherwise the program might just crash, and do horrible stuff).

Then for each of the plugins loaded it will print the name and version of it, and call the command method, and print the result of that.

#include <dirent.h>
#include <plugin.h>
#include <iostream>
#include <vector>
#include "./plugin_handler.hpp"

std::vector<PluginHandler> load_plugins() {
  std::vector<PluginHandler> plugins;

  DIR *dir;
  struct dirent *ent;
  if ((dir = opendir ("plugins/bin")) != NULL) {
    while ((ent = readdir (dir)) != NULL) {
      if(ent->d_name[0] != '.')
        plugins.push_back(PluginHandler("plugins/bin/" + std::string(ent->d_name)));
    }
    closedir (dir);
  }
  return plugins;
}

int main(int argc, char *argv[])
{
  auto plugins = load_plugins();
  for (auto ph : plugins) {
    auto plugin = ph.load();
    std::cerr << "Auto loaded plugin: " << ph.get_name() << ", version: " << ph.get_version() << std::endl;
    std::cerr << "Running plugins command method: " << std::endl;
    std::cerr << plugin->command("Command here", "options here") << std::endl;

  }
   return 0;
}

How to parse JSON in Rust

2016-09-19T08:31:11+00:00

To parse json in Rust you need a library. The one used here is called rustc-serialize.

The first step is to create a new project.

cargo new test_json
cd !$

And then include the dependency in the Cargo.toml file.

[package]
name = "test_json"
version = "0.1.0"
authors = ["Claus Witt <claus@wittnezz.dk>"]

[dependencies]
rustc-serialize = "0.3"

Next we will create a json file to parse.

{
  "Person": {
    "Name": "Claus Witt"
  }
}

Finally we create a main.rs file.

extern crate rustc_serialize;
use rustc_serialize::json::Json;
use std::fs::File;
use std::io::Read;

fn main() {
    let mut file = File::open("test.json").unwrap();
    let mut data = String::new();
    file.read_to_string(&mut data).unwrap();

    let json = Json::from_str(&data).unwrap();
    println!("{}", json.find_path(&["Person", "Name"]).unwrap());
}

First we link our file with the dependency. Next we "use" the external library objects we need.

In the main method we open the file, and read its data into a string, and finallyy using Json::from_str parse the json. We use unwrap to tell the rust runtime that we want the value, or exit the program.

Last, but not least, we print the content of the path Person.Name

How to parse JSON in C++

2016-09-16T08:31:11+00:00

In C++ there are no built-in ways to handle json. However there are many open source libraries to help you.

One of the more popular ones is JsonCpp - and it is relatively easy to use.

Include jsoncpp.cpp in your build, and include json.h where you want to handle json.

std::ifstream json_file("person.json", std::ifstream::binary);
Json::Value root;
Json::Reader reader;
bool parsingSuccessful = reader.parse( json_file, root, false );
std::string = root["name"].asString();
int age = root["age"].asInt();

First we [open the file]() - and then define a variable to hold the root value of the parsed json.

Next we setup the reader, and parse the file. The resulting Json::Value is quite nice. You can get sub keys by the operator[] method (like the example above) and call asInt, asFloat, asBool and asString on the result. Or you can iterate over it, if it is an array.

How to read a file in Java

2016-09-13T08:31:11+00:00

Java is - off course you might joke - rather verbose when it comes to reading files. However most of the verbosity has to do with importing the functionality from the standard library.

We need three things imported, Files, Paths and then the IOException that may get thrown.

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;
class Test {
  public static void main (String[] args) {
    int i = 0;
    try {
    for (String line : Files.readAllLines(Paths.get(args[0]))) {
      i++;
      System.out.println(Integer.toString(i) + " " + line);
    }
    } catch (IOException ex) {
    }
  }
}

Java forces us to handle the exception that might get thrown, so we import the exception type, and wrap our example code in a try/catch block that handles that Exception (by doing nothing: don't do this in production).

The example then is rather simple. We use Paths.get to convert the first program argument to a path object, that can be sent to Files.readAllLines - which does exactly what it says on the box: return all lines of the input file, as a List<> of strings.

We then iterate over this list, printing it to stdout prefixed by a line number.