AntonLindstrom.com

What I like about estimations

2018-02-18T00:00:00+00:00

Sometimes when you step into a room saying, "have you tried doing estimations?" you get laughed at. The people in that room does not like estimations, thinks it is a horrible idea and wants you to leave.

Everyone seems to have opinions about estimations in Scrum. Many of the opinions are strongly held and most of the time they are negative. However, I have come to enjoyed the process of estimating. Not because the estimation is ever correct but because of the discussion that arises when someone estimates a task to 24 and another to 8.

In software engineering, we need to make the collaboration and communication work as good as possible and the discussions around how we estimate tasks are valuable to get insight into how people perceive the complexity of the system and if there are unknowns that another team member has not thought about. Does the task need further specification? Is the task to open ended?

The main thing that I think that you receive from an estimation is: "Did we miss anything?"

When using estimation it is good to always discuss the task that is proposed for a while to get a feeling of if someone is unsure of how something is implemented or how it is going to be implemented. After that is done, checking to see if the estimated numbers differ, if they differ, ask what it is about and what the people with the most different numbers perceive as the biggest hurdles and what the task means to them.

If the numbers estimated are really high, decide if it is too big and too loosely specified. If the task is too big, it have to be further specified and further broken down into smaller parts so that the parts can be parallelised and easier to get done in a reasonable time.

To wrap up, I really enjoy working with estimations since they provide value in the discussions and also spread knowledge between team members. Estimations may be wrong most of the time but the format provides a framework to discuss and learn from team members.

Go func: Reducing cognitive load

2017-06-29T00:00:00+00:00

I have been maintaining several fairly large Go projects for a while. Some of these projects have been for web services (APIs as well as backend workers) and some have been command line tools. A few of these projects have been new projects (green field) where I led the design and implementation, some have been taking over maintenance and some have been projects where I contributed.

One major thing that I have learned across these projects is that even though the code format (go fmt) is set, the layout and design have been quite different. This is perfectly fine where the requirements differ. However I want to cover some things that I have noticed that makes it easier for developers to pick up and read code.

This post will cover a part that makes it hard for me to understand where configuration comes from. Reading the code path is essential for me to understand how a program functions and how to debug it.

Good code for me is code that is easy to read, that is the number one property. Writing abstractions to make it easier to write code fast and efficiently becomes unimportant when comparing to readability.

To reduce cognitive load, I want to be able to read a function/method without having to think about where some variable comes from. The input and output should be well defined (essentially a pure function). Let me give an example:

package example

import "config"

func hasSomeFile() bool {
    if _, err := os.Stat(config.MyFile) {
        return os.IsNotExist(err)
    }

    return false
}

The most common pattern I have seen is that a config package is defined and is imported in many different packages, changed by either flags or somewhere else (or in the worst of cases, both).

When I read the code above, my first question is: What is config.MyFile?

From this example, I can probably check the value and type from the editor. However, this adds extra cognitive load. Is it set manually from a flag or file?

One example of getting rid of these questions is to define the signature to get rid of the config package and call the function with the filename as an argument. This way we know that we have to provide the value ourselves:

package example

func hasSomeFile(file string) bool {
    if _, err := os.Stat(file) {
        return os.IsNotExist(err)
    }

    return false
}

The documentation will be more clear and we know what to call the function with.

Conclusion

In order to reduce cognitive load and make it easier for new developers to understand and read your code, be explicit of what you use in your signatures. Code readability should be your top priority.

Edit: This post has been updated to provide better examples.

Year 2016 in review

2017-01-03T00:00:00+00:00

While 2016 seems to have been a year that I have not written a single blog post a lot of stuff happened on a personal and professional level. Here are a few words on what I did last year.

I moved my infrastructure to Kubernetes. This was a decision that was made due to the fact that I could not keep up with the development of the Mesosphere Marathon API for my internal tools and after trying Kubernetes it was a better fit for what I was doing. kubectl has helped in more ways I thought was possible.

Since I write a lot of Go at work, I feel that I have become a better Go programmer. Hopefully this will reflect on this blog so that you will get some good things about Go here.

On a personal level, we bought a house in beautiful Gothenburg. This has meant that work at home has shifted into doing other things.

I also read a few books.

Limit processes with cgexec

2015-11-22T00:00:00+00:00

When running multiple processes it can sometimes be useful to limit the resource usage of one or many process groups. In this post I'll describe how to limit the CPU and memory of a process with the help of Linux Control Groups.

Say that you're compiling a program or doing some heavy computing with a program which uses 100% of all CPUs available. If you now were to run another program, for instance checking your mail, it would be really slow. This is where resource limiting would come into use.

To be able to use resource limiting with cgroups I usually use the cgroup-bin package. The following commands will limit the process ./compute to only use 128 (of a total of 1024) CPU shares and 32MB of memory:

# Install cgroup-bin
apt-get install cgroup-bin

# Create a cgroup for memory and cpu called "mygroup"
cgcreate -g memory,cpu:mygroup

# Add 128 CPU shares (about 12% CPU).
echo 128 > /sys/fs/cgroup/cpu/mygroup/cpu.shares

# Add 32MB of memory to the limit.
echo 32000000 > /sys/fs/cgroup/memory/mygroup/memory.limit_in_bytes

# Do not allow the process to use swap (if the limit is reached the
# process will be killed instead).
echo 0 > /sys/fs/cgroup/memory/build/memory.swappiness

# Run the command in the specified group.
cgexec -g cpu,memory:mygroup ./compute

The above commands describe how to limit resources for a command which I'm using quite often to limit commands such as compiling (./configure && make) to be able to keep some extra resources for processes such as email and IRC.

To be able to write add a non root owner of a control group, the following flags are useful:

-t user:group # User that can add tasks to the cgroup.
-a user:group # User that can add and modify tasks in the cgroup.

# Create a cgroup for user anton in group anton which is able to use and
# modify the cgroup "mygroup".
cgcreate -t anton:anton -a anton:anton memory,cpu:mygroup

There are a lot of other things that can be done with Control Groups such as monitoring and introspection. I recommend reading in on cgroups to learn more about it.

Links

To read more about cgroups, the following links helped me understand more about it:

Introduction to Apache Mesos

2015-03-29T00:00:00+00:00

Apache Mesos has been an interest of mine for quite some time and the potential of the Mesos project and the ecosystem is huge. The project has gotten a lot of attention from bigger companies such as Amazon, eBay and Netflix. Mesos has also been sponsored and used by Twitter for quite some time.

Introduction and concepts

Mesos is a cluster manager, handling workloads in a distributed environment. For instance, to increase resource allocation in clusters, Mesos provides dynamic allocation of resources which means that you won't have to allocate one machine for Hadoop, one for the web server and another one for a database. All the resources that exists on the machines in the cluster will be put in a single pool which everything you would like to run on it can draw from. The database, Hadoop jobs and web server may run on any of the machines that have resources available.

A machine in the cluster may have multiple processes running and to keep them from interfering with each other, Mesos provides isolators that keep processes from disturbing each other. Examples of isolators are CGroupsMemIsolator and NetworkIsolator. A disk qouta isolator was introduced in Mesos 0.22.0.

There are also containerizers that implements isolators to add a container around a process. The most prominent examples are Cgroups and Docker. Docker may be used to isolate and run tasks and this is a model that simplifies the operability of a large scale infrastructure that runs in Docker.

Mesos has several components and the three most important bits are the masters, slaves and frameworks. The masters work to coordinate and manage the slave daemons. It's the master that decides how many resources it has to offer to the framework. The master knows how much resources it can offer by the amount that the slaves report to have freely available.

A master is a singular entity that coordinates the work, there are however multiple masters in a high availability set up. Only one master should be leader at any time and Zookeeper is being used for leader elections.

In the diagram above, Framework A wants to run a process (in Mesos called task) on the cluster. It first sends a request to the Master which in turn will get the resource information from the slaves. The master will then send an offer to Framework A and allow it to run on the slave that has resources available, in this case Slave 1.

In the architecture diagram, the following occurs:

The slave sends the amount of resources it has available to the master.
The master may then send out an offer with resources to the framework which it may accept or reject.
If the framework accepts the offer, it sends a list of tasks it wants to run on the Mesos cluster.
Tasks are being forwarded to the slave which runs the framework executor that will execute the tasks in the list received from the master.

The roles of masters and slaves can be summarized by the following:

Masters coordinate the work and gives it to the slave that has the resources to do it.
Slaves do the work and report how much more work they can do.

Frameworks

The frameworks is probably the component that you as a user will interface with the most. The framework is responsible for a few different things and there are a few different parts to the framework as well. Frameworks must consist of at least two things, a scheduler and an executor.

The scheduler is responsible for registering to the Mesos master and handles the offers. The executor is a program or command on the slaves which runs the tasks. If you have some constraints on your tasks that should be run, for instance you only want to run task X in datacenter Y, this is done in the framework. The scheduler in the framework knows the most about your task and may deny any offer it receives. To be able to know which slaves are in datacenter Y, Mesos slaves can be started with attributes. This is a key/value property list that are added to the slave. In the framework it's then possible to deny offers that doesn't have the datacenter Y attribute.

When the framework accepts an offer it sends the details of a task back to Mesos which will dispatch it to the slave.

There are several big frameworks at the time of this writing. The major ones being Apache Aurora, Apache Spark, Chronos and Mesosphere Marathon. Apache Aurora and Mesosphere Marathon aims both to be used with long-running services, for instance running your web server. For batch jobs, Apache Spark provides both standalone and Mesos powered cluster computing. Chronos does distributed cron.

There are many more frameworks out there, for instance to run Cassandra and Elasticsearch on Mesos. The high availability and self healing capabilities of Mesos makes it perfect to manage resources in the datacenter. See the framework as the application that runs on the Mesos cluster.

Framework schedulers take responsibility of offers to the Mesos master, the offers can be accepted or denied.
Framework executors run commands on the slaves.

Operating Apache Mesos

Most people reading this will probably come in contact with Mesos by operating it. Some of the people I've talked to have been a bit afraid of running and operating Mesos in their infrastructure. Most of the time this seems to be the lack of understanding about how the system works and not trusting the "magic" of how tasks are being run.

Key points that's the strength of the Mesos system:

Fault tolerance, when set up in high availability any individual part can break without bringing down the system.
Almost everything can be done from a web browser, no need to give everyone in the organization access to servers.
Using frameworks it's very easy to deploy applications.

I've got a very small cluster that I've been using for some time now. The entire time the cluster only went down partially one or two times. The downtime was caused by bugs in the Mesosphere Marathon framework which was due to a release that wasn't 100% mature and caused deploys to be locked. Currently I've upgraded the components successfully a few times and it's still stable and no downtime. As my cluster is very small, the failures will cause more impact (too many tasks and some won't get offers if a slave goes down).

The operability of Mesos itself is according to my experience good. It has been easy to set up and run, I spend less time on the infrastructure now than I did before I implemented it. Creating new tasks to try something out is very fast, thanks to Mesos, Marathon and Docker. The hardest part to operate is in my experience, Zookeeper. Since operating Mesos doesn't involve doing any particular operation directly to Zookeper yourself, it works very well.

As Twitter has been using Mesos for quite some time and are running a huge amount of their infrastructure on it, it's both battle-tested and proven stable in a demanding infrastructure.

Conclusions

Apache Mesos is a cluster manager that's robust, battle tested and designed for failures. The cluster will simplify and save money with dynamic allocations and you won't have to dedicate hardware to different systems. To provide security, Mesos will use isolators to keep processes from interfering with each other.

When a system is set up, it will need Zookeeper, Mesos master processes, slave processes and a framework. Masters coordinate the work and decides what resources to give the frameworks. The framework then decides if it wants to accept or reject an offer with resources. If the framework decides to accept the offer, it sends a list of tasks it wants to run to the master which then sends it through to the slave which will run the executor defined by the framework.

Monitoring Mesos tasks with Prometheus

2015-02-24T00:00:00+00:00

I run an infrastructure for my own hobby projects and tests. The environment is currently running on DigitalOcean with Mesos and Mesosphere Marathon on top which runs all applications. The need for simple yet powerful monitoring without too much setup, maintenance and configuration has been high in that environment. I wanted to both know when applications failed or what the load and the performance of the system was.

My previous monitoring system comprised of a few different ones, I used a simple Uptime Robot to collect external alerts and InfluxDB with a custom dashboard and metrics collector to look at performance graphs. That system was unfortunately not good enough. It got complex and the automatic deployment of it was not configured properly, it did not get enough attention to make it work as good as it could've been.

Enter the crystal ball: Prometheus. It got some attention lately and I decided to try it out. The easiest way to install it is via Docker which is listed on their installation page. I hooked that up to my deployment system and started looking at it. I was met by a simple but powerful configuration, query language and user interface. However, once started I needed a way to get some useful data into it.

I knew about the /monitor/statistics.json endpoint for Mesos slaves and have been using that endpoint successfully a few times to find out CPU or memory usage for running tasks. The endpoint would be a good place to fetch information from with a simple program that does a GET request, parses the data and displays it in the format Prometheus wants.

To get Mesos data into the system it was as easy as building a small Go app that used the client provided by Prometheus and serve HTTP. The Mesos exporter can be found on my Github and any feedback to get it better is welcome. The installation instructions are available in the README.

After I had built the exporter, the Prometheus configuration was just a few lines to add the exporter endpoint and graphs started showing.

Previously, my relationship towards monitoring systems have always been a bit on the complicated side. I love Sensu when I have machines running entire stacks but with Docker, it is not a perfect fit. If I use Puppet, Sensu also requires a lot of infrastructure to be set up in a small environment where there is high velocity but not that much resources it is far from ideal.

Prometheus sits right in the perfect spot where the simplicity and the extensibility of exporters makes it easy to run. That it also comes packaged in a Docker container makes it even more so.

Alerting in Prometheus is done on the data collected which is then sent off to another system, alertmanager. Simple and very powerful.

So far Prometheus has been everything I wanted and provides a really good way to monitor my infrastructure and alert on any anomalies in it. Once you start learning the query language, it is amazing how intuitive it feels.

A query against the data in Prometheus to find the top 3 highest consumers of CPU in Mesos for example, looks like this:

topk(3, sum(rate(mesos_task_cpus_user_time_secs[5m])) by (app))

TLDR: Mesos and Prometheus are great, here's a mesos_exporter to get Mesos task statistics into Prometheus.

Note: Image is probably copyrighted and owned by 20th Century Fox.

Operating systems: Introduction to processes

2014-12-15T00:00:00+00:00

I've wanted to write this for a long time but always had an excuse not to. Operating systems are a big part of my every day work, especially GNU/Linux which will be the focus of this post.

The topic of processes is huge so I'm not sure how to cover all of the pieces. This post will hopefully contain enough code so you'll be able to understand how to interact with a process. These code examples will focus on GNU/Linux as it is the operating system I'm most familiar with.

So, what is a process? The Linux Information Project defines a process as "an executing (i.e., running) instance of a program". So, to define a process we need to define what a program is. Again, according to The Linux Information Project, "A program is an executable file that is held in storage".

So, what we know is that a process is some part of a program that is running. Does that mean that a process must be running? Well, not quite.

Process states

In order to understand what it means that a process is running, we need to visit the different states of a process. A process can have several states (in the Linux kernel, a process is sometimes called a task).

In the file fs/proc/array.c the following is defined:

/*
* The task state array is a strange "bitmap" of
* reasons to sleep. Thus "running" is zero, and
* you can test for combinations of others with
* simple bit tests.
*/
static const char * const task_state_array[] = {
        "R (running)",          /*   0 */
        "S (sleeping)",         /*   1 */
        "D (disk sleep)",       /*   2 */
        "T (stopped)",          /*   4 */
        "t (tracing stop)",     /*   8 */
        "X (dead)",             /*  16 */
        "Z (zombie)",           /*  32 */
};

Running does not however mean that the process is running, it denotes that the process is either running or on the run queue. Sleeping means that the process is waiting for an event to complete (the sleep here is also sometimes called interruptible sleep). Disk sleep is sometimes called uninterruptible sleep, and the process is usually waiting for IO to finish.

A process can be stopped (T) by sending a process a SIGSTOP signal. This pauses the process and can be continued by sending the SIGCONT signal.

For example, stopping and continuing a process can be done in the following manner:

kill -SIGSTOP <pid>
kill -SIGCONT <pid>

Tracing stop can be done by using gdb to stop a process. If I recall correctly, this state is basically the same as stopped state.

The dead state is a state that is returned when the kernel is running the do_exit() function in kernel/exit.c. This is just to return a status but the state should not be seen in your task list.

Zombies is a state that is a bit peculiar. Some think of it as a state that happens when the process parent dies and the child process is left. This is not the case. The parent may die but the child could still live on, the parent process of that child will be the init process, pid 1. A zombie process occurs when the process exits and the return code hasn't been read by the parent process (using the wait() system call). It remains in the process table as terminated but is waiting for the parent to read the exit status.

Here's a simple example that creates a zombie process for 30 seconds:

#include <stdio.h>
#include <stdlib.h>

/*
* A program to create a 30s zombie
* The parent spawns a process that isn't reaped until after 30s.
* The process will be reaped after the parent is done with sleep.
*/
int main(int argc, char **argv[])
{
        int id = fork();

        if ( id > 0 ) {
                printf("Parent is sleeping..\n");
                sleep(30);
        }

        if ( id == 0 )
                printf("Child process is done.\n");

        exit(EXIT_SUCCESS);
}

The post Linux process states is an excellent post describing the process states with code examples and ptrace to control it.

What does a process contain?

I briefly mentioned the process table, here I'll explain what it is. A process table is a data structure in the Linux kernel that is loaded into RAM and contains information about processes.

Every process has information in the data structure, task_struct and contains amongst other:

State (task state, exit code, exit signal..)
Priority
PID
PPID
Children
Usage (cpu time, open files..)
Tracing information
Scheduling information
Memory management information

The data structure holding the process information is called task_struct and can be found in include/linux/sched.h. All processes that are running in the system are represented in the kernel as a linked list of task_struct.

Information about a process can be queried via the /proc system. To get information about the process with pid 400, you should be able to look into the /proc/400 directory. Most of the information can also be found using user land tools such as top and ps.

Process execution

When a process is executed, it is loaded into the virtual memory, allocates the space for program variables and adds the information into the task_struct data structure.

The process contains a memory layout of four different segments:

Text, contains source instructions of the program
Data, contains static variables
Heap is the area for dynamic memory allocations
Stack is of dynamic size and grows and shrinks as the process is running, this is the storage for local variables.

There are two ways to create a process, fork() and execve(). These are both system calls but works slightly different.

To create a child process the fork() system call can be executed. The child process then inherits a copy of the parent data, stack and heap memory segments. The child process can then modify these segments independently. The text segment is also shared with the child process but can not be modified.

A new process is created with execve(). This system call destroys all the memory segments to create new ones. execve() does however take an executable or script as an argument which is also different from fork(). execve() does however take an executable or script as an argument which is also different from fork().

Note that both execve() and fork() creates a process that is a child process of the executing process.

There's a lot of more to process execution than this. There's scheduling, permissions, resource limits, library linking, memory mapping.. However, this post will unfortunately be too long to cover everything. Perhaps this will be something to revisit later on.

Interprocess communication (IPC)

For processes to communicate with each other, a couple of methods exist such as shared memory or message passing.

In the case of shared memory, a shared region is created so that several processes can communicate. The region can then be accessed simultaneously by multiple processes. This is commonly used when working with threads. This is the fastest form of IPC because it's only writing and reading memory involved. However, this requires the processes involved to agree on accessing the memory segment as restrictions on accessing other processes memory is implemented by the kernel.

Shared memory segments in use can be found with the command ipcs -m.

Implementing a server for shared memory, looks something like this:

#include <stdlib.h>
#include <stdio.h>
#include <sys/ipc.h>
#include <sys/shm.h>

#define SEGMENT_SIZE 64

int main(int argc, char **argv[])
{
        int shmid;
        char *shmaddr;

        /* Create or get the shared memory segment */
        if ((shmid = shmget(555, SEGMENT_SIZE, 0644 | IPC_CREAT)) == -1) {
                printf("Error: Could not get memory segment\n");
                exit(EXIT_FAILURE);
        }

        /* Attach to the shared memory segment */
        if ((shmaddr = shmat(shmid, NULL, 0)) == (char *) -1) {
                printf("Error: Could not attach to memory segment\n");
                exit(EXIT_FAILURE);
        }

        /* Write a character to the shared memory segment */
        *shmaddr = 'a';

        /* Detach the shared memory segment */
        if (shmdt(shmaddr) == -1) {
                printf("Error: Could not close memory segment\n");
                exit(EXIT_FAILURE);
        }

        exit(EXIT_SUCCESS);
}

By substituting *shmaddr = 'a'; with printf("Segment: %s\n", shmaddr) you will get a client instead and be able to read the data in the shared memory segment.

Running ipcs -m will output the following information about the segment set with the server:

anton@shell:~$ ipcs -m

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x0000022b 0          anton      644        64         0

The segment can be removed with ipcrm. To learn more about implementing shared memory IPC read Beej's fantastic guide, Shared memory segments.

Other approaches to IPC are files, signals, sockets, message queues, pipes, semaphores and message passing. I'm not able to dive into all of the approaches but I think that signals and pipes should provide some interesting examples.

Signals

In process states, we saw an example of signals with the help of kill. A signal is a software interrupt that informs processes of events or exceptions that occurs.

A signal is identified by an integer but is often described with SIGXXX, for example SIGSTOP or SIGCONT. Signals are used by the kernel to inform processes of events but can also be sent from a process with the kill() system call. A process that receives a signal may ignore it, be killed by it or be suspended by it. It is possible to handle signals via a signal handler and the process may do whatever it pleases when the signal occurs. The special signal SIGKILL cannot be trapped (handled), this is used when killing for example a hung process. SIGKILL should not be confused with SIGTERM which is sent by default when using Ctrl+C or kill <PID>. SIGTERM doesn't forcibly kill the process and the signal can be trapped and often a process is allowed to clean up.

Pipes

A pipe is used to connect one process output to another process input. This is one of the oldest methods of IPC. An ordinary pipe is a one-way communication, it has a unidirectional flow. A pipe can be created with pipe() and is similarly to other objects in Linux, treated as a file. The read() and write() operations apply to pipes as well as files.

Named pipes is an improvement of ordinary pipes, the communication can flow bidirectionally and several writers and readers can use the pipe. This is not possible in ordinary pipes. Named pipes can also exist even if no writers or readers are using it. The named pipes are created as a special device in the file system, in GNU/Linux named pipes are also referred to as FIFOs (First In First Out).

Here's an example of creating a named pipe:

#include <stdlib.h>
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>

int main(int argc, char **argv[])
{
        if (mknod("myfifo", S_IFIFO|0666, 0) == -1) {
                printf("Failed to mknod\n");
                exit(EXIT_FAILURE);
        }

        exit(EXIT_SUCCESS);
}

In the executing directory, we'll see the file myfifo. It will look something like the following:

prw-rw-r--  1 anton anton    0 Dec 16 16:14 myfifo

That was the basic introduction into processes. The more I started to write the more I realized that there's so much to cover. I had a hard time knowing where to start and also where to draw the line of what not to cover. Shared memory segments is something I haven't done that much of and it was really fun to revisit that part of interprocess communication. Also, by having a lot of good resources such as The Linux Programming Interface and Operating System Concepts made it easier to get back into the concepts.

References

The following resources has been used in order to gain more understanding of the field. If you want to learn more about operating systems, make sure to check these books out, they are pretty thick but a good read.

The Linux Information Project
The Linux Programming Interface, chapter 6
Operating System Concepts, chapter 3

Visual block inserting in Vim

2014-12-04T00:00:00+00:00

I've recently started using the visual features of the Vim editor. One of the features is visual block inserting. There's a lot of use cases for visual block inserting. Since being introduced to it, I've been mostly using it to prepend multiple rows with comment signs.

An example may provide a better overview, this is the example we have:

1  #include <stdio.h>
2  #include <stdlib.h>
3
4  int main(int argc, char **argv[])
5  {
6          printf("Hello World\n");
7
8          if (argc > 1)
9                  exit(EXIT_FAILURE);
10
11          exit(EXIT_SUCCESS);
12 }

In this case, we have a simple program that returns the exit code 1 when we supply some arguments to the program.

Let's decide we don't what to use that, we'll fire up Vim and comment out the if block (row 8-9).

Go to the 8:th row, press Ctrl-V to go into Visual Block mode. Use j to mark row 9 as well. Then Press Shift-i and insert // before row 8. Press Esc and you will see that row 8 and 9 will look like this:

7
8  //         if (argc > 1)
9  //                exit(EXIT_FAILURE);
10

The documentation gives some examples and there's also :help v inside Vim to get more information.

Standalone checks with Sensu

2014-11-23T00:00:00+00:00

The monitoring framework Sensu as I've written about before is being implemented at my new gig. We've been investigating some fun things about it and looked into something that's not very well documented, standalone checks.

A standalone check is a check that's not defined in the server but is defined and executed on the client. The standalone checks can be defined in two ways, either use the Sensu client or send the definition to the Sensu port. What's described below is how we send checks to the server with standard Linux utilities.

Here's the simplest way to send a check to the Sensu server:

echo '{
  "handlers": ["default"],
  "name": "check_name",
  "output": "error message",
  "status": 2
}' | nc -w1 sensu-server.example.com 3030

The command above sends a check that has an error, this means it will error and throw an alert.

To provide some client context of the alert, we can send the following:

echo '{
  "client": {
    "name": "localhost",
    "address": "127.0.0.1"
  },
  "handlers": ["default"],
  "name": "check_name",
  "output": "error message",
  "status": 2
}' | nc -w1 sensu-server.example.com 3030

The alerts can be sent in a blocking or non-blocking way (TCP/UDP). Use nc -w1 -u sensu-server.example.com 3030 for UDP.

Metrics can be sent into Sensu by using the following:

echo '{
  "client": {
    "name": "localhost",
    "address": "127.0.0.1"
  },
  "handlers": ["graphite"],
  "name": "localhost-metrics",
  "output": "localhost.disk_usage.lolcats.used   9000000    1411131484",
  "status": 0,
  "type": "metric"
}' | nc -w1 sensu-server.example.com 3030

All the settings that are used in the configuration and check files can also be used with standalone external checks. This means that it's possible to use the standard Linux toolchain to send alerts or metrics from shell scripts without any extra dependencies. The examples should provide an introduction to use Sensu with nc (1) and echo (1).

Capture stdout in Golang

2014-11-17T00:00:00+00:00

Working with Docker and Golang for a hobby project, I needed to get the output from docker build into my own log function. To do this, it's pretty easy do some redirections to solve this:

func RedirectOutput(id string) {
        oldStdout := os.Stdout
        readFile, writeFile, err := os.Pipe()
        if err != nil {
                return err
        }

        os.Stdout = writeFile

        go func() {
                scanner := bufio.NewScanner(r)
                for scanner.Scan() {
                        line := scanner.Text()

                        // Log the stdout line to my event logger
                        event.Log(event.Event{Id: id, Msg: line})
                }
        }()

        fmt.Printf("This will be logged to our event logger\n")

        // Reset the output again
        writeFile.Close()
        os.Stdout = oldStdout
}

The function will log everything printed to stdout line by line to our event logger. This can be used to override fmt.Printf and sending it to your own logger without having to import a package in every file.

My event logger stores events in a simple Redis database so I can look at them later. This was a really simple way to solve that.

Moving onto Mesos

2014-09-03T00:00:00+00:00

Instead of using Heroku for this site for a while but decided to build something to host it for myself. There are some apps I use that wasn't fast enough on Heroku so I wanted to try the stability and features of Mesos and Marathon.

What I wanted from the platform:

Fault tolerant
Easy to deploy
Fast
Minimum maintenance
Isolated
Work with a lot of different apps

Mesos is something I've been eyeballing for quite some time and decided it was time to use it to host something I really care about. Mesos also satisfies goal 1, 3, 4, 5 (with Docker isolators) and 6 (also with Docker).

Marathon basically meets all the goals with the help of Mesos.

There's a lot of information about how to run Mesos and Marathon so I'm not going to get into much details about the installation more than the pitfalls I have discovered so far with my three node cluster.

The pipeline to deploy this site is the following:

Add a new file into git
Make sure I have a Dockerfile and a file called .build.json
Push it to my private git repository
A git hook takes care of building the Docker image
The hook also pushes the image to a private docker repository
also, pushing the app definition to Marathon
Marathon pulls the image and runs the instance
Bamboo is used and listens to Marathon events.
Everything is staged and ready to serve requests

The only thing I built myself for this is a git hook that takes a file, .build.json and interacts with Docker and Marathon. What it does is that it builds the Dockerfile provided, pushes the image to the private docker repository and then sends a POST to Marathon to create or update the application.

The only things that have been problematic for me has been that I didn't read enough about Zookeeper, so I forgot to purge old logs (this caused some interesting failures with Zookeeper). And then also that I've been a bit too fast on upgrading Mesos. When I used Mesos 0.20.0, Marathon and Bamboo did not support it, so I had to run on experimental branches for a while, which has worked so-so.

Another advice is to put a master cluster, with Mesos masters and Zookeeper separate from the slaves so that it wont interfere with the services running.

So far I am really happy with the setup and will continue to run it in an experimental phase until I feel that I'm comfortable running it in a somewhat production environment and trust it's stability.

The pipeline works really good and it's almost as fast as Heroku to deploy new applications. If I were to add some more caching in the build steps I'm sure I can speed this up a bit more.

Speeding up

2014-08-30T00:00:00+00:00

Improvements:

Added Rack::Deflate.
Reduced minimum requests from 4 to 1.
Page size has been reduced from 30.6kB to 3.7kB.
From ~12 to ~260 requests per second.
98th percentile of responses within 100ms.

tcpdump OSPF

2013-07-23T00:00:00+00:00

Just because I always forget the syntax for dumping OSPF packets:

sudo tcpdump -Xv -i eth0 ip[9] == 89

So, that's hex and verbose on the eth0 interface.

Into Docker

2013-06-01T00:00:00+00:00

Docker is a really sweet piece of tech. It is built on top of LXC and provides an abstraction to be able to easily build images for deployment.

I've been playing with Docker for a few hours and can definitely recommend it for easy deployments. What it does provide from traditional deployment strategies and configuration management is mostly speed and a standardized way to deploy different types of applications. Think of Docker as your own private PaaS.

When I first started out there was one thing I wanted to do, I wanted to deploy a Sinatra application written in Ruby. To get started I decided to go with a Dockerfile and write the steps to build the image and create a snapshot of it.

I'll be building this from an Ubuntu base box, so define that in the Dockerfile and install ruby and rubygems:

FROM ubuntu
RUN apt-get install -y ruby rubygems

I felt that getting the artifact into the image was the tricky part, the documentation describes a way to insert a file from a URL and also a way to add a path from the local filesystem into the LXC container. Both of these commands however, did unfortunately not work in this version (it works if you specify add or insert via the CLI interface).

What I did was download it from a URL with wget and untar it at a known location.

RUN wget http://example.com/sinatra-app.tar -O /opt/sinatra-app.tar
RUN mkdir /opt/sinatra-app
RUN tar xf /opt/sinatra-app.tar -C /opt/sinatra-app

I installed the dependencies with bundler and exported the port which Sinatra was going to run on:

RUN cd /opt/sinatra-app && bundle install
ENV PORT 5000
EXPOSE PORT 5000

To build and run this Sinatra application, the following commands are being used:

docker build < Dockerfile
# outputs the steps and hash, for example 9de15c84a9a1

docker run -d 9de15c84a9a2 foreman start -d /opt/sinatra-app
# runs the newly built image and starts the app with foreman

To view the app, run docker ps:

ID            IMAGE         COMMAND                CREATED       STATUS           PORTS
96147c4d2090  9de15c84a9a1  foreman start -d /op   Up 22 minutes 22 minutes ago   49171->5000

The port specified is the one that is forwarded on your host server, so if your host server is docker.example.com you'll see the app on docker.example.com:49171.

Docker seems like a great way to start out with nicer and better deployments for applications and services. I'm really looking forward to see what we can do with it and what others come up with to make it faster and easier to build new applications.

Apache rewrite maps

2013-03-17T00:00:00+00:00

At $WORK we have a couple of domains for our clients and need to redirect them to a specific sub-URL. Previously we've done this with a script which required some manual work and unfortunately app downtime.

We wanted to implement SSL terminators in front of our app servers and decided that Apache would be a good fit. Hearing that Fastly praises Apache for it's speed, it was decided to be put up to a test. By using Apache we also got a nifty little function called RewriteMaps.

For example, a customer domain can look like http://antonlindstrom.com and should redirect to http://example.com/anton.

To enforce this for over 100 domains we used the following Rewrite semantics:

RewriteEngine   On

RewriteMap      exampledomain           txt:/etc/apache2/domains.txt
RewriteRule     ^/$                     ${exampledomain:%{HTTP_HOST}}

RewriteLog      /var/log/apache2/rewrite.log
RewriteLogLevel 5

In the file /etc/apache2/domains.txt we can add the following:

antonlindstrom.com   http://example.com/anton
exampledomain.com    http://example.com/exampledomain

When using RewriteMaps it's possible to add domains on the fly without restarting or reloading services. It's a simple solution on a simple problem. I'm really happy about solving it with a simple and off the shelf solution with having room to grow with database backends.

Adventures with Puppet and Sensu

2013-03-17T00:00:00+00:00

I took some time to play around with the Sensu Puppet module this weekend. It's isn't the most mature module but it's working terrific. Once I did some diving into the module it was pretty easy to get around to using. This will be somewhat a description of what I did to get Sensu up and running in my own infrastructure.

The readme will probably get you up and running, the thing I missed however was that the default handler isn't included (I really should make a pull request).

Note that the module might change a lot so this will probably be some faults in this config in a few days/weeks/months.

Current README example:

node 'sensu-server.foo.com' {
  class { 'sensu':
    rabbitmq_password => 'secret',
    server            => true,
    plugins           => [
      'puppet:///data/sensu/plugins/ntp.rb',
      'puppet:///data/sensu/plugins/postfix.rb'
    ]
  }

  sensu::check { 'check_ntp':
    command     => 'PATH=$PATH:/usr/lib/nagios/plugins check_ntp_time -H pool.ntp.org -w 30 -c 60',
    handlers    => 'default',
    subscribers => 'sensu-test'
  }

  sensu::check { '...':
    ...
  }
}

What we'll have to add is the following:

sensu::handler { 'default':
  type      => 'set',
  command   => 'true',
  handlers  => [ 'mailer' ],
}

sensu::handler { 'mailer':
  type        => 'pipe',
  source      => 'puppet:///modules/data/sensu/handlers/notification/mailer.rb',
  config      => {
    mail_from     => 'sensu+alert@example.com',
    mail_to       => 'ops+alerts@example.com',
    smtp_address  => 'localhost',
    smtp_port     => 25,
    smtp_domain   => 'example.com',
  }
}

The example above will create a default handler that is a set, which means that it's passing information to other handlers, in this case mailer. I have a separate module called data where I put the handlers and plugins, so that's where mailer.rb is. The config is a hash which is generated into json as the config for the handler. It'll be called mailer.json.

Let's add a new check to better explain how it works. You can either add it in the class { 'sensu': } block or by defining it as a plugin.

sensu::plugin { 'puppet:///modules/data/sensu/plugins/http/check-http.rb': }

sensu::check { 'http_antonlindstrom_com':
  command     => '/etc/sensu/plugins/check-http.rb -u "http://antonlindstrom.com/"',
  handlers    => 'mailer',
  subscribers => 'sensu-test',
}

By using a little hiera magic we can make a pretty clean node definition and it's pretty awesome.

I've been running sensu for a few hours now and can say that it's the easiest thing to get running when you have a dynamic and configuration management controlled environment. The adventure has been great as it alerts pretty smooth and works faster than Nagios. There aren't that much of reports and GUI fanciness but it does really what it should do. Sensu alerts when it should. The only problem I found was that it didn't alert that fast when nodes became unresponsive (for instance when shutting down a machine).

The problem could be solved by running checks that alerts on the services on the node and do real service metrics instead of host metrics. By running checks on your service and checking load and metrics for how it is performing, a host should not really be important if the service still responds well.

Conclusion: I really found Sensu great as it pinpointed the most troublesome parts of Nagios. Configuration. In Sensu it really works great with the supported configuration management modules.

Out of the Crisis

2012-11-21T00:00:00+00:00

Currently reading the book Out of the Crisis by W. Edwards Deming.

Deming's key principles

Create constancy of purpose toward improvement of product and service, with the aim to become competitive, stay in business and to provide jobs.
Adopt the new philosophy. We are in a new economic age. Western management must awaken to the challenge, must learn their responsibilities, and take on leadership for change.
Cease dependence on inspection to achieve quality. Eliminate the need for massive inspection by building quality into the product in the first place.
End the practice of awarding business on the basis of a price tag. Instead, minimize total cost. Move towards a single supplier for any one item, on a long-term relationship of loyalty and trust.
Improve constantly and forever the system of production and service, to improve quality and productivity, and thus constantly decrease costs.
Institute training on the job.
Institute leadership (see Point 12 and Ch. 8 of "Out of the Crisis"). The aim of supervision should be to help people and machines and gadgets do a better job. Supervision of management is in need of overhaul, as well as supervision of production workers.
Drive out fear, so that everyone may work effectively for the company. (See Ch. 3 of "Out of the Crisis")
Break down barriers between departments. People in research, design, sales, and production must work as a team, in order to foresee problems of production and usage that may be encountered with the product or service.
Eliminate slogans, exhortations, and targets for the work force asking for zero defects and new levels of productivity. Such exhortations only create adversarial relationships, as the bulk of the causes of low quality and low productivity belong to the system and thus lie beyond the power of the work force.
1. Eliminate work standards (quotas) on the factory floor. Substitute with leadership.
2. Eliminate management by objective. Eliminate management by numbers and numerical goals. Instead substitute with leadership.
1. Remove barriers that rob the hourly worker of his right to pride of workmanship. The responsibility of supervisors must be changed from sheer numbers to quality.
2. Remove barriers that rob people in management and in engineering of their right to pride of workmanship. This means, inter alia, abolishment of the annual or merit rating and of management by objectives (See Ch. 3 of "Out of the Crisis").
Institute a vigorous program of education and self-improvement.
Put everybody in the company to work to accomplish the transformation. The transformation is everybody's job.

The list is copied from Wikipedia, 2012-11-21 and originates from the book Out of the Crisis.

Starting out with grok in Logstash

2012-09-24T00:00:00+00:00

grok seems to be the default way to filter events in Logstash. I got in contact with it last week and found some great documentation that I thought I'd save for a rainy day.

First thing to read is the excellent documentation about grok on the Logstash website. Then, jpmens has written some awesome and very informative posts, especially this one about grok.

In my example we're matching the following string:

1999-02-19 20:59:59 Hi, Peter. What's happening? We need to talk about your TPS reports.

A simple example of the grok filter can be seen below.

filter {
  grok {
    type    => 'innotech'
    pattern => "%{DATE} %{TIME} Hi, %{USERNAME:name}. What's happening\? We need to talk about your %{DATA:report_type} reports."
    add_tag => "to_%{name}"
  }
}

This matches the our string and we can collect the name in the %{name} variable, and TPS will be in the variable %{report_type}. The important thing to notice here is that the filter will only act on the input with the type set to innotech. If the input is not set to innotech it will be ignored by this filter.

Now, why do I use %{USERNAME} and %{DATA}? What do they match? In Logstash there are predefined patterns which are defined here. The easiest way to test grok out is to use the excellent grok debugger.

That's the quick introduction of how to get started with grok filters in Logstash. Below is a complete example of a shipper:

input {
  file {
    type   => 'innotech'
    path   => [ '/home/pgibbons/memoirs' ]
    format => 'plain'
  }
}

filter {
  grok {
    type    => 'innotech'
    pattern => "%{DATE} %{TIME} Hi, %{USERNAME:name}. What's happening\? We need to talk about your %{DATA:report_type} reports."
    add_tag => "to_%{name}"
  }
}

output {
  stdout { }
}

Log on %{name}!

hostedmunin.com and Puppet

2012-07-22T00:00:00+00:00

Found out about hostedmunin.com a few days ago and decided it would be nice to implement for my own few hosts. Earlier I have used Munin with a MQ (RabbitMQ) to send metrics to Graphite. A few weeks ago I decided to remove those extra servers to save me a few bucks.

As I have all my hosts in Puppet it would be the easiest way to deploy munin through it. I wrote a damn simple module for munin-node and added the IPs listed on hostedmunin as pollers. All I did after that was to add the hosts in the web interface of the service and the data started to show up.

If you want to use the munin puppet module, all you have to do is add these lines for your nodes:

# hostedmunin.com
class { 'munin::config':
  pollers => [
    '^2001:67c:27ec:249::.+$',
    '^2a02:20c8:1670:249::.+$',
    '^91\.227\.249\.12$',
    '^91\.227\.249\.13$',
    '^91\.227\.249\.20$',
  ],
}

include munin::package
include munin::service

Now, the next step is to add support to the module to add your own plugins to munin. So, check out the awesome hostedmunin.com.

SPDY on Nginx development release

2012-07-01T00:00:00+00:00

Just had to test SPDY on the development version of Nginx as it is now supported by Automattic. This is how I did to get it running.

Download the latest development version (currently 1.3.2) and apply the patch from spdy patches page by using patch -p1 < spdy.patch.

Install

Configure and compile as usual (with SSL support):

./configure --with-http_ssl_module
make && make install

Create a self signed (or a real) certificate and add in /usr/local/conf and add the following settings to get it running:

listen 443 ssl spdy default;

ssl_certificate      /usr/local/nginx/conf/cert.pem;
ssl_certificate_key  /usr/local/nginx/conf/cert.key;

Run

Start it up (sbin/nginx) and go to the site, everything should now work as expected.

Sources for checking SPDY status in Chrome:

MongoDB Replica Sets

2012-05-25T00:00:00+00:00

This blog post demonstrates how to set up Replica Sets in MongoDB. Replica Sets is a way to get High Availability (HA) in MongoDB without much hassle. To be able to use a slave for reads, backups or as a standby master we use the MongoDB Replica Sets.

This post is mostly for my own future reference.

Upgrade to replica sets

First, we have to upgrade to replica sets. This is done by doing the following on a single node configuration.

On the MongoDB nodes, shut down the MongoDB service and start it with the following command (fooReplicaSet is the replica set name):

mongod --replSet fooReplicaSet

Config files

To add replSet in configuration files, add replSet = fooReplicaSet into configuration (such as /etc/mongodb.conf).

Check Status

Check replica set status by running:

rs.status();

Information about states is located in the MongoDB documentation.

Adding members

To add members in the replica set we can do it by adding all the nodes via rs.initiate().

> c = {
... _id : "setName",
... members : [
... { _id : 0, host : "rs1.alley.se" },
... { _id : 1, host : "rs2.alley.se" },
... { _id : 2, host : "rs3.alley.se" } ] }
> rs.initiate(c)
> rs.status()

This will initiate and add the nodes rs1.alley.se, rs2.alley.se and rs3.alley.se.

Why can't I query the slave?

The error, error: { "$err" : "not master", "code" : 10107 } may occur when reading from a slave, in that case you have to enable slaveOk.

rs.slaveOk();

Conclusion

When using replica sets, we are able to do backups or store a database for DR.

If the master gets unavailable a new master will be elected. This will be a way to support automated failover and HA.

Not using Replica Sets is a dangerous business but is of course no silver bullet. As the CAP theorem states:

According to the theorem, a distributed system can satisfy any two of these [Consistency, Availability, Partition Tolerance] guarantees at the same time, but not all three.

The MongoDB replica sets will only satisfy Availability and Partition Tolerance (AP) but should be good enough for many systems.

Updating rack gems on Heroku

2012-04-13T00:00:00+00:00

Running my blog on Heroku is great. It's stable, it's cheap and it's powerful. Recently I have been pretty bad at updating this blog. As a result I have not updated the gems involved.

Today I was about to update the blog and move from the old .gems file to Bundler. That made me update all the Rubygems involved and resulted in failure.

`raise_if_conflicts': Unable to activate rack-jekyll-0.3.7, because
rack-1.4.1 conflicts with rack (~> 1.2.1) (Gem::LoadError)

Due to this problem I decided to search for alternative ways of deploying Jekyll on Heroku. As I've been working a lot with Sinatra lately the approach I decided to use was the one described by @jstorimer in his post Jekyll On Heroku.

To make it compatible with Ruby 1.9.2 you have to include the current directory in config.ru. So, in the top of the file, add $: << '.'. That will do it and everything should work as intended.

So, I replaced rack-jekyll with sinatra and it works like a charm.

DNSSEC implementation in Sweden

2012-01-02T00:00:00+00:00

The single biggest web hosting company in Sweden, Binero signed all their domains and made a big impact on total signed domains. The signing frenzy caused that about 1 in 10 domains in Sweden are now secured with DNSSEC.

Source: .SE

I realized today that I was using Binero's nameservers for my domain antonlindstrom.com which they also signed. So, the domain was signed but there were no DS-records that would make the domain valid. This also meant that while the RRSIG was there but not the DS-record it caused the domain name to be invalid. I had the domain name at another registrar and decided to change both name servers and registrar to DNSimple which I have heard much good about. It is not very advanced but for this domain it will suffice. For more advance usage and labs I will use my alley.se domain which I am using with my own name servers.

In the recent few weeks I have been signing over 10k domains for another domain name host in Sweden, which I am very proud to say turned out very well. There was a really smooth transition and I really felt like contributing to something which was also really nice. As I had just a little knowledge of DNSSEC at the start of the project I had to learn about it in just a few weeks and then implementing it.

The first problem was that some parts were not very good documented and it was a lot to grasp in a short time. I needed to upgrade nameservers, research common problems and how to implement in a way that would not interfere with the original implementation. In the end it worked out very well without any major problems. Some parts are still left to complete the project but over all I am very pleased with how it turned out.

From this project I learned that persistence is the key and I'm sure that some weeks were more than mere 40 hours of work to get a grasp of everything. Legacy implementations can sometimes bite you in the ass. Technical debt should be prevented by having good documentation, commented code and what I would prefer, configuration management. So, by documenting well and using configuration management the implementation phase would have gone faster ans smoother. The DNSSEC learning however were some well spent hours and in the end DNSSEC is very similar to other techniques used for validating in a chain of trust.

A great way to start 2012!

Varnish and JenkinsCI

2011-10-22T00:00:00+00:00

I have been playing around some with Continous Integration (CI) recently. The software I have tried is Jenkins. Unfortunately, untuned on a 512MB RAM virtual server it has got some slow moments.

I ran Jenkins with nginx but decided to try Varnish instead to get some caching in place. I also decided to find out what the requests made by Jenkins were. Turns out it does some requests to /static/, well that is good. Let us just cache /static/ and try to hit refresh.

Add the following to /etc/varnish/default.vcl, on top of your other tweaks:

sub vcl_recv {
    # Cache everything static
    if (req.url ~ "^/static/") {
      remove req.http.cookie;
      return (lookup);
    }
}

sub vcl_fetch {
    if ( req.url ~ "^/static/" ) {
      remove beresp.http.Set-Cookie;
    }
}

The result is that most of the static files gets cached and the dynamic content gets collected from the CI. Hits are shown as "|" and misses as "#".

This has unfortunately not been tested on a large environment with many users and better hardware but I assume it would be a great difference in speed on better hardware as well.

Using the Glesys beta API with Fog

2011-09-18T00:00:00+00:00

Currently Glesys is developing an almost finished API for the Cloud-services. Currently only the compute interface is done and beta access for it can be enabled by contacting Glesys support.

The Ruby cloud computing library, Fog now also supports Glesys and as this is being written the next release will include the support into the Ruby gem. This post will show how to use the Glesys services with Fog.

As mentioned, the API is currently in beta and by contacting Glesys access can be enabled for your account. When enabled, an API key must be created. That is done by going to the API tab then "API Keys" and click, "Create new key". The default access for the key is not to allow anything, change this by clicking "None" under "Allowed Hosts" and under "Permissions". If you want to enable the key for all hosts and give them access to all parts, set 0.0.0.0/0 in Allowed Hosts and Allow all in Permissions.

After creating an API key, if the version of Fog is over v0.11.0 then just install the Ruby gem, otherwise pull Fog from Github. Edit the file ~/.fog and make it look like the file below with your login and your API key you just created.

default:
  :glesys_username: 'CLXXXXX'
  :glesys_api_key: 'secret2g3zX72kXq1w31MXRzkRfxLMjXJL9Q6X6X'

To test Fog out, the interactive prompt is a great way to get started. Type fog or if you pulled the source code from git, use ./bin/fog to get a prompt running. It is then possible to start interacting with Glesys. Below are a few examples on commands and how to use them:

As this is a beta, some of the functionality might change but over all this would stay the same. Fog::SSH will be changed as soon as I get the opportunity to make use of xm.ssh(commands), currently the password is not saved in the session and to access the IP we will have to go through the "iplist", this will probably be aliased to from :public_ip_address.

Now, go on and create some servers and provision them with Puppet or Chef to make the process repeatable and choose several compute providers to make your service solid in case of failures. It is not a question of if, it is a question of when a server will fail.

Bridged IPv6 network on Hetzner box

2011-09-05T00:00:00+00:00

A while back I found out about cheap dedicated servers in Germany from Hetzner. I decided to give it a shot as they had native IPv6 and customers are given /64 subnets. This was the ideal testbed for me to try out IPv6 and Xen together.

As I did not intended to use a public IPv4 subnet for my Xen domUs I was going to give them RFC1918 addresses and put them in another network. I concluded that my setup would use the topology described in the Xen wiki as Virtual Network. This will be the easiest way to combine both NAT for the IPv4 addresses and plain routing for the IPv6 addresses.

Before you start any of the other steps, make sure you can reach your dom0 from IPv6 and install Xen and some domUs to test things out on.

First off, just include the dummy kernel module, either by modprobe dummy or editing /etc/modules. Then add the interface (if Debian) into /etc/network/interfaces just as a regular interface with the RFC1918 address (ex. 192.168.0.1). Then to enable it as a bridging interface in Xen, modify your /etc/xen/xend-config.sxp to enable the following network settings.

# -*- sh -*-

## Bridged
(network-script 'network-bridge netdev=dummy0')
(vif-script vif-bridge)

(dom0-min-mem 196)
(dom0-cpus 0)

Add the following into /etc/sysctl.conf:

net.ipv6.conf.all.forwarding=1
net.ipv6.conf.all.proxy_ndp=1

After that just boot your Xen domU up and assign an IPv6 address to it. The IPv6 address could be any IP in your /64 subnet. I chose ::20. Check if you can ping6 the address assigned to the domU from within the dom0. After that it is just a matter of advertising the address to the Hetzner boxes, this is done by using:

ip -6 neigh add proxy 2002:dead::beef::20 dev eth0

Check your neighbors with:

ip -6 neigh show

Notes on Keepalived and DNS

2011-07-22T00:00:00+00:00

Following the HA trend I created with the last post I though I would write down some notes on high availability DNS with Keepalived. When implementing this I found out that it was poorly documented when using both UDP and TCP traffic. It seems like some people refer to DNS as UDP only. This would disable features like zone transfers and queries over 512 bytes.

Keepalived is a load balancer with HA features such as VRRP (RFC 3768). VRRP is a protocol that enables the two gateways (or in this case load balancers) to share an IP. Keepalived has also got checks for service availability. What this means is that if a server, in this case a DNS server stops responding it will be removed from the load balancer and will not be sent any queries.

The two load balancers are sharing the IP address of 10.0.2.16 (VIP). The two DNS servers has the IP addresses 10.0.2.20 and 10.0.2.21. These will be DNS servers responding on TCP and UDP and could be any DNS like Bind, PowerDNS or Unbound..

So, the /etc/keepalived/keepalived.conf is configured as follows:

As it seems like Keepalived does not officially support UDP there is no specified check like the TCP_CHECK I dug out a MISC_CHECK that huangmingyou had created. The script does a lookup in the DNS for a specified serial number that is in a TXT record. If it is not found the server will be removed from the load balancer until the script returns OK.

DRDB on VirtualBox

2011-07-10T00:00:00+00:00

When you use the cloud as hardware for your services it does not mean that the cloud should be any better at keeping your hardware working. The cloud can provide high availability measures as hot migration but that is not a guarantee that your systems will have 100% uptime.

I have heard many stories on companies losing a lot of money due to downtime because the server provider is experiencing hardware problems. Some are yelling at the server provider for not keeping their hardware up at all times. Those are the ones that have not managed hardware and has to know that hardware does fail.

Issues needs to be solved by building your own uptime and building your own redundancy. By using techniques such as load balancers and HA tools such as HAProxy you can keep your system up even though some systems fail. Use the Chaos Monkey to test it. I decided to implement my own redundancy using DRBD on two MySQL servers in VirtualBox.

DRBD

I found this guide on Virtualbox and DRBD but it did not work for me. I followed the first steps and created a loop block device with dd.

dd if=/dev/zero of=/opt/drbd-test.loop bs=1M count=200
losetup /dev/loop1 /opt/drbd-test.loop

The resource, is then configured with two clients, 192.168.1.10 and 192.168.1.11. These are configured to use the loop device we created with dd and losetup. The file /etc/drbd.conf is configured as follows.

common { 
  protocol C; 
}
resource test {
  on drbd1 {
    device    /dev/drbd1;
    disk      /dev/loop1;
    address   192.168.1.10:7789;
    meta-disk internal;
  }
  on drbd2 {
    device    /dev/drbd1;
    disk      /dev/loop1;
    address   192.168.1.11:7789;
    meta-disk internal;
  }
 }

Load the kernel module, drbd with modprobe drbd.

I restarted the services and ended up with error code 10. That seemed to be issues with the syncing between the two clients and I had to reinitialize the metadata of the resource which caused me to do the commands below. The first one is initializing the metadata, the second is bringing the resource back up and the third is making the node primary. The last command should only be executed on the primary node (for example drbd1).

drbdadm create-md test
drbdadm up test
drbdsetup /dev/drbd1 primary -o

Then you can check the status of drbd by using cat /proc/drbd. Hopefully it will say something like the following. If it looks like that, drbd is successfully running.

version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@drbd1, 2011-07-10 02:26:35

 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----
    ns:56267 nr:9 dw:6112 dr:51106 al:8 bm:8 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

Then, on the primary I ran the following to create an ext3 filesystem and mount it under /mnt/mysql.

mkfs.ext3 /dev/drbd1
mkdir -p /mnt/mysql && mount /dev/drbd1 /mnt/mysql
touch /mnt/mysql/syncing_drbd
umount /mnt/mysql && drbdadm secondary all

To see that it all works, run the following on the secondary and see the file syncing_drbd in the directory.

drbdadm primary all
mkdir -p /mnt/mysql && mount /dev/drbd1 /mnt/mysql 
ls -l /mnt/mysql

And finally, add drbd to /etc/modules to get the module running at boot.

Heartbeat

To be able to automatically make the secondary primary when the primary goes down we have to use something like heartbeat. Heartbeat sends messages between two servers and if one of them stops responding the other one is taking over.

To configure heartbeat, specify, port and nodes in the file /etc/ha.d/ha.cf

logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 30
initdead 120
bcast eth1
udpport 694
auto_failback on
node drbd1 drbd2

To create a shared key between the two clients create a shell script with the following content and copy the contents in /etc/ha.d/authkeys from the node where the script was executed to the other.

cat <<-!AUTH >/etc/ha.d/authkeys
    # Automatically generated authkeys file
    auth 1
    1 sha1 `dd if=/dev/urandom count=4 2>/dev/null | md5sum | cut -c1-32`
!AUTH

Heartbeat needs to be configured with the services that it should control when the primary or secondary server hits the floor. This is done in the /etc/ha.d/haresources file. The contents of this file is described very well at the MySQL documentation. The file will mount our drbd disk in /mnt/mysql and start mysql when the active node is lost.

drbd1 drbddisk Filesystem::/dev/drbd1::/mnt/mysql::ext3 mysql

MySQL

To make MySQL use the directory, the easiest way is just to link the configuration file from /mnt/mysql/my.cnf to /etc/mysql/my.cnf so that configurations are changed between all servers. Then in the configuration set the datadir to your drbd directory. This will cause all data to be synced.

I got the error "operation="mknod" pid=12802 parent=3974 profile="/usr/sbin/mysqld" requested_mask="c::" denied_mask="c::" fsuid=106 ouid=106", just remove apparmor or check its configuration. I still got some errors because I moved the directory, then I used:

mysql_install_db --user=mysql --ldata=/mnt/mysql

This is one way to make your service a bit more resistant from failure and keep your uptime a bit higher.

Airport Express and FreeRadius

2011-06-05T00:00:00+00:00

I have been experimenting some with FreeRadius recently. RADIUS is a protocol for AAA and can be used for authentication in amongst other Cisco and HP network equipment.

We recently bought an Apple Airport Express to be able to bridge the Internet connection from one part of our apartment to another. As they are rather cheap and works very well with other Apple products it is a pretty good choice. To be strengthen the security I am using MAC filtering with WPA2.

While it is simple to add more MAC addresses in the Airport it is not as scalable and simple as using a script to update a row in a file, MySQL or LDAP. An issue related to the Airport is that it has to restart each time the configuration is updated. With these things in mind I decided to implement RADIUS and use it with the Airport Express. The free implementation FreeRadius is a big, modular pretty recognizable RADIUS-server which I have been researching for a few weeks is the server I am going to use.

When searching for the MAC address access control for the Airport it was not very easy to find good material, at last I found this which I am going to use. Using what the author did there worked well. I am pasting them in here as well. First, install FreeRadius:

apt-get install freeradius

Add (or replace existing examples with) this in /etc/freeradius/clients.conf

client 10.10.10.1 {
    secret = yourShareds3cret
    shortname = airport
    nastype = other
}

After adding the client, client 10.10.10.1, is the IP of your access point next step is to add the MAC addresses in the users file. Add the following in /etc/freeradius/users and use the same secret as you did in the clients.conf file. The first part is the MAC address of your clients (laptops etc.).

00FF00-FF00FF  Cleartext-Password := "yourShareds3cret"

In the Airport, choose "Access Control" and then under MAC Address Access Control, use RADIUS. Supply the IP address of your FreeRadius server and under "Primary Shared Secret" add the same secret you added in clients.conf and users, in this example it would be "yourShareds3cret".

If you want to verify your configuration, use "freeradius -X" to go into debug mode. Then check for lines like: Calling-Station-Id = "00-FF-00-FF-00-FF". That should be the MAC address of the client you want to authenticate.

London

2011-05-07T00:00:00+00:00

Me and Sophia were in London for a few days this week. Just to get some time off, not thinking about work or anything. It was really, really fun and we had not planned to much to do. Much of the things we did were really spontaneous, we had a clue about some of the things we wanted to do but mostly decided in the last minute.

We managed to take a few more photos, some are on my flickr. And maybe Sophia will post some later as well.

After the wedding in the UK there were a lot of flags in the city. It also seemed like they waited to clean up until the bank holiday was over.

We went to the Aquarium to see some fish and sea life. We saw turtles, sharks, corals, crocodiles and many many more cool things living in the water.

I quite like London, it is a huge city with a lot of stuff to do and there are always things to look at.

Downtime in the Cloud

2011-04-22T00:00:00+00:00

Yesterday my website went down a few times. As it is hosted on Heroku it is always a possibility that it goes up and down without me being able to fix this. Heroku in turn is using Amazon EC2 and is therefor vulnerable to any problems occurring at their datacenters.

Amazon apparently had some problems with their EBS service which caused Heroku to get some hiccups. There are several great posts that have discussed whether or not to trust cloud services or to build own datacenters. To be secure in the cloud I think you should, as with everything, spread the services on different providers. Putting all your money in one stock is risky business, spreading the money in several stocks will make you more resistant to failure if one of the stocks plummets.

Of course my website is not that critical and there is now way that I state that it should have 100% uptime (or even five nines). 98-99% is good enough for me. 98% will be about a week of downtime, hopefully this would not occur at the same time. If I had a service that would lose money each time it went away from the Internet it would be a completely different matter.

I would recommend these posts about the Amazon issues and availability:

CloudFoundry and PaaS

2011-04-18T00:00:00+00:00

Recently VMware announced Cloud Foundry, an open PaaS. It works quite like Heroku but has the possibility to host other frameworks than Rails, Sinatra and Logo. Frameworks like Django, Node.js are some that exists in Cloud Foundry currently. VMware News Release states:

"VMware Delivers Cloud Foundry, The Industry’s First Open PaaS"

I think the best part is that the project is Open Source. It also means that developers can add their specific plattform to the PaaS (like PHP), which will make it really useful. Another good thing is that hosting companies are able to install Cloud Foundry without paying a single dime. Modifications can also be made to build custom extensions to suite the company.

There are also Services, which currently are databases (Redis, MongoDB and MySQL). These can be provisioned and used within the CLI. The idea of this is quite nice, when the developer needs a service he just provisions it and it starts up with zero configuration. The simplist in me loves this kind of abstraction. There is one word for this, awesome.

I have not tried the PaaS to the fully, just checked it out. For now I think it is great. Cloud Foundry uses a CLI which can be installed from a Ruby Gem. It does everything I want. I can change password, add services, add applications and more.

gem install vmc

What is my verdict? I think this is a quite nice idea and so far neat executed. The more organizations that adopt the Cloud Foundry plattform, the more it will grow. With some open source spirit it will grow even more. So far it is only in beta and there is a free signup at CloudFoundry.com. Are there any possible downsides? Sure, it could break. But as it is open source I think that it is highly unlikely.

PaaS in general are great for abstraction and software developers has the opportunity to just get things going. Although for more specific things and configuration options I think the use of a VPS is more useful. I think this mostly depends on the user and the scale of the application. Is the application big, well then it would be better to use raw hardware. If it is small, the PaaS fits quite well if no customizations should be done.

I have accually used PaaS for a while with this site and Alley.se and it works well. I do not program that much and the usage is quite low. Blog posts and a few small applications works fine.

Building for repeatability

2011-03-15T00:00:00+00:00

When building or configuring something I am always thinking about whether it is possible to repeat the process of installing this in a large scale. Do I remember each step? Is it possible to do all the steps created in an easier way?

Configuration management like Puppet or Chef are ways to cope with this. But there are also ways to do this without having a fully automated environment. When I install and setup my main computer I use git to get settings and dot files I have.

Another really smart thing to create an easy setup on Mac is the Mac App Store, this ensures that favorite applications are stored in one place and if moving to another computer this will make it easier to just fire the application up and download and install the software you bought and chosen.

If using repeatability it is also easy to get the state of a computer without any hassle when a crash should occur (provided that you have backup of those dot files in another location).

Viewing binary files in hex

2011-02-25T00:00:00+00:00

When writing rules in Snort or otherwise get information about binary files it can be helful to view them in hex format. Like capturing binary data from tcpdump. For a school asignment in Intrusion detection we were asked to write rules in Snort to match defined binary files. To view binary in hex or hex in binary we can use xxd:

$ xxd binary.bin
0000000: 616e 746f 6ef2 2a7b 4180 a713 c9eb 1da2  anton.*{A.......
0000010: d236 19eb bd7a 80a7 13c9 fa07 dc80 1924  .6...z.........$
0000020: 29ba dc1f c0ea 886b 0462 56f1 7246 350a  )......k.bV.rF5.
0000030: 5980 a713 c905 d101 0a                   Y........

When editing in VIm it is possible to use the command :%!xxd and reversing :%!xxd -r.

Pingdom and SSH

2011-01-28T00:00:00+00:00

Some days ago I noticed a huge amount of errors in auth.log with identification strings. The error often occurs when the packets received by sshd are malformed and does not meet the usual format of the protocol. In my case this happened with a regular interval of about 5 minutes and was not spammed at the port. I decided to wait further and analyze the data.

Jan 23 06:10:57 qward sshd[15399]: Did not receive identification string from 83.170.113.102
Jan 23 06:15:57 qward sshd[3722]: Did not receive identification string from 95.211.87.85
Jan 23 06:20:58 qward sshd[25690]: Did not receive identification string from 207.218.231.170
Jan 23 06:25:57 qward sshd[13899]: Did not receive identification string from 207.97.207.200
Jan 23 06:31:01 qward sshd[3523]: Did not receive identification string from 67.192.120.134
Jan 23 06:35:58 qward sshd[24067]: Did not receive identification string from 78.136.27.223

I decided to block all the IP addresses that sent the data to the port. This became interesting, my phone began to buzz and I heard emails drop to my inbox. Pingdom sent me a notice that my SSH service was down on my server, at that point I got it. Pingdom sends pings to my port but of course it does not send SSH packets and that will result in an error in auth.log. The strange part was that it was not just one IP that sent this, it was about 15-20 addresses which made it seem like there were several servers sending attacks on the port.

If you find these errors and is using Pingdom to monitor the status of the SSH port, bare in mind that it might cause these errors in auth.log.

wroopia DNS API

2011-01-24T00:00:00+00:00

I have been searching with lights and compass to find a DNS provider that has an API to be able to update the records.

When having an API to update the domain records it is then possible to use custom interfaces such as command line or a desktop application. Every provider I have tested before this has an awful AJAX interface which is both slow and does not support batch updating. With the API I can do batch or even dynamic updates. This makes everything a bit easier and more smooth.

Recently I have been working in Ruby and decided to write a wrapper for the Loopia API to make it easier to write applications and command line tools against the API. It is called wroopia (wrapper, ruby, loopia) and is available at Github. It is not fully tested and just quickly hacked together. Improvements are welcome!

More examples will be added later on as I will test it more. Currently it is not even in alpha version.

The Practice of System and Network Admin

2011-01-16T00:00:00+00:00

A while back I decided to buy The Practice of System and Network Administration and had some time off today and decided to read a few chapters.

The book is good, having relevant points and explains things that are not that much technical but instead using an approach mindset which is easing the SAs all tasks. As it is a job that requires time management to solve issues with customers waiting there are several points on saving valuable time and trying to avoid outtakes in the infrastructure. Configuration details and technical issues are not covered and this is not that kind of book. The book is about management and how to avoid troubles in configuration.

As I have not read the entire book I can not say how good the entire book is. The first chapters are very good and inspired me to read more in it. As I find more time off I will indeed read the entire book.

Skype

2011-01-07T00:00:00+00:00

When using international (or national) calls, Skype is the way to go. I have been using it to talk to my parents for a while (on and off) but it is a major player. It is very cheap, costs nothing more than the Internet connection if you use it computer-computer or computer-iphone app.

I have been talking to a hotel room in Thailand from my home here in Sweden and the voice quality was exceptional and even the video quality was pretty decent. Skype is also aquiring Qik which is a video company and might make the video calls even better.

"Skype and Qik share a common purpose of enriching communications with video, and the acquisition of Qik will help to accelerate our leadership in video by adding recording, sharing and storing capabilities to our product portfolio." - Skype blog

It also seems that Skype is able to connect to SIP PBXes. This will give the opportunity to connect Skype to both soft- and hardphones in a company.

While having some trouble with a "global blackout" a while ago it is still a very good product and is based on a peer-to-peer technology which means that there are no centralized servers that the system depends on. Skype uses supernodes that basically are clients with more "power". Skype is also good in couping with NAT and firewalls.

I am going to start using Skype a bit more now and will be trying to integrate Skype with an Asterisk PBX to be able to use the extensions for it and build a VoIP infrastructure with both Skype and SIP. Then also use existing hardware and software for both protocols.

Virtualization

2010-11-25T00:00:00+00:00

I have tested out a few different virtualization techniques in the latest year. I have since this summer used VMware ESXi to virtualize my lab environment and a server for my own webserver. Unfortunately I would like to try out some of the more advanced features and those are not included in the free version of ESXi. I am a fan of Open Source and want to support it as much as possible and I have chosen to change plattform from ESXi to Xen.

Xen does, in paravirtualization, not support virtualization of systems like Windows or BSD if you are using a Linux host as a Dom0. That means that my Windows server either has to be migrated to a separate host or it is possible to use Hardware-assisted virtualization.

When using Xen I am able to do easy backups and do live migration of virtual guests. It does require some extra time in building the Xen templates but after that it is as fast as copying and booting up a new, freshly installed server. I have used debootstrap to install an Ubuntu server on a CentOS one. It works as a charm and is quite fast. Next step is to use an iSCSI target and a secondary fail-over Dom0. The definition of a VM looks like the following:

name = "ubuntu"
uuid = "053b0ed9-e985-82b7-0659-fc9f4c8b3aef"
maxmem = 512
memory = 512
vcpus = 1
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "destroy"
disk = [ "tap:aio:/srv/xen/ubuntu.img,xvda1,w" ]
vif = [ "mac=00:16:36:ff:ff:ff,bridge=xenbr0,script=vif-bridge" ]
bootloader = "/usr/bin/pygrub"

mc-auth Auth.log Scanner for MCollective

2010-11-21T00:00:00+00:00

I have tried to make my own MCollective plugin and decided to make a plugin that checks all the nodes auth.log for authentication failures. Instead of using Fail2ban, I wanted to have more control and see which IPs that are brute forcing my servers and to see how many attempts they are trying against them.

The writing of an agent was fairly easy but there were some bumps. I do not really know how to send multiple values as a hash in MCollective. I instead used a join method and then in the client a split method. It works but I do not know if that is the best method.

The later plan is to integrate my mc-auth plugin with the mc-iptables plugin to block the failed authentications over some sort of threshold value to be able to keep it even better. Some sort of YAML export function in mc-auth would be nice as well.

Jekyll on Heroku

2010-11-20T00:00:00+00:00

I have been writing about the move of my blog and several other components of my domain to the cloud. I moved the website to Heroku and is now using Jekyll as a blog plattform which caused some new issues.

The problem was that I used other name conventions for files, as: /journal. Now the pages are named as /journal.html and the rewrites are using 302 HTTP responses (permanently moved) to support the old links. This is used with the help of a plugin called rack-rewrite and imported into Heroku with a gem manifest. Then in the config.ru file just type the following:

require "rack/jekyll"
require "rack/rewrite"

use Rack::Rewrite do
  r302 '/journal', '/journal.html'
  r302 '/moved_from', '/moved_to'
end

run Rack::Jekyll.new

Then it is just a push away from rewriting paths in Jekyll on Heroku. The readme on the rack-rewrite page is very useful for writing rewrites like in the Apache mod_rewrite way.

For more information about the rack/jekyll gem check out the readme on Github.

Puppet Problem

2010-11-18T00:00:00+00:00

I have been working on a minor problem with one of my Puppet clients. The problem occured after the domain switching and I was wondering why because I removed all of the certificates several times and purged the package several times. One thing I did not do though; remove the CA. After doing that it worked like a charm.

info: Retrieving plugin
err: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 
  'eval_generate': undefined method `closed?' for nil:NilClass
err: /File[/var/lib/puppet/lib]: Failed to retrieve current state of resource: 
  undefined method `closed?' for nil:NilClass Could not retrieve file metadata for 
  puppet://puppet/plugins: undefined method `closed?' for nil:NilClass
err: Could not retrieve catalog from remote server: undefined method `closed?' for nil:NilClass
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run

Puppet is working very well in my environment and it does actually ease the work very much. Together with the puppetd mcollective plugin it provides even more control over the environment.

When API does not help

2010-11-17T00:00:00+00:00

APIs are built to help programmers build their applications around data provided by another part. Our school assignment is to build JavaScript webapplications and to write functions around data from an API build by our teacher. Problem is that the API is undocumented (apart from some examples that mostly complicates things) and the interface is quite weird. I have been stuck all day (or at least half a day, my phone died and refused to wake us up) trying to solve a problem I am having. It would be easier to use a cleaner API with some more documentation but I guess this is the way we are going to do it.

Other things I have done today includes updating my (now three) DNS servers and rearranging my domains and servers. I have also moved my STOMP server to ensure faster lookups with Marionette Collective. I have also studied WLAN technologies and read a few pages in new book I have picked up. It is nice to feel that there is time to study in the class instead of just doing things as fast as possible.

Breaking a leg

2010-11-15T00:00:00+00:00

Yesterday when climbing Sophia broke her foot. It happened when she jumped down and unfortunately missed the mattress and hit the cement floor. After about 3 hours at the emergency we could go home with just a bandage. Wait 3 hours for about 5 people to say just about the same thing. I had no idea that the medical care were so slow, I know that we were the most prioritized case but they could just have sent us to the x-ray immediately and then just tell us to go home and rest.

Now I am waiting to leave for school, listening to Blue on MTV. It is not that great, not at all actually. I have also been writing a CLI for Posten's Package search. That is mostly because I am waiting for a delivery again.

Soon time for school but I rather want to spend the day working at home, I have been working pretty much to get as much as possible done to concentrate on more SysAdmin centric tasks. Experiments with mcollective and Puppet is on the run. I have realized that my STOMP-server is a bit slow which results in slow queries and answers from the mcollective servers. Also, I have been outsourcing my DNS-server and will soon make the move from my own PHP-webserver to a Heroku rails server.

Sinestro, EC2 and Xen

2010-11-12T00:00:00+00:00

I decided to connect my old server today while showering. It is just an EEE Box but I think it will work OK just to try out Xen and to check it out. I thought I would edit the domain from antonlindstrom.com to a more suitable laboration domain name. But I do not really know if it would cause any disruptions in the current services running.

I am currently (and has been all night) compiling Xen with kernel on an Ubuntu 10.10 server. If I had more time with the physical hardware I would have installed Debian as a Dom0. Hopefully it will work quite well and I am able to use it more extensive. I would also like to get rid of VMware ESXi and use Xen instead, that would make it a lot easier to script and backup.

Yeah, I have been testing out a few things on EC2 and I am thinking of getting a permanent EC2 instance for offloading and mostly testing.

A test of Jekyll

2010-11-11T00:00:00+00:00

I am testing Jekyll as new blog and website plattform. Am a bit tired of using my own servers as I cannot guarantee a 100% (or at least 5 9s) uptime when I am laborating with virtualization plattforms and such.

Usally I write my posts at Tumblr in plain HTML, that is how I like it. Now I am using Textile and I am not satisfied. Mostly because I am not that familiar with Textile but also that I feel that I write plain HTML faster (mainly due to the first thing, lack of knowledge in Textile). I have also noticed that Heroku is a bit slower than my own server but it costs less and is hopefully more stable than my own.

The decision of moving to another host came mainly from the fact that my ISP decides to shut down all of the subscribers connections because they want us to go to their meeting. Well, that is not going to happen. That also causes trouble for people that are unable to attend the meeting and in some way rely on their Internet connection (VPN etc). I have complained about this ISP before. They are cheap and provide a fast connection but are not at all stable and I feel that they could improve in many ways if they just put their mind to it. It seems as they use the network as their own lab. network.

Best thing setup

2010-11-09T12:22:08+00:00

I just realized, after a lot of tests and different servers in my virtualized setup that the best thing I have setup is my Git-server. Building the server on Gitosis, all the repositories are working great and my code is gaining much more structure and it is easier to see changes and to rollback or test new functions.

The server is in use so much that I consider moving it from the virtual server to a real hardware server to get rid of any overhead and to get it a bit faster. If I had a faster infrastructure I would consider having it in the VMware ESX-server but as I know it is a bit too pricey to upgrade my SAN to become faster, more scalable, flexible and redundant. The alternative is to use either a SAS or DAS.

Currently the infrastructure is sometimes dragging with high latency due to slow storage connections. While having a 1Gb/s connection to the SAN it is using a backup 100Mb/s connection because it was more stable at implementation. Now the VMs have become so many and disk intensive that it requires upgrades. If I had several 1Gb/s NICs I would use NIC bonding to provide a more fail-safe solution. With the use of Puppet, the disk writes has become more frequent and also the syslog messages sent over the network are increasing.

As I tested out Amazon EC2 last week I saw that they seem to use Xen as virtualization technology I thought it would be cool to test that out. As it feels that on ESX, it is hard to script and program against while Xen on the other hand is more open and after seeing this it feels even more usable. The function of being able to transfer live VMs is a license feature of ESX and is a bit pricey for my private budget. By using open software it is also possible to transfer the data as I want.

Movements

2010-10-29T18:37:07+00:00

Complaining about the classes at the University will not get me far even though the classes are mostly about legacy stuff. Not enough about new and exciting protocols (read IPv6) and ways to solve things and most of all, not enough relevant stuff. Those things you have to learn on your spare time. Well some of the things learned are just great and I am happy to have gotten the knowledge but some things have just been a waste of time.

Well, I am posting right now because I have been checking out on the latest Twitter chatter about #DevOps. That is really interesting and looks like a new movement in the SysAdmin area, that and virtualization.

I will make sure to do some tests in Puppet and Chef. I will happily post it here later as well. I thought I would check out Rails 3, Heroku as well as some IPv6 testing to be able to score some future job opportunity.

About job opportunities, I am looking for a job in the Systems Administration area but is currently studying and would like to write a Bachelor’s Thesis in the field and if anyone knows a company that would like to work with me that would be a pleasure.

And I find this video hilarious; Velocity 2010 - Adam Jacob on DevOps.

HP2140 files lost

2010-10-22T13:47:18+00:00

Sorry! I just realized that I do not have the make.conf and xorg.conf files for HP2140 as referenced in Gentoo on HP 2140 avaliable for download anymore. For that I am terrible sorry, I think I have them saved somewhere in a backup and I will try to retrieve them as soon as possible.

As I do not have the HP2140 in my possession anymore I will not continue to research Gentoo on it. It is now owned by my father which uses Ubuntu.

I truly regret not taking a Clonezilla image of the harddrive in the laptop before installing Ubuntu on it. But hey, I hope I have learned something from that. So, if you read this. Please make a backup of all your files and do Clonezilla images of your harddisks. It gets a bit easier reinstalling your system when you have an image.

Now, back to exam studies for me.

[Update]: Found a more or less updated make.conf in my Gists on GitHub, check it out here; make.conf.

Time = t

2010-10-19T14:09:12+00:00

Ever since Sophia went to work this morning I have been studying non stop. Writing on a lab report about Kerberos, reading in on a few RFC’s, VoIP, Optic-fiber and much, much more. Finally got the activation codes for a software that should help us forward in a course I am taking (although it does not always help).

The thing with the software was that I had it installed on an Windows XP client that ended up in a corrupt filesystem which made it impossible to start it. Weird part is that the ZFS pool is still reporting errors even after deletion and scrubs. Hopefully there are not more errors in other VMs. Thankfully there are backups of most of the VMs on the server.

Now I guess it is time to end the break and start working again. Have to read some more and then it is time for another break. There are not that much time left before the exams and then there are at least three reports that has to be sent in the same day. And a presentation after that. This university has the weirdest planning of all.

Current state

2010-09-20T12:10:26+00:00

I thought I would give a status quick state report. I am currently moving files around my LAN, there are several changes going on in the storage infrastructure. A slight increase in speed will also be seen.

I thought I would do a backup routine with Git when changing configuration files. other Files like photos, music, movies and data with more size will use an SMB/NFS share with rsync for clients and just rsync/SSH with servers. The main problem with backups are the connection reliability and speed. A mobile connection will take ages to finish and might corrupt data. Currently I am using checksums to verify the data but the speed will still be a problem.

I have been experimenting with OpenLDAP and Kerberos as well lately. It is a nice combination which will ease the administration of users in the system. Everything will be nicely integrated between Windows and Unix/Unix-like systems to get a nice consistency when opening the VPN to my end users. I am going to publish a help document on Kerberos and OpenLDAP soon to get a system to work as quick as possible.

I think some of my end users might read this, so I thought I would briefly explain Kerberos and OpenLDAP. Kerberos is a system for authentication which in the log in phase gives the user a ticket to the services in the domain. This gives the user access to all the services without having to supply the password more than once. This is called Single Sign-On (SSO). Kerberos is implemented in Windows since Windows 2000.

OpenLDAP is an open implementation of LDAP which is a directory service. A directory service can hold information about users, computers, printers and more. In this system it will be used to hold user information such as user names, contact information et cetera. When using this it is possible to enter the information about users just once at the server and then it is possible to access that information across all the configured clients.

Alley.se

2010-08-28T13:40:52+00:00

This Tuesday was a really, really good day. Lovely morning, took a walk to get my car and print some papers. Then ate a really good breakfast and in the midst of that, my phone rang. It was a woman offering to buy my domain name; Alley.se. The rest of the day consisted of lying in the sofa, watching TV and studies.

Domain trading is a common thing in the Internet world, though I did not think it would happen to me. I know that there are people that use this as a bad thing, then it is called Cybersquatting.

“The term is derived from “squatting,” which is the act of occupying an abandoned or unoccupied space or building that the squatter does not own, rent or otherwise have permission to use. Cybersquatting, however, is a bit different in that the domain names that are being “squatted” are (sometimes but not always) being paid for through the registration process by the cybersquatters. Cybersquatters usually ask for prices far greater than that at which they purchased it. Some cybersquatters put up derogatory remarks about the person or company the domain is meant to represent in an effort to encourage the subject to buy the domain from them. Others post paid links via Google, Yahoo, Ask.com and other paid advertising networks to the actual site that the user likely wanted, thus monetizing their squatting. Some argue that the dividing line of cybersquatting is difficult to draw, or that the practice is consistent with a capitalistic and free market ethos.” - Cybersquatting article, Wikipedia.

As I have had Alley.se for about 5 years I can not class myself as a cybersquatter as I did not buy it for selling, I was a little doubtful whether to sell the domain or not since I really enjoy it and use it very much internally and for friends. It is a short and good domain name. It is not without a tear in my eye I sell it, it was the first domain I bought and has been really good.

Outside the rain was pouring and I could not have been happier enough to be indoors, enjoying MySQL queries while listening to the thunder rumbling, closer and closer. The good day was in me, all day long.

This was written on that Tuesday but updated later and not published until the domain trade had gone through. Sadly enough the trade did not go through. They made the impression of being ready to buy the domain but after I gave my OK they had to check with some lawyers to be on dry land. Then they said that they were not completely sure of the name as well.

So I still have my lovely Alley.se domain.

Masquerade

2010-08-28T13:35:20+00:00

Tonight there is a masquerade, or carnival called Wrågårdskalaset. I am going to spoil what I am going to be dressed as. First I was a bit into being the Batman but someone did not want to be Robin. It would be a bit funny to be Batman when I guess there will be a few Joker’s there. After watching the old Batman movies on the telly, especially Batman Returns and Batman & Robin we thought of being Batman villains.

There are a few Batman villains to choose from, after Nolan’s exceptionally good The Dark Knight. Many masquerades has had visits from the Joker etc. Now it is time to bring out the old villains such as the Pinguin and Poison Ivy. Yes, we are going to go as the Penguin and Poison Ivy from Batman Returns and Batman & Robin. Photos will be tagged to this post later on.

Deployment

2010-08-27T00:11:43+00:00

As I am more of a Linux man I tend to have less knowledge about the Windows side of things. Well, no more. I decided to do some experimenting on Windows to get more knowledge and get some control over what is happening in that not-so-good operating system. So today, I have been studying and installing AD and some Windows clients. Then I came across a Firefox MSI to deploy across these clients.

I sure like the idea of AD and it is very nice to use across multiple clients. I would love to see an AD for Linux, say Ubuntu that would use APT to download packages marked from a server. Maybe it will be a new project to work on. Integration with Kerberos and LDAP is of course a plus.

Remote desktop is one nice thing as well, it works like a charm and plays nice with my MacBook. It is also pretty efficient as it is possible to run on my slow 3G connection.

Nostalgia

2010-08-20T15:34:04+00:00

When I and Sophia were out checking out 3D TV’s, hair dryers and hard drives we started talking about things from the past. You know the good old stuff like those red things that you could view dinosaurs in 3D (View-Master) or the books that came with cassette tapes that made a sound when it was time to turn the page.

Photo from ansik on Flickr licensed under Creative Commons Attribution 2.0 Generic.

Remote worker

2010-08-19T16:50:00+00:00

Well, I told you about having a mobile connection to the Internet. As I got a new server it means a lot of configuring. Even when I am away. This is something I must say, I enjoy. With a VPN implemented it is easy to do the stuff like I was sitting right next to the server.

There is a problem now that I would like to have some more lab hardware. A rack case, an UPS, an iKVM, a new Gbit switch and an iSCSI SAN. A few more items would be nice. The only problem is the price of course, so if anyone would like to sponsor me that would be really great.

The hardware that I use are mostly for fun and for learning purposes. It is great to have something to test new things on and it is good to practice things before you can manage a system in a corporate environment.

Now, remote access is great to use when you need to check things in the system or get access to resources behind a firewall. It would be really difficult to manage my network if I would not be able to access it from a remote location.

Home network

2010-08-12T12:29:34+00:00

Bandwidth is not all, I say. Even though today would be a good day with a little bit more bandwidth and stability. My mobile 3G connection has a bit to go to have that stability and speed of a nice LAN connection. I would rather have a static IP with VPN to my real home network at AntonLindstrom.com which may see a real upgrade in the near future.

I have been talking about virtualization for a few months and I would really like to see that realized and implemented into my home/lab network. The backbone LAN network is quite stable and has been configured to maximize performance as well as optimize data flow. The issue is the server hardware at the moment, a new server might come in place this autumn and the old one will serve as a media server. At the moment I have seven virtualized servers on my MacBook and two of them is running most of the time I am using the computer. If/when the new server is implemented, these VMs will be transferred to the server and will free my laptop from the heavy usage it faces today.

Frankenjura

2010-06-21T06:24:22+00:00

This week has been really great, lots of fantastic climbing and wonderful scenery. I have been plotting down some notes from every day that we were down there and I thought I would share them with you here. In the tent we called it “captains log” and I guess it was.

Monday 14/6; cloudy, some sun in Denmark but mostly cloudy when waiting for the Rostock ferry. Tight squeeze on the ferry parking deck, had to climb in and out of the car. In Germany no one takes VISA! Chased (or rather followed a short distance) by police in Kritzkow. Found a speeding camera in Pritzwalk and saw a police escort close to Leipzig. Hot and sweaty night in the car south of Leipzig.

Tuesday 15/6; My sisters birthday, congratulations! A yogurt for breakfast and then back on track. Found our way to Wolfsberg, tent was smoothly assembled and a powernap to get the power back after our drive. The hostess at Gasthof Eichler was very, very kind to us and showed us around. Talked to an Australian man who seemed like a really good climber. Climbed at two nice places close to the camping, the crags are pretty different from Sweden and so fun! Saw a mouse, fireflies and strawberries. Great day, a bit tired though.

Wednesday 16/6; Captains log: Eaten a really good breakfast. Far distance between the bolts. Bought stoppers in Forchheim and belayed from a doubtful bolt. Talked about writing about Frankenjura for the swedish climbing magazine Bergsport. Drank pretty good (and cheap) beer, Fortuna Hell. They have many beautiful churches and there are a lot of crosses with Jesus next to the road. Very cozy villages.

Thursday 17/6; Been at a castle, poop in their toilette. The man there was very special. Climbed at a crag called Schlosszwergwand, very nice climbing and a great overhang which tired me out. The schnitzel at the castle was not better than Benidorm’s. Have been very cultural. Gößweinstein is incredible beautiful. A few good photos have been taken. A few beers this night as well.

Friday 18/6; Day for rest, rain. Bought a gift. Drove to Erlangen and Egloffstein to go to some climbing shops, they were pretty so-so. Bought a t-shirt. Met some other swedes from Malmö which we were going to climb with the next day. Ate old fish sticks that could have given us a serious case of bad stomach, Jägermeister heals stomachs. In Erlangen we also met a couple that were climbing in Schlosszwergwand the day before. A lot of cheap beer. Is going to be great to climb Saturday. Two worn out climbers in a tent.

Saturday 19/6; Went to a place with Via Ferrata but did not do that, we were out for the wall. Took a pretty bad fall which might have lightly sprained my ankle. We were out there with the two Malmö-guys. Accidentally hit a German car, had to wait for the police for a little dent. The Polizei was really nice and we just signed a few papers and it was all good. Had barbecue and beer-drinking with our fellow swedes. Pretty cold at night.

Sunday 20/6; Going home. Slow morning, said goodbye to Cristopher, Daniel (the two other swedes) and Martha (hostess). The way home went pretty smooth but we missed the Rostock-Gedser ferry with 10 minutes, had to wait 2 hours for the next. Leaving my climbing partner in Malmö and heading home. Taking a coffee and hot dog in Ljungby, a few kilometers after that I hit a deer. Car looks like scraps from the front, hopefully it will be fixed soon.

Wireless in Gentoo on HP2140

2010-05-25T23:06:00+00:00

The last time I was tinkering with Gentoo on HP 2140 there was a problem with wireless and dhcpcd. Today I found a fix at the Gentoo Forums.

Device Drivers —->

Network device support —->

Wireless LAN —->
[*] Wireless LAN (IEEE 802.11)
.
.
.
<M> IEEE 802.11 for Host AP (Prism2/2.5/3 and WEP/TKIP/CCMP)
[ ] Support downloading firmware images with Host AP driver
<M> Host AP driver for Prism2/2.5/3 in PLX9052 PCI adaptors
<M> Host AP driver for Prism2.5 PCI adaptors

As it takes a while to compile the kernel if you have not optimized it well. I have not so when you compile, it is a good idea to take a cup of coffee or if you prefer, a cup of tea.

After adding these as modules in the kernel you are probably safe to go. Remember to use ACCEPT_LICENSE in /etc/make.conf for broadcom-sta. As the author says;

“Re-emerge broadcom-sta and it should work with Network Manager.”

When I compiled broadcom-sta I got some error about a few entries in the .config file. Just comment them out and recompile, after that it should work fine. The entries I had were

CONFIG_B43
CONFIG_SSB
CONFIG_MAC80211

Previous configuration with Gentoo on the HP 2140 can be read in “Gentoo on HP 2140”. Best of luck to you. After all it pays off to have a snappy machine when you put some effort in it.

Steve Jobs on Flash

2010-04-29T16:59:41+00:00

Just read the open letter Steve Jobs has written about Flash.

I can say that I fully agree on the aspects of why Flash is bad. It just feels like it has served it’s purpose, let the new technologies, like HTML5, JavaScript and more, take over. There is a greatness in letting things go, open standards will be easier to adapt and use.

Please Adobe, focus on your other products of creativity.

Docs and Google Docs

2010-04-18T11:22:06+00:00

I have a certain loath for Microsoft Word, not because it is Microsoft software but more because it is so expensive and does not support open standards. Word uses its .doc and .docx formats which is pretty much proprietary and is expected to be some sort of industry standard. If you get a document from someone you can be at least 80% sure that it is a .doc or .docx format. The software does not support .odt which is an open format for these types of documents.

So, does that mean I love OpenOffice.org (OO.org)? No, it does not. I dislike OO.org just as much. Not because of the lack of support for open formats, that is working great. It is because of the time it takes to load a document and then process it. On a Mac it is pretty sluggish and I feel like it is a nightmare to use.

As I often write short texts and want to be able to quickly go into the text, write a few more words and then close it, it gets pretty slow. I do not want to have a program that I do not regularly use, in the background. Especially when it is not optimized (as OO.org is not) for my computer. I have a program that is almost always running, my web browser.

Google Docs is a word processor that is optimized for the web and has had a rewrite just a couple of days ago. I thought I would write what differs from the old code base.

First of all, my workflow is destroyed. The shortcuts has been edited, the text formats is one example. Also some functions have been moved and it feels like it is more practical and smarter now. There is now a ruler which controls your indents better and it might be a few more tools added. The most important thing is that everything is so much faster! Before it was a little bit slow and sluggish and I complained a lot about it but now it is really, really improved.

There are many things that Google Docs does not have, like support for templates etc. So I have to write texts in the desktop word processors for essays and reports at the University but would like to see that Docs became even better so I can use it for work as well.

The importance of backup

2010-03-08T20:04:04+00:00

Think about the data stored on our computers. I store my photos, school assignments and a lot of code projects. What if they get lost, someone could steal my computer or the hard drive may fail. What am I supposed to do? The data could be lost forever. This is where backup is really important, if a hard drive fails you are able to get the backup from another place and get it back. Many people I know have a lot of data on their computers that they are afraid to loose but does not take action to prevent it.

How to backup? One way to solve this is to get an external hard drive and copy all the files that you have on your computer onto that one, remember to keep copies on your computer so it is stored on more than one place. There are also solutions like Dropbox where you can store your documents, Dropbox will also sync your files with your other computers if you install a small piece of software. Other services are Mozy, Backblaze and the more advanced alternative Amazon S3.

This is how I run my backups:

Photos and music are backed up via a bash-script to my local server
Documents are stored on Dropbox for revision control and high availability
Documents are also stored on Google Docs as I am often using that as primary writing platform
Code projects are saved on Github, also with revision control
My laptop is using TimeMachine to an external drive which is connected about once a week and does a full copy of the laptop
The most important files are uploaded to Amazon S3

In my backup plan there is one thing I want to improve right now. The local server does not have mirrored disks which is something I have to improve to get a little bit higher fault tolerance. I really take care of my data and truly want to save it. I hope you do the same!

Information and convergence.

2009-11-26T12:23:00+00:00

Had a discussion with my friend @TjiffTjoff about emails and how much of the emails sent being spam.

Linux CLI tips.

2009-10-08T12:39:00+00:00

Ctrl-a - Move to the start of the line.
Ctrl-e - Move to the end of the line.
Alt-] x - Moves the cursor forward to the next occurrence of x.
Alt-Ctrl-] x -Â Moves the cursor backwards to the previous occurrence of x.
Ctrl-u - Delete from the cursor to the beginning of the line.
Ctrl-k - Delete from the cursor to the end of the line.
Ctrl-w - Delete from the cursor to the start of the word.
Ctrl-y - Pastes text from the clipboard.
Ctrl-l - Clear the screen leaving the current line at the top of the screen.
Ctrl-x - Ctrl-u - Undo the last changes.
Ctrl-_Alt-r - Undo all changes to the line.
Alt-Ctrl-e - Expand command line.
Ctrl-r - Incremental reverse search of history.
Alt-p - Non-incremental reverse search of history.
!! - Execute last command in history
!abc - Execute last command in history beginning with abc
!n - ExecuteÂ nth command in history
^abc^xyz - Replace first occurrence of abc with xyz in last command and execute it

Re-blog fromÂ http://www.makeuseof.com/tag/15-great-tips-for-ubuntu-power-users/

Gentoo on HP 2140

2009-07-02T23:53:00+00:00

I bought myself an HP 2140 and I thought I should install Gentoo 2008.0 on it. I just followed the Gentoo Installation Docs, and the .config file for my kernel (2.6.29) was just about configured on feeling. If you think something can be added or removed, please contact me. My /etc/make.conf was partly taken from the Gentoo Wiki on the HP 2133 and MSI Wind (which has the same Intel Atom processor). Two problems arose, Xorg.conf and the Broadcom BCM4322 802.11a/b/g/n WLAN Controller.

I solved the Xorg.conf by installing xf86-video-intel. With some help from the Gentoo Wiki.

emerge xf86-video-intel

Then it worked with this /etc/xorg.conf

Section "Device"
Identifier "Intel Corporation Mobile 945GME Express Graphics Controller"
Driver "intel"
Option "AccelMethod" "UXA"
VendorName "Intel(R) DEG"
BoardName "Embedded Graphics"
BusID "0:2:0"
Screen 0
EndSection

Section "Screen"
Identifier "Default Screen"
Device "Intel Corporation Mobile 945GMEExpress Graphics Controller"
Monitor "Generic Monitor"
DefaultDepth 24
Subsection "Display"
Depth 24
Modes "1024x600" "800x600"
ViewPort 0 0
# Virtual 1024 768
EndSubsection
EndSection

The Wireless is yet to be fixed.

Update (26/7-09): Wireless is fixed!

Use: emerge net-wireless/broadcom-sta and it will download the drivers for the NIC, there are some bugs and some problems (like dhcpcd for me..). Then there is one thing in the kernel that needs to be added, right now I dont remember which it was but I am going to find out soon. You also have to install unzip so broadcom-sta will work.

According to http://en.gentoo-wiki.com/:

“You need to have wireless setup in your kernel (tkip support as well) modules or built in works.”

Old make.conf

As about all of the config files are unavailable (unfortunately lost) these are all the files I have got.

History of the Internet

2009-01-09T11:32:00+00:00

History of the Internet from Melih Bilgil on Vimeo.

AntonLindstrom.com

What I like about estimations

Go func: Reducing cognitive load

Conclusion

See also

Year 2016 in review

Limit processes with cgexec

Links

Introduction to Apache Mesos

Introduction and concepts

Frameworks

Operating Apache Mesos

Conclusions

Related papers

Monitoring Mesos tasks with Prometheus

Operating systems: Introduction to processes

Process states

What does a process contain?

Process execution

Interprocess communication (IPC)

Signals

Pipes

References

Visual block inserting in Vim

Standalone checks with Sensu

Capture stdout in Golang

Moving onto Mesos

Speeding up

tcpdump OSPF

Into Docker

Apache rewrite maps

Adventures with Puppet and Sensu

Out of the Crisis

Deming's key principles

Starting out with grok in Logstash

hostedmunin.com and Puppet

SPDY on Nginx development release

Install

Run

MongoDB Replica Sets

Upgrade to replica sets

Config files

Check Status

Adding members

Why can't I query the slave?

Conclusion

Updating rack gems on Heroku

DNSSEC implementation in Sweden

Varnish and JenkinsCI

Using the Glesys beta API with Fog

Bridged IPv6 network on Hetzner box

Notes on Keepalived and DNS

DRDB on VirtualBox

DRBD

Heartbeat

MySQL

Airport Express and FreeRadius

London

Downtime in the Cloud

CloudFoundry and PaaS

Building for repeatability

Viewing binary files in hex

Pingdom and SSH

wroopia DNS API

The Practice of System and Network Admin

Skype

Virtualization

mc-auth Auth.log Scanner for MCollective

Jekyll on Heroku

Puppet Problem

When API does not help

Breaking a leg

Sinestro, EC2 and Xen

A test of Jekyll

Best thing setup

Movements

HP2140 files lost

Time = t

Current state

Alley.se