numerodix blog

Writing a new memory device - /dev/clipboard

Thu, 29 Apr 2021 20:40:05 +0000

April 29th, 2021

This article contains code for how to implement a simple device driver in the kernel. The quality of this code is... proof of concept quality. If you're planning to do serious kernel development please don't use this code as a starting point.

Today we're going to use what we learned last time about /dev/null and take it to the next level. We're going to write a new memory device in the kernel. We will call it /dev/clipboard. No, it's not an existing device. Go ahead and check on your system!

Why the name /dev/clipboard? Because our new device will work just like a clipboard does. First you write bytes to it, and those bytes are saved. Later you come back and read from the device and it gives you back the same bytes. If you write to it again you overwrite what was there before. (This is not one of those fancy clipboards with multiple slots where you can choose which slot to retrieve.)

Here's a demo of how it works:

# save to clipboard
$ echo "abc 123" > /dev/clipboard 

# retrieve from clipboard
$ cat /dev/clipboard
abc 123

For convenience we will also print a message to the kernel log whenever someone writes to or reads from the clipboard:

$ dmesg  # just after the 'echo' command
Saved 8 bytes to /dev/clipboard

$ dmesg  # just after the 'cat' command
Returned 8 bytes from /dev/clipboard

Pretty neat, huh? Being able to just add a feature to the kernel like that.

We'll do all our work in drivers/char/mem.c, right next to where /dev/null and the other /dev/xxx memory device drivers live.

First, let's think about where the user's bytes will be kept.. they obviously have to be kept somewhere by the kernel, in the kernel's own memory. We'll keep it simple here and just use a static array of a fixed size. That means we're placing this array in one of the data segments of the kernel's binary, a segment that will be loaded and available at runtime. We'll use 4096 bytes as the size of the array, and that means we're adding 4kb to the kernel's memory footprint (so we don't want to make it too big):

#define CLIPBOARD_BUFSIZE 4096
static char clipboard_buffer[CLIPBOARD_BUFSIZE];
static size_t clipboard_size = 0;

CLIPBOARD_BUFSIZE is a macro that decides how big the fixed array is - a constant. clipboard_buffer is the array itself.

clipboard_size is a variable that reflects how many bytes of user data the array currently holds. This will change every time we write to the clipboard.

Okay, that's it for storage. Now let's turn to how we tell the kernel that this new device exists. We'll add it to the bottom of the list of devices implemented in this file:

static const struct memdev {
	const char *name;
	umode_t mode;
	const struct file_operations *fops;
	fmode_t fmode;
} devlist[] = {
	 [DEVMEM_MINOR] = { "mem", 0, &mem_fops, FMODE_UNSIGNED_OFFSET },
	 [2] = { "kmem", 0, &kmem_fops, FMODE_UNSIGNED_OFFSET },
	 [3] = { "null", 0666, &null_fops, 0 },
	 [4] = { "port", 0, &port_fops, 0 },
	 [5] = { "zero", 0666, &zero_fops, 0 },
	 [7] = { "full", 0666, &full_fops, 0 },
	 [8] = { "random", 0666, &random_fops, 0 },
	 [9] = { "urandom", 0666, &urandom_fops, 0 },
	[11] = { "kmsg", 0644, &kmsg_fops, 0 },
	[12] = { "clipboard", 0666, &clipboard_fops, 0 },
};

And we need to populate a file_operations struct. This is how we tell the kernel what a user can actually do with this new device. In our case, we want the user to be able to open it, read from it, or write to it, then close it. To achieve that we just need to implement a read and a write function. We can populate the other fields with stub functions that do nothing:

static const struct file_operations clipboard_fops = {
	.llseek		= null_lseek,
	.read		= read_clipboard,
	.write		= write_clipboard,
	.read_iter	= read_iter_null,
	.write_iter	= write_iter_null,
	.splice_write = splice_write_null,
};

Alright, we need a read function and a write function. Let's start with write first, because it's what the user will use first. What does the write function need to do?

Well, when the user writes to the clipboard that write should overwrite whatever was in the clipboard before (as we said before). So let's zero the memory in the array.

Next, we have an array of a fixed size. The user might write a byte string that fits or a byte string that exceeds our capacity. If the input is too big to store, we will just truncate the remainder that doesn't fit. (I haven't checked how other clipboards do this, but presumably they too have an upper limit.) Once we know how many bytes to keep, we'll copy those bytes from the user's buffer (which is transient) into our fixed array.

Then we'll print a message to the kernel log to explain what we did. And finally return the number of bytes the user sent us, in order to confirm that the write succeeded. Put all of that together and we get more or less the following:

static ssize_t write_clipboard(struct file *file, const char __user *buf,
			  size_t count, loff_t *ppos)
{
    // erase the clipboard to an empty state
    memset(clipboard_buffer, 0, CLIPBOARD_BUFSIZE * sizeof(char));

    // decide how many bytes of input to keep
    clipboard_size = count;
    if (clipboard_size > CLIPBOARD_BUFSIZE) {
        clipboard_size = CLIPBOARD_BUFSIZE;
    }

    // populate the clipboard
    if (copy_from_user(clipboard_buffer, buf, clipboard_size)) {
        return -EFAULT;
    }

    printk("Saved %lu bytes to /dev/clipboard\n", clipboard_size);

    // acknowledge all the bytes we received
    return count;
}

What's the EFAULT thing all about? copy_from_user is a function provided by the kernel. If it fails to copy all the bytes we requested it will return a non-zero value. This makes the boolean predicate true and we enter the if block. In the kernel the convention is to return a pre-defined error code prefixed with a minus sign to signal an error. We'll gloss over that here.

That does it for the write function. Finally, we need the read function.

The read function is a bit more tricky, because it's intended to be used to read incrementally. So a user can issue read(1) to read a single byte and then call read(1) again to read the next byte. Each time read will return the bytes themselves and an integer which is a count of the number of bytes returned. Once there are no more bytes to return read will return the integer 0.

How does the read function know how many bytes have been returned thus far? It needs to store this somewhere between calls. It turns out this is already provided for us - it's what the argument ppos is for. ppos is used as a cursor from the beginning of the array. We'll update ppos each time to reflect where the cursor should be, and the function calling our read function will store it for us until the next call.

Other than that, the read function is pretty analogous to the write function:

static ssize_t read_clipboard(struct file *file, char __user *buf,
			 size_t count, loff_t *ppos)
{
    size_t how_many;

    // we've already read the whole buffer, nothing more to do here
    if (*ppos >= clipboard_size) {
        return 0;
    }

    // how many more bytes are there in the clipboard left to read?
    how_many = clipboard_size - *ppos;

    // see if we have space for the whole clipboard in the user buffer
    // if not we'll only return the first part of the clipboard
    if (how_many > count) {
        how_many = count;
    }

    // populate the user buffer using the clipboard buffer
    if (copy_to_user(buf, clipboard_buffer + *ppos, how_many)) {
        return -EFAULT;
    }

    // advance the cursor to the end position in the clipboard that we are
    // returning
    *ppos += how_many;

    printk("Returned %lu bytes from /dev/clipboard\n", how_many);

    // return the number of bytes we put into the user buffer
    return how_many;
}

And that's it! That's all the code needed to implement a simple memory device in the kernel!

What happens when you pipe stuff to /dev/null?

Wed, 28 Apr 2021 00:50:33 +0000

April 28th, 2021

This question is one of those old chestnuts that have been around for ages. What happens when you send output to /dev/null? It's singular because the bytes just seem to disappear, like into a black hole. No other file on the computer works like that. The question has taken on something of a philosophical dimension in the debates among programmers. It's almost like asking: how do you take a piece of matter and compress it down into nothing - how is that possible?

Well, as of about a week ago I know the answer. And soon you will, too.

/dev/null appears as a file on the filesystem, but it's not really a file. Instead, it's something more like a border crossing. On one side - our side - /dev/null is a file that we can open and write bytes into. But from the kernel's point of view, /dev/null is a device. A device just like a physical piece of hardware: a mouse, or a network card. Of course, /dev/null is not actually a physical device, it's a pseudo device if you will. But the point is that the kernel treats it as a device - it uses the same vocabulary of concepts when dealing with /dev/null as it does dealing with hardware devices.

So if we have a device - and this is not a trick question - what do we need to have to able to use it? A device driver, exactly. So for the kernel /dev/null is a device and its behavior is defined by its device driver. That driver lives in drivers/char/mem.c. Let's have a look!

Near the bottom of the file we find an array definition:

static const struct memdev {
	const char *name;
	umode_t mode;
	const struct file_operations *fops;
	fmode_t fmode;
} devlist[] = {
	 [DEVMEM_MINOR] = { "mem", 0, &mem_fops, FMODE_UNSIGNED_OFFSET },
	 [2] = { "kmem", 0, &kmem_fops, FMODE_UNSIGNED_OFFSET },
	 [3] = { "null", 0666, &null_fops, 0 },
	 [4] = { "port", 0, &port_fops, 0 },
	 [5] = { "zero", 0666, &zero_fops, 0 },
	 [7] = { "full", 0666, &full_fops, 0 },
	 [8] = { "random", 0666, &random_fops, 0 },
	 [9] = { "urandom", 0666, &urandom_fops, 0 },
	[11] = { "kmsg", 0644, &kmsg_fops, 0 },
};

We don't need to understand all the details here, but just look at the names being defined: mem, null, zero, random, urandom. Where have we seen these names before? We've seen them endless times as /dev/mem, /dev/null, /dev/zero, /dev/random, /dev/urandom. And it's in this file that they are actually defined.

If we focus on the line that defines null we can see that it mentions something called null_fops. This is a struct that defines the behavior of /dev/null. And the struct looks like this:

static const struct file_operations null_fops = {
	.llseek		= null_lseek,
	.read		= read_null,
	.write		= write_null,
	.read_iter	= read_iter_null,
	.write_iter	= write_iter_null,
	.splice_write	= splice_write_null,
};

The values being used to populate this struct are function pointers. So when /dev/null is being written to the function that is responsible for this is write_null. And when /dev/null is being read from (it's not often we read from /dev/null) the function responsible for that is read_null.

Alright, we are just about to find out what happens when you write to /dev/null. The moment you have been waiting for. Are you ready for this? Here we go:

static ssize_t write_null(struct file *file, const char __user *buf,
                          size_t count, loff_t *ppos)
{
	return count;
}

write_null gets two arguments that are of interest to us: the bytes that we sent to /dev/null - represented by buf, and the size of that buffer - represented by count. And what does write_null do with these bytes? Nothing, nothing at all! All it does is return count to confirm that "we've received your bytes, sir". And that's it.

Writing to /dev/null is literally calling a function that ignores its argument. As simple as that. It doesn't look like that because /dev/null appears to us as a file, and the write causes a switch from user mode into kernel mode, and the bytes we send pass through a whole bunch of functions inside the kernel, but at the very end they are sent to the device driver that implements /dev/null and that driver does... nothing! How do you implement doing nothing? You write a function that takes an argument and doesn't use it. Genius!

And considering how often we use /dev/null to discard bytes we don't want to write to a (real) file and we don't want to appear in the terminal, it's actually an incredibly valuable implementation of nothing! It's the most useful nothing on the computer!

htop: cpu and memory usage stats

Tue, 13 Apr 2021 01:55:33 +0000

April 13th, 2021

htop is an enhancement over regular top and it's a very popular tool. But did you ever ask yourself how it actually works? In this article we'll be looking at where htop gets cpu and memory utilization information from. Given that htop runs on many different platforms, we'll be discussing just the Linux version of the story.

Cpu utilization per cpu

htop displays cpu utilization for each cpu. This is one the most key things we use htop for, in order to gauge the current load on the system.

The information comes from /proc/stat (documented here). This file contains a few different counters, but what's of interest to us right here are just the cpu lines which look like this:

cpu  5183484 9992 1575742 162186539 903310 0 27048 0 0 0
cpu0 1355329 2304 389040 40426679 299055 0 6431 0 0 0
cpu1 1234845 2602 423662 40594393 187209 0 16487 0 0 0
cpu2 1347723 2837 413561 40442958 239035 0 4085 0 0 0
cpu3 1245586 2246 349478 40722507 178009 0 44 0 0 0

So what are these numbers? Well, each number represents the amount of time spent in a particular state, by that cpu. This is true for each line that begins with cpuN. So cpu0 (which in htop is displayed as cpu #1) spent:

1355329 units of time in user mode (ie. running user processes)
2304 units of time in nice mode (ie. running user processes with a nice setting)
389040 units of time in system mode (ie. running kernel processes)
40426679 units of time in idle mode (ie. not doing anything)
299055 units of time in iowait (ie. waiting for io to become ready)
0 units of time servicing interrupts
6431 units of time servicing soft interrupts
0 units of time where the VM guest was waiting for the host CPU (if we're running in a VM)
0 units of time where we're running a VM guest
0 units of time where the VM guest is running with a nice setting

The first line in the file is simply an aggregate of all the per-cpu lines.

This is effectively the granularity that the kernel gives us about what the cpu spent time doing. The unit is something called USER_HZ, which is 100 on this system. So if we spent 1,355,329 units in user mode, that means 1355329 / 100 = 13,553 seconds (3.76 hours) spent running user processes since the system booted. By contrast, we spent 4.67 days in idle time, so this is clearly not a system under sustained load.

So how does htop use this? Each time it updates the ui it reads the contents of /proc/stat. Here are two subsequent readings one second apart, which show just the values for cpu0:

# time=0s
cpu0 1366294 2305 392684 40566185 300222 0 6590 0 0 0
# time=1s
cpu0 1366296 2305 392684 40566283 300222 0 6590 0 0 0

# compute the delta between the readings
cpu0 2 0 0 98 0 0 0 0 0 0

We can see that between the first and the second reading we spent 2 units in user mode and 98 units in idle mode. If we add up all of the numbers (2 + 98 = 100) we can see that the cpu spent 2 / 100 = 2% of its time running user processes, which means cpu utilization for cpu0 would be displayed as 2%.

Cpu utilization per process

Cpu utilization per process is actually measured in a very similar way. This time the file being read is /proc/<pid>/stat (documented here) which contains a whole bunch of counters about the process in question. Here it is for the X server:

939 (Xorg) S 904 939 939 1025 939 4194560 233020 15663 1847 225 297398 280532 25 14 20 0 10 0 4719 789872640 10552 18446744073709551615 93843303677952 93843305320677 140727447720048 0 0 0 0 4096 1098933999 0 0 0 17 3 0 0 3235 0 0 93843305821872 93843305878768 93843309584384 140727447727751 140727447727899 140727447727899 140727447728101 0

Fields 14 and 15 are the ones we are looking for here, because they represent respectively:

utime, or the amount of time this process has been scheduled in user mode
stime, or the amount of time this process has been scheduled in kernel mode

These numbers are measured in the same unit we've seen before, namely USER_HZ. htop will thus calculate the cpu utilization per process as: ((utime + stime) - (utime_prev + stime_prev)) / USER_HZ.

htop will calculate this for every running process each time it updates.

Memory utilization on the system

htop displays memory utilization in terms of physical memory and swap space.

This information is read from /proc/meminfo (documented here) which looks like this (here showing just the lines that htop cares about):

MemTotal:        3723752 kB
MemFree:          180308 kB
MemAvailable:     558240 kB
Buffers:           66816 kB
Cached:           782608 kB
SReclaimable:      87904 kB
SwapTotal:       1003516 kB
SwapCached:        13348 kB
SwapFree:         317256 kB
Shmem:            313436 kB

Unlike the cpu stats, these are not counters that accumulate over time, they are point in time snapshots. Calculating the current memory utilization comes down to MemTotal - MemFree. Likewise, calculating swap usage means SwapTotal - SwapFree - SwapCached.

htop uses the other components of memory use (buffers, cached, shared mem) to color parts of the progress bar accordingly.

Memory utilization per process

Memory utilization per process is shown as four numbers:

virtual memory, ie. how much memory the process has allocated (but not necessarily used yet)
resident memory, ie. how much memory the process currently uses
shared memory, ie. how much of its resident memory is composed of shared libraries that other processes are using
memory utilization %, ie. how much of physical memory this process is using. This is based on the resident memory number.

This information is read from the file /proc/<pid>/statm (documented here). The file looks like this:

196790 9938 4488 402 0 28226 0

This is the X server process once again, and the numbers mean:

virtual memory size
resident memory size
shared memory size
size of the program code (binary code)
unused
size of the data + stack of the program
unused

These numbers are in terms of the page size, which on this system is 4096. So to calculate the resident memory for Xorg htop does 9938 * 4096 = 38mb. To calculate the percentage of system memory this process uses htop does (9938 * 4096) / (3723752 * 1024) = 1.1% using the MemTotal number from before.

Conclusion

As we have seen in this practical example the kernel provides super useful information through a file based interface. These are not really files per se, because it's just in-memory state inside the kernel made available through the file system. So there is minimal overhead associated with opening/reading/closing these files. And this interface makes it very accessible to both sophisticated programs like htop as well as simple scripts to access, because there is no need to link against system libraries. Arguably, this API makes the information more discoverable because any user on the system can cat files on the /proc file system to see what they contain.

The downside is that these are files in text format which have to be parsed. If the format of the file changes over time the parsing logic may break, and a program like htop has to account for the fact that newer kernel versions may add additional fields. In Linux there is also an evolution in progress where the /proc file system remains, but more and more information is exposed through the /sys file system.

Un poème pour Didier

Sun, 22 Jul 2018 00:21:55 +0000

July 22nd, 2018

Une fois, Aimé, avec son capitaine, Didier, a tout gagné.

Ensuite Didier, vingt ans après, son objectif à bien fixé.

Parmi tous les joueurs, une équipe il a dû sélectionné.

Il a pas tardé, que tout le monde lui est venu la contesté.

Patiemment, tous les adversaires il les a éliminé.

Même les sublimes croates, ils ont le mieux essayé.

À la fin, quand même, c'est Didier qui a gagné.

Backup/reinstall checkist

Wed, 14 Mar 2018 08:50:55 +0000

March 14th, 2018

Backup checklist

* Top level dirs in ~

* Dot files/dirs in ~

* dpkg -l

Prepare install media

dd if=image.iso of=/dev/sdX bs=4M status=progress oflag=sync && sync

Boot from install media

Switch BIOS to use SATA in AHCI mode, not RAID mode. Otherwise the drive may not be detected.

To see block devices: lsblk

To see details of hardware: lspci -vvv or lshw

Reinstall checklist

* /etc/sudoers

* apt: curl emacs-gtk evince gimp git gitk htop iotop ipython3 kdiff3 mpv net-tools network-manager-openconnect openssh-server python3-pip ttf-bitstream-vera ttf-mscorefonts-installer vim-gtk vlc yakuake

* pip: nametrans reps

* rustup: ripgrep

* manual: chrome dropbox skype vscode zoom

* ~/.xmodmaprc (use xev to discover mouse button numbers)

* common_local.sh

* re pull

Config checklist

* focus follows mouse, 0ms delay

* kde: desktop search off

* kde: startup with empty session

* kde: taskbar do not group items

* konsole: set font size & scrollback buffer for profile, set profile as default

* browsers: smooth scrolling off (chrome: chrome://flags/#smooth-scrolling)

Test checklist

* office vpn

ttf-bitstream-vera

django girls den haag 2016 welcome speech

Sat, 05 Mar 2016 21:26:56 +0000

March 5th, 2016

Given on March 5, 2016 at Django Girls Den Haag 2016 in Den Haag.

Welcome, everyone!

We are thrilled that you're all here. Some of you have come far to be here today with us and we really appreciate it. We have people from all over the Netherlands!

I also wish to thank our sponsors. Without the sponsors this event just wouldn't be possible. Thank you: Amazon, Elastic, KPMG and GingerPayments!

---

Today is all about learning. You're about to learn programming, which is a completely new skill for most of you. It's going to be exciting. And some of it will be difficult. Learning *is* about pushing your boundaries, and it takes effort to learn anything.

I know it will seem as though your coaches know everything. Later today you might be asking yourself: "how am I supposed to master all this? There is so much new information here!" You may feel overwhelmed. But I promise you that every programmer in this room, myself included, has been through that. It's completely normal, and you're on the right track. So don't feel bad about that!

I want you to make mistakes today. Making mistakes is a good thing - whenever you make a mistake you learn something.

And don't hesitate to ask questions. If you get an answer that you don't fully understand, keep asking! Don't settle for answers you don't understand.

Finally, don't work so hard that you forget to take a break and get to know the people around you!

---

I will now introduce the organizing team. Raise your hand when you hear your name.

So I'm Martin, you've been getting my emails. We also have Chris. Robbie. Alexandra. And Tom.

So we are the organizers and you might think this means we know what we're doing. But no. We're just a bunch of software developers. We've never tried to organize an event for 100 people before.

Speaking for myself I had many moments of self doubt. How are we going to find a space big enough to host the workshop? How are we going to find enough volunteers to be coaches? How are we going to find all of you?

We made a lot of mistakes. And we learned from them.

We did it together as a team. It took effort, but it was also a pleasure.

---

I hope all of you have met your coach by now. We have 26 coaches with us today. They are nice enough to spend their Saturday to make Django Girls possible, isn't that awesome? They've *also* come from all over the country.

Please raise your hand when you hear your name.

Could we have a round of applause for the coaches?

---

Django Girls is a global initiative. It all started two years ago at a technical conference in Berlin. It was a big international conference with people from all over Europe. Now you should know that IT conferences are very male dominated. Two of the participants of that conference, the founders of Django Girls - Ola and Ola, had a simple idea: what if we organize a workshop for beginners - and for women! So they organized the very first Django Girls in 2014.

Since then it has spread to 57 countries. There have been over 100 events already. This is the sixth one in the Netherlands.

More than 3,500 women have participated! Many have gone on to become coaches. And you remember that very first Django Girls workshop? 80% of the participants there have organized their own Django Girls events since then in their country.

---

Now for some practical information.

Today's workshop lasts from 9 to 5pm. You can find the schedule for today on the posters.

The bathrooms are right around the corner, out that door and to the right.

We have coffee, tea and snacks available now. We have a limited number of cups, so please try to reuse your cup!

We have lunch coming in later on. It will be served in the lobby.

We have a mystery event in the afternoon.

We're a bit unsure of the wifi capacity with so many people using it all at once. So try to be gentle with the wifi connection. In case of problems it's possible to do big parts of the tutorial offline. Your coaches also have a copy of all the files on usb sticks you need today so you don't have to download them.

Last but not least we have a code of conduct. The code of conduct is in place to ensure that everyone can feel safe at our event. You can find the full text on one of the posters. If you see anyone breaking the code of conduct, please talk to one the organizers!

---

what django girls is

Wed, 24 Feb 2016 18:45:51 +0000

February 24th, 2016

Given on Feb 23, 2016 at the Dutch Django Meetup in Amsterdam.

Today I want to tell you about a part of the Django community that you may not even know about.

---

This is how I think most of us see the Python community. A comfortable couch. Nice people. Right? Just a nice place to be.

It's easy to make an analogy with software. Almost every developer has had the experience of working on a codebase that they inherited from someone else. The dreaded *legacy project*. And it wasn't the most fun project to work on. Spaghetti code, no tests, no documentation, really hard to make changes without breaking something.

So after working on this day in and day out after a while you start to get a little sick of it. I start thinking to myself "I've had it with this code. It expend so much energy on this and I have so little to show for it. Jeez, give me a codebase that's in good shape for a change. One that is modular, and has clean code. With good test coverage. If I had that I could *really* make it fly!"

I want to submit to you that the Python community is that nice codebase. It's when your code is in good shape and is easy to work on, that it's a great starting point to make it even better. To take the next step.

I'm sure you've heard this expression before: Came for the language, stayed for the community. People come to Python for the language, or for the libraries, or for the frameworks (like Django). Right? Just because they need to get some work done. And then they realize: hey there are conferences, there are meetups, and there are great people to hang out with. So they stay with Python not necessarily because it's the best programming language in the world, but rather because the community is a great place to be. It makes them feel at home.

So I'm saying: let's take the next step. Let's make it even more welcoming, even more inclusive!

---

Let me tell you about what we do. Since you're programmers I will explain this as a program. Django Girls is a one day programming workshop for women. Our mission is to bring in women who are new to IT and get them hooked on programming. Now the workshop is the box you see in the middle.

Then as input we take non-programmers. Most people who attend Django Girls have never written a program before. During the workshop they build a blog, from scratch. So they come in, never written a program before, and they go home, they've built a blog. Amazing, right? They do this following a tutorial which is very beginner friendly, so it explains everything they need to know.

Also as input we take developers. These are the programming coaches. They don't have to be professional developers, just have enough experience with programming to help someone else through the tutorial.

As output, we get programmers. They are beginners, of course, but they are programmers. We've given them the first step on a path that could potentially lead them to this room. Some percentage of Django Girls participants land junior developer jobs a few months down the road. But not everyone wants to be a developer, and that's fine. Some people are academics, and they want to learn how to automate parts of their work. Some work in marketing, and work with developers and want to understand what their colleagues do. And some people are just curious about programming and want to try it. We encourage all different demographics.

And also output, we get the same developers, and you may think all they did was volunteer a day of their time and that's it. But they've just spent a whole day coaching people in programming. If you're already a mentor to people at your job, that's excellent! But if you're not, coach at Django Girls for a day and you get really good practice that will come in very useful in your career, believe me.

---

So Django Girls is coming to Den Haag Saturday next week. We're going to be on the top floor of this office building. If you're familiar with Den Haag it's right next to the station Hollands Spoor.

We received applications to participate from more than 100 women. We're also lucky enough to have found more than 30 coaches to help us.

Unfortunately we only have space to host 60 participants, so we had to turn down a lot of people.

But the level of interest has been really impressive. We have people coming from cities all over the country, like Groningen and Leuwwarden, Nijmegen and Deventer. We have people coming from Belgium, where Django Girls hasn't taken root yet.

And then there's the 5 of us, the organizers. A few of us went to a Django Girls event as coaches and we loved it. We decided we wanted to organize one of these ourselves, in our city. Now we're just a bunch of software developers, we've never organized an event like this before. But we somehow pulled it off.

If we can do it, you can do it. Think about it.

---

Now, Django Girls has been organized before in the Netherlands. There is a group of organizers in Groningen who are the pioneers in our country - and they're planning another even this year. It's also been organized in Amsterdam before. And then there's our event, which is the upcoming one.

But like I said there is a lot of interest from people out there for a workshop like this. I would love to see more cities on this list in the future. So you could organize one. And you can be a coach! Most organizers have trouble finding enough coaches, and if they had more coaches they could accept more participants!

Thank you!

---

You can find the slides on Slideshare.

why you should coach at django girls

Mon, 22 Feb 2016 19:31:14 +0000

February 22nd, 2016

Given on Dec 10, 2015 at PyLadies Amsterdam.

Dear PyLadies,

My name is Martin and I'm one of the organizers of Django Girls The Hague. Which will take place next year in the beginning of March.

Django Girls is a non-profit organization that aims to give women a really positive first experience with programming. I'm not going to explain why we think this is important, because I think if you're a member of PyLadies you already know that.

I will explain how we do it. We invite a room full of women to participate. We seat a group of 3 women around a table. And we give them a programming coach. What they will be doing is building a web application from scratch. Using a tutorial special made for Django Girls. Now, the programming coach is there to help them, to explain things, and to give moral support.

Now, the coaches are integral to the event. We as organizers can provide a nice room in a nice location, a good tutorial, and food etc, but the coach is the human factor.

We are currently trying to find these coaches. That's why I'm here today. I would really like to encourage you to sign up to be a coach.

I've been a coach at two Django Girls events in the past. And in fact tomorrow morning I'm flying to Italy to be a coach at Django Girls in Rome on Saturday. It's a really fun thing to do.

The people who come to Django Girls aren't the kind of people who say alright, so what is this programming thing all about? If it doesn't take more than 5 minutes I'm willing to give it a try and then I'll be on my way. These are people who decide to spend 8-9 hours learning something new, that they've never done before. I mean can you imagine? It takes little courage to do that. So they're really interested and motivated to learn programming. And as a coach you see programming happen for the first time, through their eyes. It's really exciting. It's a really fun way to spend a day.

In terms of what it takes to be a coach. I want to make this very clear: You don't have to an expert on Python or Django. I would recommend that you work through the tutorial that we use yourself. Ask yourself if you understand everything. Ask yourself if you think you can help someone complete it. If so then you can be a coach. Ultimately if you encounter a problem at the workshop we will have some more experienced coaches to help you.

So if you think this sounds awesome and you want to help us make this possible then sign up to be a coach. You can sign up on our website, just google for Django Girls and find our chapter for The Hague, Den Haag. We also have a twitter feed where you can follow our updates.

We would love to have you there! Thank you very much!!

what would you do with 6 months?

Sun, 24 Jan 2016 11:49:07 +0000

January 24th, 2016

In the first months of 2015 I realized that I was burned out. Work was fun, but it had expanded so much that it had taken over my entire life. Fergus O'Connell gives this definition of burnout: you're at work, you're thinking about work, you're bringing work home with you, or you're cancelling other things so you can work. Open and shut case, Johnson.

I decided it was time to change, everything. It was my very own "everything stops!" moment. I would leave my job in June and take a six month break from work. From all work to no work. The cover story was "sabbatical", but the working title was "life improvement project".

And it has been the most liberating time of my life. There was much planning and project management involved - I'll spare you that. Here are the milestones.

Health

Lost weight.
Experimented with the paleo diet.
Saw a physical therapist.
Saw a psychologist.

Sports

Swam in the pool.
Biked intercity road trips.
Practiced yoga.
Practiced strength training.
Went running. Ran a 10km distance. Ran a 5km race.
Practiced interval training.
Played football.

Social

Saw friends.
Attended a Devnology weekend.
Gave pitches for Django Girls to recruit coaches and participants.
Took an improv class.
Took part in a theater workshop.
Hosted a friend for a long weekend.
Practiced speed dating.
Went on normal speed dates.
Attended my best friend's wedding.
Frequented 24 different social groups.
Attended 36 social events in total.
Met more new people than I've ever done before in 6 months.

Travel

To Dublin for a job interview.
To Friesland for Devnology weekend.
To Groningen for Django Girls.
To Bordeaux for Django Girls.
To Rome for Django Girls.

Language study

Had Skype sessions with Dutch tutors.
Completed a Portuguese course.
Attended language café meetups.
Read books in Portuguese.
Made polyglot-ish Youtube videos.
Wrote polyglot-ish blog entries.
Coached in French at Django Girls.
Coached in Italian at Django Girls.

Technology

Studied x86 assembly.
Read about computer architecture.
Learned some Prolog.
Wrote a memcached clone in Rust.
Read computer science papers.

Community work / Volunteering

Coached at three Django Girls events.
Taught a basic computer course at a community center.
Worked on organizing Django Girls Den Haag.
Submitted a talk proposal to PyCon Italia.

Career / Work

Reached out for career advice to my network.
Spoke to a lot of recruiters.
Interviewed with a lot of different companies, local and remote. Turned down many offers.
Applied for positions I wasn't qualified for. Was turned down.
Finally found the kind of companies that I was after. Accepted an offer.

Many of these were a first. Over time there has been a conscious trend from "things I do at home alone (in my pyjamas)" to "things I do with others". I started from a core of health and fitness and moved on to more and more social challenges.

two weeks of rust

Sun, 10 Jan 2016 11:53:40 +0000

January 10th, 2016

Disclaimer: I'm digging Rust. I lost my hunger for programming from doing too many sad commercial projects. And now it's back. You rock, Rust!

I spent about two weeks over the Christmas/New Year break hacking on emcache, a memcached clone in Rust. Why a memcached clone? Because it's a simple protocol that I understand and is not too much work to implement. It turns out I was in for a really fun time.

UPSIDES

The build system and the package manager is one of the best parts of Rust. How often do you hear that about a language? In Python I try to avoid even having dependencies if I can, and only use the standard library. I don't want my users to have to deal with virtualenv and pip if they don't have to (especially if they're not pythonistas). In Rust you "cargo build". One step, all your dependencies are fetched, built, and your application with it. No special cases, no build scripts, no surprising behavior *whatsoever*. That's it. You "cargo test". And you "cargo build --release" which makes your program 2x faster (did I mention that llvm is pretty cool?)

Rust *feels* ergonomic. That's the best word I can think of. With every other statically compiled language I've ever used too much of my focus was being constantly diverted from what I was trying to accomplish to annoying little busy work the compiler kept bugging me about. For me Rust is the first statically typed language I enjoy using. Indeed, ergonomics is a feature in Rust - RFCs talk about it a lot. And that's important, since no matter how cool your ideas for language features are you want to make sure people can use them without having to jump through a lot of hoops.

Rust aims to be concise. Function is fn, public is pub, vector is vec, you can figure it out. You can never win a discussion about conciseness because something will always be too long for someone while being too short for someone else. Do you want u64 or do you want WholeNumberWithoutPlusOrMinusSignThatFitsIn64Bits? The point is Rust is concise and typeable, it doesn't require so much code that you need an IDE to help you type some of it.

Furthermore, it feels very composable. As in: the things you make seem to fit together well. That's a rare quality in languages, and almost never happens to me on a first project in a new language. The design of emcache is actually nicely decoupled, and it just got that way on the first try. All of the components are fully unit tested, even the transport that reads/writes bytes to/from a socket. All I had to do for that is implement a TestStream that implements the traits Read and Write (basically one method each) and swap it in for a TcpStream. How come? Because the components provided by the stdlib *do* compose that well.

But there is no object system! Well, structs and impls basically give you something close enough that you can do OO modeling anyway. It turns out you can even do a certain amount of dynamic dispatch with trait objects, but that's something I read up on after the fact. The one thing that is incredibly strict in Rust, though, is ownership, so when you design your objects (let's just call them them that, I don't know what else to call them) you need to decide right away whether an object that stores another object will own or borrow that object. If you borrow you need to use lifetimes and it gets a bit complicated.

Parallelism in emcache is achieved using threads and channels. Think one very fast storage and multiple slow transports. Channels are async, which is exactly what I want in this scenario. Like in Scala, when you send a value over a channel you don't actually "send" anything, it's one big shared memory space and you just transfer ownership of an immutable value in memory while invalidating the pointer on the "sending" side (which probably can be optimized away completely). In practice, channels require a little typedefing overhead so you can keep things clear, especially when you're sending channels over channels. Otherwise I tend to get lost in what goes where. (If you've done Erlang/OTP you know that whole dance of a tuple in a tuple in a tuple, like that Inception movie.) But this case stands out as atypical in a language where boilerplate is rarely needed.

Macros. I bet you expected these to be on the list. To be honest, I don't have strong feelings about Rust's macros. I don't think of them as a unit of design (Rust is not a lisp), that's what traits are for. Macros are more like an escape hatch for unpleasant situations. They are powerful and mostly nice, but they have some weird effects too in terms of module/crate visibility and how they make compiler error messages look (slightly more confusing I find).

The learning resources have become very good. The Rust book is very well written, but I found it a tough read at first. Start with Rust by example, it's great. Then do some hacking and come back to "the book", it makes total sense to me now.

No segfaults, no uninitialized memory, no coercion bugs, no data races, no null pointers, no header files, no makefiles, no autoconf, no cmake, no gdb. What if all the problems of c/c++ were fixed with one swing of a magic wand? The future is here, people.

Finally, Rust *feels* productive. In every statically compiled language I feel I would go way faster in Python. In Rust I'm not so sure. It's concise, it's typeable and it's composable. It doesn't force me to make irrelevant nit picky decisions that I will later have to spend tons of time refactoring to recover from. And productivity is a sure way to happiness.

DOWNSIDES

The standard library is rather small, and you will need to go elsewhere even for certain pretty simple things like random numbers or a buffered stream. The good news is that Rust's crates ecosystem has already grown quite large and there seem to be crates for many of these things, some even being incubated to join the standard library later on.

While trying to be concise, Rust is still a bit wordy and syntax heavy with all the pointer types and explicit casts that you see in typical code. So it's not *that easy* to read, but I feel once you grasp the concepts it does begin to feel very logical. I sure wouldn't mind my tests looking a bit simpler - maybe it's just my lack of Rust foo still.

The borrow checker is tough, everyone's saying this. I keep running into cases where I need to load a value, do a check on it, and then make a decision to modify or not. Problem is the load requires a borrow, and then another borrow is used in the check, which is enough to break the rules. So far I haven't come across a case I absolutely couldn't work around with scopes and shuffling code around, but I wouldn't call it fun - nor is the resulting code very nice.

Closures are difficult. In your run-of-the-mill language I would say "put these lines in a closure, I'll run them later and don't worry your pretty little head about it". Not so in Rust because of move semantics and borrowing. I was trying to solve this problem: how do I wrap (in a minimally intrusive way) an arbitrary set of statements so that I can time their execution (in Python this would be a context manager)? This would be code that might mutate self, refers to local vars (which could be used again after the closure), returns a value and so on. It appears tricky to solve in the general case, still haven't cracked it.

*mut T is tricky. I was trying to build my own LRU map (before I knew there was a crate for it), and given Rust's lifetime rules you can't do circular references in normal safe Rust. One thing *has to* outlive another in Rust's lifetime model. So I started hacking together a linked list using *mut T (as you would) and I realized things weren't pointing to where I thought they were at all. I still don't know what happened.

The builder pattern. This is an ugly corner of Rust. Yeah, I get that things like varargs and keyword arguments have a runtime overhead. But the builder pattern, which is to say writing a completely separate struct just for the sake of constructing another struct, is pure boilerplate, it's so un-Rust. Maybe we can derive these someday?

Code coverage. There will probably be a native solution for this at some point. For now people use a workaround with kcov, which just didn't work at all on my code. Maybe it's because I'm on nightly? Fixed!

---

So there you have it. Rust is a fun language to use, and it feels like an incredibly well designed language. Language design is really hard, and sometimes you succeed.