There’s a famous saying about hammers and problems. I think it goes like if all you have is a hammer, everything looks like a nail. That’s a common thing to happen on the devops/sysadmin trade and goes completely against the UNIX Philosophy of small tools getting together to achieve big things.

Sometimes we will essentially try to fit the problem to the tool, just because we have all this variety of do everything tools. That, in my opinion, is not worth it. We like to call us devops, right? So let’s throw some dev’ing and create tools tailored to the tasks.

For example, it is possible to load balance, cache and serve PHP using Nginx. That doesn’t mean you should do that unless it’s a small install or non-critic server (go ahead and do it on your personal site, I mean, even I do it), but if you want to keep flexible and scalable, spread out and try to find the best of it’s kind… or the close second. Cache on Varnish, serve PHP from a different box (maybe using Apache), load balance on HAProxy (or Varnish really it’s very good at it).

We put this thought in practice at Vox Media all the time. Not only because we agree with it, it also helps us increase the visibility of our services and do a better job monitoring everything.

Don’t consider it a not built here syndrome. As I said, use the best available or make your own, and don’t feel bad about having to change later when something better appears. Constant evolution!

For example, those are two tools that I made for very specific objectives:

• slowqwatch: It watches a log and increases a StatsD counter when a regex is matched. Nothing great or fantastic, but allows us to measure rates and plot nice Graphite graphs of stuff like MySQL slow queries (hence the name).
• ipcount: This may have been a little over-engineered but I wanted to do some stuff using Redis and Go. What this does is basically watching access logs and sending the IPs to Redis, and then a web page shows the top IPs. You know, to catch those spidering robots that hammer your serves like crazy.

I am sure a lot of people will point me to different tools that do the same thing or/and are better at doing those things. The point is, we had an itch and we scratched it, custom made. Of course we can make the same with Splunk, but not every company has the money for that…

TL;DR: stop hitting your problems with the same tools over and over and think about expanding your toolbox, even if that requires you to create the tools from scratch.

cya

Doing some reading about cluster administration using SSH I stumbled into this:

1
2
3
4

Host *
  ControlMaster auto
  ControlPath /tmp/ssh_mux_%h_%p_%r
  ControlPersist 4h

What witchery is that, you ask? Has science gone too far? Well, no, that’s actually pretty old and I should be probably ashamed of only discovering it so recently.

Those options cause your client to create a master connection to the server you are connecting, with all SSH overhead (key exchange, session setup, authentication, etc) on your first connection. The real magic occurs when you open more connections to the same server: they piggyback on the master, and skip all the long ass initialization phase.

That’s a good thing if you (like me) spend most of your day logged into many servers at the same time, and even multiple times to the same server (because you forgot to run tmux).

Another neat thing is that when a master connection is open, you get a bonus tab-completion feature on scp for remote directories/files. You have to try it to see how much difference it makes.

The ControlPersist option is kinda of a controversy IMO because it means (on the above case) that the master connection will remain open for that amount of time after you disconnect the server. That helps a lot when you are going in and out of servers, but still a connection dangling around the network… so, YMMV.

To read the full description and caveats of each option, check the ssh_config man page, as always.

cya!

If in some of the commands below you see a lot of gibberish on the screen, check if you didn’t forget to add --armor to it. It makes the output to be ASCII text and not binary gibberish.

Step One – Generate a key pair:

You should have gnupg installed. Use your package manager to do that on Linux, or download it on Windows. If you use Mac OS X use the Homebrew version but be aware that there are some incompatibilities between GnuPG and MacGPG2/GPGTools that require the usage of a nightly version of the later.

This is a full session describing a key pair creation, just answer the questions and you will end with a fully functional PGP key pair ready to use:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62

% gpg --gen-key
gpg (GnuPG) 1.4.13; Copyright (C) 2012 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048)
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 10y
Key expires at Thu Apr 27 17:16:04 2023 EDT
Is this correct? (y/N) y

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"

Real name: Testing Smith
Email address: example@example.com
Comment:
You selected this USER-ID:
    "Testing Smith <example@example.com>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
You need a Passphrase to protect your secret key.

gpg: gpg-agent is not available in this session
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
.+++++
....+++++
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
+++++
+++++
gpg: key 64E4DFD8 marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   4  signed:   4  trust: 0-, 0q, 0n, 0m, 0f, 4u
gpg: depth: 1  valid:   4  signed:   0  trust: 0-, 0q, 0n, 3m, 1f, 0u
gpg: next trustdb check due at 2015-08-18
pub   2048R/64E4DFD8 2013-04-29 [expires: 2023-04-27]
      Key fingerprint = 3546 93CD 098B E5E6 0F05  5DBE F5BD 6066 64E4 DFD8
uid                  Testing Smith <example@example.com>
sub   2048R/9A0F9041 2013-04-29 [expires: 2023-04-27]

According to most literature, a 2048 bit key should be good for securing data until 2030, so a 10 year expiry date is fine. If you want to use more than 10 years use 4096 bit, but don’t go over 30 years expiry and NEVER make a key that doesn’t expire.

Also, NEVER create a keypair without a passphrase. And remember, it’s a passphrase not a password. Use something long, creative and that you won’t forget. It’s not like you are going to type it every 5 minutes.

Seriously if you just created a keypair that never expires or without a passphrase, delete it with gpg --delete-secret-key and create a new, proper one, or edit your ridiculously unsafe one with gpg --edit-key (using the commands passwd and expire and then save).

If at any point you don’t like or get something wrong, just press CTRL+C and start again, the key is only saved after those last lines are printed.

Step Two – Export your keys

You have a key pair. That means you have a private and a public key. The public key you want to make know to people to be able to encrypt/sign things to you while the private key you want to keep safe. For that reason, you want to export both keys to text files to facilitate both those tasks:

Exporting the public key to a file:

1

% gpg --armor --output public_key.gpg --export example@example.com

Exporting the private key to a file:

1

% gpg --armor --output private_key.gpg --export-secret-keys example@example.com

The private key file you want to save somewhere else. Put it on a thumb drive, backup disk, online backup, anywhere you are confident you won’t lose it. If you lose your private key you won’t be able to access anything that was encrypted using it. For reals, if you lose it, there goes the data.

The public file is another story. You can lose it, share it, put it on your Facebook, Twitter, send it via email, and if you lose it you can just regenerate it from your private key. Anyone that wants to encrypt or sign stuff for you must have it so make it easily accessible.

If you are a really nice guy you should send your public key to one of the key servers around the world, so people can search your email and retrieve your key automatically. For that you need your Key ID, and since this is a thing that people may ask you someday, this is how you find it:

1
2
3
4

% gpg --list-keys example@example.com
pub   2048R/64E4DFD8 2013-04-29 [expires: 2023-04-27]
uid                  Testing Smith <example@example.com>
sub   2048R/9A0F9041 2013-04-29 [expires: 2023-04-27]

When you list a key, the ID is the number after pub ??????/. In this case the ?????? is 2048R because the key is 2048 bit RSA, but that will vary. So in this case the key ID is 64E4DFD8. With that information you can do:

1

% gpg --keyserver pgp.mit.edu --send-keys 64E4DFD8

And your public key will be available on the PGP network after a few days (the servers are connected so they will replicate the key worldwide eventually)

Step Three – Encrypt and Decrypt stuff with your keys

Finally, after creating your keypair and exporting your keys, it’s time to do something useful with them. This is a simple example to encrypt a simple text to someone (in this case, myself):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

% gpg --armor --encrypt
You did not specify a user ID. (you may use "-r")

Current recipients:

Enter the user ID.  End with an empty line: jose

Current recipients:
4096R/17898967 2010-09-10 "José de Paula Eufrásio Júnior <jose.junior@gmail.com>"

Enter the user ID.  End with an empty line:
This is just a text message
-----BEGIN PGP MESSAGE-----
Version: GnuPG v1.4.13 (Darwin)
Comment: GPGTools - http://gpgtools.org

hQIMA7sOk1QXiYlnAQ//U74eBVPt7XLBOhvRAWtEJ2Q2L9eN5wsjtvQaPiTOHSiF
JiD8ZT9giPJLn2Dr2ViS6Qhhm3MOGc3V1SxpQvavigC6nIHgzTXPG5CdsiQgLPIk
iXrxpd3rnrnMiCMb/rodbEeYNiJ6q5OiztCfpjFNMBD4lwCSf7wbQNL/TUzADqlA
gIi1mH9Dw3r5PsY3mD6omoRj5MlVCpKDqnAj58iarvRm2NCUvdnWciwXX1KTBFM1
9EnXH3Kyox2RFW78xfh9/OVcX8UAmN25PdgtoSOQMBLLScslKn5ivlTxqhiTW/8N
XS0G1hkTMKJ6ya6vHshp+ei4pOYtqTlVoTZ5xYVc8jv6xhJC/mbTtC3TEdgrcrsY
UIA4F4BfzTi8Si8JeYK50wjdy+425Zqub7+CkYb8n2uTtyqZRrlvd3vvAYtbcmeO
TgD74CCAGomVmN0uQeLKxoFNE6Fre02wEg0m/37eEustpbI1qRtY4z5MHOFsIYU+
+xBSUEcYaVjK95KIpoaGjBA4l8FrwEQRK1/hlFgQkzm9kCiQGjaeOqXr69MFqvFs
OOkCo1asQGBv3CKTPVX/wg2vTi11KPtpBSIV2+pL+U/UwjG6Z7LF0y++y/7obYdu
QBTKPd9tnVVwuC47kENn33vq7454P70hxwHvDc1BWLHsrHf5GMq7lem8ibpLEEvS
VgFhC0qjKyvFD+Aa24BKqQusBiRhSuV/U2oCV/fnkTmIpjLJMAGlmQPojbrCw3lF
Dyv9VZpmb1do6MobHc+dAieKpptNL94yHxppakOIsCAXfADyKolm
=lBMX
-----END PGP MESSAGE-----

So I just typed jose so it could find my key on my keychain and added it as my single recipient, typed the message to be encrypted (“this is just a text message”) and finished it with an EOF signal (CTRL+D). The output is the encrypted message that you can send via email or IM or print on a piece of paper, and it’s all the content included between the -----BEGIN PGP MESSAGE----- and -----END PGP MESSAGE-----.

Decrypting is just as easy:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

% gpg --decrypt
-----BEGIN PGP MESSAGE-----
Version: GnuPG v1.4.13 (Darwin)
Comment: GPGTools - http://gpgtools.org

hQIMA7sOk1QXiYlnAQ//U74eBVPt7XLBOhvRAWtEJ2Q2L9eN5wsjtvQaPiTOHSiF
JiD8ZT9giPJLn2Dr2ViS6Qhhm3MOGc3V1SxpQvavigC6nIHgzTXPG5CdsiQgLPIk
iXrxpd3rnrnMiCMb/rodbEeYNiJ6q5OiztCfpjFNMBD4lwCSf7wbQNL/TUzADqlA
gIi1mH9Dw3r5PsY3mD6omoRj5MlVCpKDqnAj58iarvRm2NCUvdnWciwXX1KTBFM1
9EnXH3Kyox2RFW78xfh9/OVcX8UAmN25PdgtoSOQMBLLScslKn5ivlTxqhiTW/8N
XS0G1hkTMKJ6ya6vHshp+ei4pOYtqTlVoTZ5xYVc8jv6xhJC/mbTtC3TEdgrcrsY
UIA4F4BfzTi8Si8JeYK50wjdy+425Zqub7+CkYb8n2uTtyqZRrlvd3vvAYtbcmeO
TgD74CCAGomVmN0uQeLKxoFNE6Fre02wEg0m/37eEustpbI1qRtY4z5MHOFsIYU+
+xBSUEcYaVjK95KIpoaGjBA4l8FrwEQRK1/hlFgQkzm9kCiQGjaeOqXr69MFqvFs
OOkCo1asQGBv3CKTPVX/wg2vTi11KPtpBSIV2+pL+U/UwjG6Z7LF0y++y/7obYdu
QBTKPd9tnVVwuC47kENn33vq7454P70hxwHvDc1BWLHsrHf5GMq7lem8ibpLEEvS
VgFhC0qjKyvFD+Aa24BKqQusBiRhSuV/U2oCV/fnkTmIpjLJMAGlmQPojbrCw3lF
Dyv9VZpmb1do6MobHc+dAieKpptNL94yHxppakOIsCAXfADyKolm
=lBMX
-----END PGP MESSAGE-----
You need a passphrase to unlock the secret key for
user: "José de Paula Eufrásio Júnior <jose.junior@gmail.com>"
4096-bit RSA key, ID 17898967, created 2010-09-10 (main key ID 031161FF)

gpg: gpg-agent is not available in this session
gpg: encrypted with 4096-bit RSA key, ID 17898967, created 2010-09-10
      "José de Paula Eufrásio Júnior <jose.junior@gmail.com>"
This is just a text message

Paste the message, enter the passphrase for the private key, read the message. Simple like that.

Those are arguably the two most useless uses of GPG, though, but they are a very good way to show how it works on a quick and dirty way.

There are tools and plugins out there to make GPG usage painless and quick, like graphical interfaces for encrypting files, key management and plugins for mail clients for easy signing/encrypting of email messages, but creating and managing keys via the command line are good basics to pick up.

cya

One of the things I really like to do is learning new languages. So after leaving it on the back burner for almost a year I decided to really concentrate on the Mark Summerfield’s Programming Go book that I had since before printing (I had like 4 different rough cuts that I never started reading for serious). The book is very good, and is doing a fantastic job on giving me the basics on the language, and I am starting the more advanced chapters and not feeling totally lost.

Go contrasts a lot with the ones I use day-to-day (mainly python and ruby). It’s statically typed, compiled and uses pointers, so the growing pains are understandable. Like trying to print an URL but discovering that you can’t simply wait for the language to cast that down to a string automatically. I still get a lot of compiling errors just because I try to skip steps or use the wrong type on the wrong part.

There are some nice parts. Go doesn’t really do exceptions (unless it’s an exceptional behavior), so the traditional ‘ask forgiveness later’ thing will not help here. What Go and it’s standard lib and the good 3rd party libs do is always return at least an error code on it’s functions. So most of the time you will have things like this:

1
2
3
4
5

data, err := json.Marshal(result_struct)
if err != nil {
    fmt.Printf("Error ocurred while marshaling data: %v\n", err)
    // do stuff needed to recover from the error and such
}

So you catch errors as they happen, instead of catching an exception 4 levels higher with a lot of crappy stack trace. Error messages in the Go standard library are also very good, no more cryptic voodoo and Googling just to discover wth is that message.

As soon as I got a little confidence on at least how to organize a Go program I decided I wanted to try and start doing some small things, some book exercises and what not. So I got inspiration from XKCD and created the Is it creepy? site. Want to know if that person is too old or too young for you? Check it.

Note that the code is available on Github so if you want to check how Go looks, go ahead. Remember that this is a first program, and also that I over engineered some simple parts to check more advanced techniques (for my current level). Now I have some more ambitious plans for projects, but those will need to wait a little more reading and practice.

Thanks to Yves Junqueira, that annoyed the hell out of me until I finally started putting effort in learning the language.

cya

PS: Changed the theme to light solarized. I like it.

Why I stopped using Ubuntu

tl;dr: Canonical crap. I am using Arch Linux.

Debian is great. There’s nothing better than apt in the planet to manage packages. Debian Project’s commitment to free software is also great. The thing is that you end having to use Debian testing or unstable to have access to software on a timely manner. With time, you just decide that using Ubuntu is a good time saver: you install stuff that is basically Debian Sid (unstable) and it just works 98% of the time.

The problem is the whole Canonical cruft and crap that they add to Debian. I was watching Artur Bergman talk during 2011 Surge Conference and part of his annoyed sighs was aimed at distros and the Linux kernel trying to do stuff more user friendly that ends being stupid on servers. That’s how I felt with Ubuntu sometimes, in their effort to make an user friendly desktop they end pissing off users like me. Some of the ideas that I find horrible: upstart, the new display manager and don’t even get me started on that ugly Compiz patch they call Unity Desktop. Why they could’t just go with Gnome 3 that is great and works fine? Who knows.

All that quest for user friendleness ends adding a lot of crappy configs and layers and layers that, as all complex systems do, fails in miserable ways. It’s not a common thing to happen I agree, but in the last Ubuntu upgrade my notebook started acting up like a spoiled child. Each update consisted in installing Ubuntu and removing every bad idea that Canonical included so I could get a system that, for me, was the ideal. In the end I was faced with the prospect of a full reinstall, so I went all the way and simply changed distros to Arch Linux.

From their wiki:

Arch Linux is an independently developed, i686/x86-64 general purpose GNU/Linux distribution versatile enough to suit any role. Development focuses on simplicity, minimalism, and code elegance. Arch is installed as a minimal base system, configured by the user upon which their own ideal environment is assembled by installing only what is required or desired for their unique purposes. GUI configuration utilities are not officially provided, and most system configuration is performed from the shell by editing simple text files. Based on a rolling-release model, Arch strives to stay bleeding edge, and typically offers the latest stable versions of most software.

Is it easy and user friendly? No. How many fucks I give? 0

At the end of the first install I had choices. Arch Linux uses that braindead BSD style init, but systemd was just a pacman install away and I took the plunge. I know I complained about systemd before but after using it for two months I want to change all my linux computers/server to use it. It’s a pretty ingenious software, if you ignore the whole journald thing and use syslog.

I give credit that I only changed to Arch because my co-worker Alfredo uses it and kept bugging me to give it a try, so I did. Arch is a great distro for devs/sysadmins to have in their work computers because it’s simple, updated and made for people that know how to use a computer — we don’t need our computers to be friendly, we need them to be well oiled tools that integrate our body as an extra brain/member or anything cyberpunk-compatible you can think.

BTW you should check the crazy things Alfredo does. He is like a greasy monkey/coder that builds robots and monster cars on his free time.

Why I stopped using Gnome

tl;dr: Tiling Window Managers, more keyboard and console use. Dotfiles at the end of the post.

After I changed from Ubuntu to Arch I started feeling that I was missing something. Had I been using mainstream stuff for so long that I forgot that Linux is about using the tools that you like best? Not that I dislike Gnome 3, I love it, but what else existed around?

Well, Tilling Window Managers of course. Those weird, beautiful things. (I hope you read the Wikipedia article, I am not explaining what TWMs are).

I tried xmonad and Awesome WM, and ended deciding for Awesome because of the way xmonad deals with multihead setups (it’s really weird). Awesome also uses Lua as ‘customization/configuration’ language, Xmonad uses Haskell. I felt like I had better chances debugging lua than haskell (I can’t even understand Haskell code, even more fix stuff).

TWM and Awesome, specifically, are aimed at focusing on keyboard usage more than the mouse. To be honest I currently use 80% keyboard when doing my work. TWM are also great for people that like me spend most of the time dealing with terminals. Tiling was what I was doing when using multiple terminals on Terminator, so why don’t use a WM that actually knows how to deal with tiled windows? And why lose screen space showing the useless desktop and what not? Makes no sense at all.

This also caused me to focus more on console applications. At this moment I have command line clients for EC2 (So I can skip using the web interface), a screenshot tool, a script that uploads images to imgur.com, music player, IRC/Jabber/Twitter client using bitlbee, etc. The extra benefit is that my computer uses way less memory than what it used when running all the bloat necessary for Gnome. If you check the screenshots you will notice that I still using Network Manager but it’s an independent tool, I also use Gnome Keyring to manage SSH and GPG keys: those tools are simply the best in their areas and replacing that would be not a smart move (believe me I tried to replace NM with netcfg and that thing is horrible).

Finally, I’ve got a dotfile repo with my configurations, mostly because it’s good to keep track of changes I do to my setup but if someone wants to get some ideas, feel free.

It’s a never ending process, I guess. That’s the beauty of FLOSS in general: choice to make things work the way you want and expand from there. No walled garden here, just a plain full of space and adventure.

cya

From bash to zsh

I changed my shell after years using bash. zsh makes for a somewhat better experience and has some options that really help. I tried to use fish some years ago but it let me down, so I was a little scared of changing shells again but now after almost two weeks using zsh I am comfortable to say that it was a good choice.

You can check my .zshrc on this link. I based it on @wisqnet’s one plus some bashisms I like (!$ comes to mind). This is a screenshot from a simple shell showing the rake command completion options inside a git repo (master branch, uncommited changes):

From rvm to rbenv

I never had any big problems with RVM, but rbenv seems to be so less intrusive and works so well that it was a no-brainer. Remember to install the ruby-build too so you don’t suffer too much to install new rubies (it provides the rbenv install command).

From Shopify to Vox Media, Inc.

After almost a year working with the fine people at Shopify I changed airs to Vox Media, Inc. Vox Media is the parent company for all star sites like The Verge, Vox Games (currently it’s inside The Verge but stay tuned) and SBNation.

Change is always hard and I enjoyed my time at Shopify but I am sure that the new challenges at Vox Media will keep me happy for a long time too. The team is awesome and it’s the kind of work I enjoy.

I think this is it.

cya

1

Feb 22 13:28:27 serverXX kernel: [1953364.275653] gm[16356]: segfault at 0 ip 00007fd137bd41e0 sp 00007fff5770dcd0 error 6 in libGraphicsMagick.so.3.7.0[7fd1379b9000+29d000]

No core dumps since ulimit -c is zeroed. What to do to at least have an idea of what is happening?

Well luckily I build the packages for our internal use so I had he build directory available with the unstripped binaries, with that it’s trivial to use the GNU Debugger (gdb) and find what is going on.

First, notice that the segfault is happening on a shared lib, this is per se a complication. You see, when you have the segfault to happen on a non-shared lib binary the ip (instruction pointer) value points to the instruction on the binary, in this case it is pointing to a shared lib, dynamically linked on the gm binary.

To find the instruction, then, subtract the offset given on the segfault message (it’s the 7fd1379b9000 part after the lib’s name) from the ip:

1

00007fd137bd41e0−7fd1379b9000 = 21B1E0

Finally, using GDB you can check what is happening @ that addres on the library, provided you have an unstripped object (you can get it with -dbg packages on debian/ubuntu):

1
2

(gdb) info symbol 0x21B1E0
WriteOnePNGImage + 13648 in section .text

There’s the culprit. You can also find some info on the stripped library using nm, remember that nm will not show anything on shared libs if not used with the -D option (showing just part of the output):

1
2
3
4
5
6
7
8
9
10
11

root@XXX:~# nm -D /usr/lib/libGraphicsMagick.so.3.7.0                                                                                                                                                             
0000000000104650 T AccessCacheViewPixels
0000000000104700 T AccessDefaultCacheView
00000000000ea030 T AccessDefinition
00000000001061e0 T AccessImmutableIndexes
0000000000106170 T AccessMutableIndexes
[...]
0000000000210d50 T RegisterJP2Image
0000000000213170 T RegisterPNGImage
0000000000210cf0 T UnregisterJP2Image
0000000000213110 T UnregisterPNGImage

You can see that there are some PNG related symbols around the address 0x21xxxx. If you check the code for GraphicsMagic PNG support you will see that WritePNGImage is part of the RegisterPNGImage code.

In this case I correlated the logs and found that the request that caused the segfault completed without problems and the PNG image was correctly generated, so my conclusion is that the segfault is happening on some non-crucial part of those functions, but there’s not a lot of things to do exactly pinpoint the problem.

gdb, nm and ldd are powerful tools when debugging or trying to do a postmortem on a segfault. It would be easier to find what exactly is going on with a core dump and maybe more info.

cya!

Second: most of you probably know the #monitoringsucks movement/hashtag/discussions. I totally agree that the current monitoring tools only do part of the job and getting them to work together is horrible. I have some ideas on how to solve the problem, but the path from idea to code is a long one. You can read this post about monitoring and understand what is this all about.

Even so, if you are using HoardD with the default sample time and sending intervals, you are getting data 1 minute old. This is normally cool for simple graphing but if you want a realtime feeling I suggest you to send data everytime you sample it. Your graphs will also look better, and the overhead of 6 connections each minute is negligible on a true server.

cya

So I decided to flex my node.js dev muscles and here is HoardD. This is a node.js app written in coffee-script that basically runs scripts and tools to get information about a server and sends it to carbon (Graphite’s storage backend). It’s easily expansible to include more metrics and very very fast and small (11MB or so, depending on scripts loaded, most of it is node).

At this moment there are basic scripts (uptime, memory, cpu, vmstat) and also 2 DB specific ones (redis and mysql). I will be fixing and adding scripts as I need on my day to day work, but the tool is already very usable. I am accepting push requests and patches to.

You can get HoardD on my github repo, there’s more info on the README there.

Below are some examples of graphics created with Graphite using the metrics sent by HoardD:

CPU stats from vmstat. Not a very busy server right?

Redis memory usage.

MySQL Bytes transferred and received:

MySQL counters for various operations:

Catching the exact moment when InnoDB flushes its dirty pages

MySQL slave lag:

cya!

One problem that I had for some weeks — after some upgrade I am sure — was that Alt+Tab and Alt+(key below ESC) behavior with Google Chromium windows was completely annoying: all my browser windows, including the ones I use --app and a different data directory, were grouped under “Google Chromium” icon, like in this screenshot I took (click on it for full size):

As you can see the Gmail screen is under the Google Chromium “group” during the Alt+Tab. This is ANNOYING. This happens because Gnome 3 uses the WM_CLASS property from X to decide what is going to be considered the same Application, this is a change from the Gnome 2 behavior, when it considered Windows, not Applications, for grouping. Nice test to do (you have to click a window after running the command):

1
2

$ xprop WM_CLASS
WM_CLASS(STRING) = "chromium-browser", "Chromium-browser"

So, how to fix it? Well, it’s pretty very trivial. What must be done is to add the --class=value to your .desktop Exec line. Also, add StartupWMClass with the same value to give window managers and docks a tip of what application that belongs, this is my Gmail.Desktop:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

#!/usr/bin/env xdg-open

[Desktop Entry]
Version=1.0
Type=Application
Terminal=false
Icon[en_US]=/home/coredump/Pictures/Gmail-Icon.png
Name[en_US]=Gmail
Exec=chromium-browser --user-data-dir="/home/coredump/.config/gmail-chromium" --app="https://mail.google.com" --class=Gmail
Comment[en_US]=Gmail on Chromium
Name=Gmail
Comment=Gmail on Chromium
Icon=/home/coredump/Pictures/Gmail-Icon.png
StartupWMClass=Gmail
StartupNotify=true

Just remember to close all Chromium sessions (including the background one if you use it) before testing it since it will reuse the binary that is not using --class and you will lose minutes in rage (I did). After doing that with as many .desktop files you want, you will have the expected behavior on Alt+Tab:

Three distinct browser sessions, as it was supposed to be. This problem also affects some docks like Docky and even BAMF (the thing Unity uses to match windows).

cya!

1
2
3
4
5
6
7
8
9
10
11

exec    = require('child_process').exec

result =
    error: null
    stdout: ''
    stderr: ''
exec 'echo testing', (error, stdout, stderr) ->
    result['error'] = error
    result['stdout'] = stdout
    result['stderr'] = stderr
console.log "'#{result['error']}' '#{result['stderr']}' '#{result['stdout']}'"

This is the js result compiled by coffee:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

(function() {
  var exec, result;
  exec = require('child_process').exec;
  result = {
    error: null,
    stdout: '',
    stderr: ''
  };
  exec('echo testing', function(error, stdout, stderr) {
    result['error'] = error;
    result['stdout'] = stdout;
    return result['stderr'] = stderr;
  });
  console.log("'" + result['error'] + "' '" + result['stderr'] + "' '" + result['stdout'] + "'");
}).call(this);

It took me some minutes to understand that callback functions like the one that exec takes aren’t really executed synchronously, they are more like fire and forget, what really happens with this code is that the exec is called and the flow continues, arriving at console.log before the callback has actually finished and updated the values. I am sure that most of the js programmers are looking at me with their ‘noob’ faces but as I said, I am testing and learning as I go!

The code below illustrates it very well:

1
2
3
4
5
6
7
8
9
10
11
12

exec    = require('child_process').exec

result =
    error: null
    stdout: ''
    stderr: ''
exec 'sleep 10', (error, stdout, stderr) ->
    result['error'] = error
    result['stdout'] = stdout
    result['stderr'] = stderr
    console.log "'#{result['error']}' '#{result['stderr']}' '#{result['stdout']}'"
console.log "Out the function"

And this is the result of running this code. As you can see, the code reaches the Out of the function line and THEN finishes waiting for the callback to finish.

1
2
3
4

coredump@janie:~ $ coffee test.coffee
Out the function
'null' '' ''
coredump@janie:~ $

Functions like that are common specially on browser/javascript applications and force you to think differently about program flow and how exactly to solve that. If the console.log line is moved inside the callback function body, it works, but again it may end printing on a totally different time than what you intended. There are talks and a feature request for Node to support a synchronous version of exec and relatives, but if this is going to happen it will be on version 0.7 or later.

cya!

Me and Linux go way back to 1997, the first Slackware installs using too many disks, the long hours trying to get X to work on those blasted Trident cards.

It was a rough beginning, I usually say that in those times Linux didn’t had users, he had enthusiasts. Me and Linux didn’t hit very well in the first time too. It took a while until I started to really get into it, maybe a year or so watching co-workers battle kernel configurations and etc.

But, when we finally started, it was great. From Slackware I moved to SuSE, then finally Debian (and Ubuntu) where I am to this day, and I could not imagine myself doing the stuff I do with any other OS (yes *BSD I’m looking at you, someday we will talk again).

So, happy birthday Linux, and thanks to all kernel developers and people that make this OS so great.

cya

I suggest you to paste them one by one, and by you I mean the future me, that will try to paste them all at once and do something stupid.

1
2
3
4
5

openssl genrsa -des3 -out server.key 1024
openssl req -new -key server.key -out server.csr
cp server.key server.key.org
openssl rsa -in server.key.org -out server.key
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

cya.

Also, they were originally one liners but I changed it for the sake of readability.

1
2
3
4
5
6

for file in roles/*
do 
  role=$(basename $file)
  echo -n "Role ${role%.rb}: "
  knife search node "role:${role%.rb}" -i |grep -v '^$' |wc -l
done

You may not have a directory with roles, so you may need to use that:

1
2
3
4
5
6

for file in $(knife role list|egrep -v '\[|\]'|tr -d ' ,"')
do 
  role=$(basename $file)
  echo -n "Role ${role%.rb}: "
  knife search node "role:${role%.rb}" -i |grep -v '^$' |wc -l;
done

To search for those pesky empty run_lists (rare, but it happens, automatic cloud instaces, etc…)

1
2
3
4
5
6

for node in $(knife node list |egrep -v '\[|\]' |tr -d ' ",')
do 
  echo "Node $node:"
  knife node show -r $node
  echo "-----------"
done

Yes, in this last case you will still have to do some manual work and actually find the empty run lists. Search for empty lines, the only ones present on the file will be the empty run lists.

That’s it, cya!

Stay tunned for more posts on the days coming, I promise that I will try to keep this updated! :)

cya!