jelder'd

Tracking request queue time on New Relic RPM with Varnish

2010-07-16T17:48:00.007-04:00

The nice folks at New Relic have added an under-hyped feature to RPM which allows for the tracking of the time a given request spent in the server's work queue before processing began. This information in crucial in determining when you need to add more workers. It only requires that your front-end add an X-Request-Start header containing the epoch time in microseconds when the request was received. They offer a patch for NginX and a one-line config change for Apache.

But what about the new hotness, Varnish?

That's what. Save this to /etc/varnish/newrelic.h and include it in your vcl_recv declaration.

Serving static files from Varnish

2010-07-16T17:30:00.002-04:00

At LocaModa, most of our APIs are eventually consumed by ActionScript 3 applications run on large outdoor screens or in users' web browsers. Flash's cross-domain request protection incurres some round-trip latency, which essentially means wasted screen time, which is money, and I'd like to share one way we optimize for this.

We use the excellent Varnish HTTP accelerator in front of our application servers. True, we could easily be serving crossdomain.xml from Tomcat, or Apache, or NginX, but Varnish is the only part of the infrastructure I'm really happy with for the long-term. This is why I decided to serve a static file from Varnish which, officially, doesn't do this.

"Error 843? What the hell is that?" vcl_recv is intercepting requests for crossdomain.xml and returning an illegal, non-standard, stupid error. 843 is meaningful to those familiar with Flash socket connections as the TCP port number for socket policy requests. (I wrote a server for this a while ago.)

In vcl_error, we catch 843's and synthesize a response. The response has a far-future expiration and a 200 response code. These requests are served even faster than Varnish's already blindingly fast cache hits.

Fun with SSL

2009-08-25T15:59:00.016-04:00

I'm sure you love renewing SSL certificates almost as much as I do. Nothing beats the satisfaction of... slogging through a bunch of complicated commands you haven't typed in years just to maintain status quo. Right.

Having had an opportunity to take better notes on this process for my employer, I present to you a few brief commands to make renewing your SSL certificates almost painless.

First, check your expiration date.

openssl s_client -connect host01.example.com:443 < /dev/null 2> /dev/null |\
   sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' |\
   openssl x509 -enddate

Renewal time is an great opportunity to rotate your key. This is technically optional but it's good practice:
```
openssl genrsa -out example_com.key 1024
```
You'll need both the key and the about-to-expire certificate in the current directory for the next step. This will create a new CSR with all of the same parameters you used last time. Take this over to your preferred certificate authority and begin their renewal process.
```
openssl x509 \
    -x509toreq \
    -signkey example_com.key \
    -in example_com.crt
```

If you only need the key and certificate pair for your application (eg, Apache, NginX, slapd... most applications), you're done. However, if you need to use this certificate with Java servers as well, keep reading (wildcard certificates for the win). This procedure is recommended for ActiveMQ but should be directly applicable to Tomcat et al.

The Java keytool command doesn't offer any way to import an x.509 key by itself. The workaround is to first merge the x.509 certificate and key to a new PKCS12.
```
openssl pkcs12 \
    -export \
    -in example_com.crt \
    -inkey example_com.key \
    -out example_com.p12
```

Now use keytool to convert that PKCS12 file into a Java keystore file.

keytool \
    -importkeystore \
    -deststorepass changeme \
    -destkeypass changeme \
    -destkeystore keystore.jks \
    -srckeystore example_com.p12 \
    -srcstoretype PKCS12 \
    -srcstorepass changeme \
    -alias 1

That's it. Like I said, I highly recommend wildcard certificates if you can afford the initial expense. I hate having to justify the expense and effort buying a new certificate each time I bring up a new service. Keeping your certificates in a git repository also takes away some of the stress here: no fears of screwing something up and deleting a file that cost you hundreds of dollars to generate.

Getting busy with INFORMATION_SCHEMA

2009-04-08T15:13:00.004-04:00

I've been wanting to use this trick forever. Here's how to quickly, effortlessly, and I should add, destructively, convert all tables in a given database to UTF8.

mysql --batch information_schema -e "select concat('ALTER TABLE ', TABLE_NAME,' ONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;') as '--' from TABLES where TABLE_SCHEMA='somedb' and TABLE_COLLATION not like '%utf8%'" | mysql somedb

The addition of INFORMATION_SCHEMA combined with CONCAT makes it possible to do arbitrarily cool things with MySQL. Love it.

In all your travels...

2009-03-01T22:37:00.003-05:00

I can't tell you how much time I've spent fantasizing about this concept. Battlestar Galactica presents us with a great way to discuss it.

This is the closest I've ever seen teledramas come to doing real, honest, hard science fiction. So very well done, and such a tragedy that there are only three episodes remaining for this singular series.

Mushrooms tell us "the internet was inevitable" and something about fractal supertrees

2009-01-20T23:56:00.002-05:00

Imogen Heap is proof that the future worked

2009-01-20T23:30:00.006-05:00

I was promised the future. By that I mean, yes, it went almost without question (statistically) that I would still be alive in 2009. The future was something more; something with a recognizable taste. As a child I knew without question that as I grew up, even more so could the world I may inherit. I mean something more than just years when I say the future.

Superman, The Jetsons, James Kirk and a black and white TV told me things, beautiful things. Adventure and possibility beyond imagination. Perfect worldwide peace without disease or poverty. Ubiquitous, effortless flight. Probably, a black president with a stage presence like Morgan Freeman. Above all, I expected to see the limitless reach of human exploration and expression. I want to talk about that last point for a second.

Imogen Heap confirms for me that the future has not only already arrived, but has delivered fully on one promise. Here we see a young woman controlling a fantastic array of electronic equipment. Any single component, if presented to NASA's best engineers in 1969, would have seemed impossibly sophisticated. She, alone on stage, uses this future artifact not to explore space or design pharmaceuticals, but to project without restraint her imagination onto our consciousness.

That's the highest quality recording I can find on YouTube. I strongly suggest you go support this artist in any way you can, and with that being said, here's my favorite rendition of Hide and Seek.

Female Vocalists on YouTube

2009-01-20T21:56:00.010-05:00

I've had a post on female vocalists brewing for a while and this isn't it. However, you really should take a listen to Nataly Dawn. She has that timeless, ethereal sort of sound which I find mesmerizing, and I won't lie; when she makes eye contact with the camera I fall apart a little.

On a somewhat related note, seeing Barack Obama's face on the new WhiteHouse.gov is like watching boobs bounce in slow motion.

Huygens' Titan descent telemetry, visualized

2009-01-18T16:30:00.007-05:00

This movie, built with data collected during the European Space Agency's Huygens probe on Jan. 14, 2005, shows the operation of the Descent Imager/Spectral Radiometer camera during its descent and after touchdown. The camera was funded by NASA.

The article doesn't do justice to how much beauty exists in this visualization of data. The brief and glorious life of the Huygens lander, dropped from the Cassini probe into the atmosphere of Titan, is compressed from 4 hours to just a few minutes. A unique chime as if from a child's xylophone accompanies each sensor reading. As Huygens' last meters close, the final intrusion of poetry into science is utterly disarming.

If anyone can find a higher resolution form of this video it would be greatly appreciated.

Link

Life imitating art: WALL•M

2009-01-18T16:00:00.004-05:00

Meanwhile, at the Carl Sagan Memorial Station (née Pathfinder) on Mars:

Unlike Spirit and Opportunity, Sojourner could not directly communicate with Earth and had to rely on the lander to transmit messages about its health and travels. So when contact with the lander, which was designed to last one month, was lost after three months, ground controllers were not sure what became of Sojourner.
Now, it seems the rover kept moving, apparently trying to reach its companion. "The rover was programmed so that if it didn't get commands from the lander, it would assume it somehow got out of radio contact behind a ridge or rock," Parker told New Scientist. "So it would drive to the lander as best it could and keep trying to re-establish contact", circling it as it got close, he says.
When it was last seen, the rover was 13 metres away from the Pathfinder lander. Now, it appears to be about 6 metres away, according to the new MRO images. "I think the simplest explanation is it started to drive back, got about half way, and stopped for whatever reason - it may have thought it got there," says Parker. "The other possibility is it drove around and around for who knows how long and simply failed at that location."

Emphasis mine. Full article at New Scientist.

Why art abhors censorship

2009-01-14T20:34:00.004-05:00

Perhaps I'm being generous here, but just watch. It gets really great around 1:10 in.

It's not just that they say "fuck." They say nothing except fuck for over 3 three minutes, and it's still a very compelling and expressive bit of dialog.

Shackleton Crater and the "shit knife"

2009-01-10T23:03:00.000-05:00

Here's something for the before-you-die list

2009-01-09T23:36:00.001-05:00

Of course, it would be wise to put this at the very end of the list.
wingsuit base jumping from Ali on Vimeo.

Unofficial Git & GitNub for MacOS Leopard

2009-01-07T13:36:00.004-05:00

I have made available a completely unofficial but functional MacOS Leopard build of git and the excellent GitNub viewer. To install, pretend I'm a trustworthy individual and run:

curl http://jelder.s3.amazonaws.com/git-1.6.1-leopard.tar.gz | sudo tar xvzf - -C /

This will put git in /usr/local/bin/git (with other support files in /usr/local/libexec and /usr/local/share). The nub script in /usr/local/bin/nub will launch GitNub from the current directory.

Mint needs a better workflow

2009-01-03T10:30:00.002-05:00

Just posted this to the Mint.com user forums.

Most of my Mint's value for us users comes from knowing how our money is getting spent, but one critical part of that use case is missing: a workflow for rapid categorization.
Mint needs a really slick way to let me sit down once a month and quickly categorize all of my uncatagorized transactions. I should be presented with a queue of transactions from unknown merchants and some method for applying labels, splitting bills, etc. Ideally, I should be able to keep my hands on the keyboard for this, and it should take like 5 minutes a month or less.
This can be done manually now, of course, but it's far from efficient. The ajax stuff in the transactions tab has improved but it's still too chunky and slow on a modern MacBook Pro and Firefox 3.0. The experience should be less hunt-pick-wait and more next-next-next.
With more complete vendor preference profiles for each user, Mint should be able to monetize via more focused opportunities. Good luck guys! Mint is great but it's not complete!

The thread is here.

Git is Exquisite

2008-12-29T17:44:00.006-05:00

Possibly the best thing about git is that you can start from an existing directory. This makes it amazing for tracking configuration files.

cd /etc/ldap ; git init ; git add . ; git commit -m "got some config religion"

Too bad one can't use GitHub for this, but remote backups and batch config changes get really elegant with kind of setup.

flashpolicyd; Making myself use Git

2008-12-27T14:40:00.007-05:00

People who work with me know I treat version control like a sacrament. "If it's not in Subversion, it never happened." My desktop is a Subversion checkout. It has tags.

But Subversion, despite having a superior name, is losing favor to Git. I really like some of Git's philosophical decisions, so I'm diving in. I have one open source side project these days (flashpolicyd) and I have just migrated it to GitHub.

http://github.com/jelder/flashpolicyd/

Here we go...

Passwords are so 1997

2008-12-27T13:16:00.010-05:00

It's practically 2009 and too many of us still use a jumble of letters and numbers as passwords. A New Year's resolution to keep: here's an ultra-short tutorial on never typing your password into a remote host again. First, the MacOS client version.

Create a key and encrypt the private side with a strong passphrase: ssh-keygen -t dsa
Add it to your Keychain: ssh-add -K .ssh/id_dsa
Establish trust: cat .ssh/id_dsa.pub | ssh some.remote.net tee -a .ssh/authorized_keys (or email that file to someone who can do this for you)
Now go disable passwords for SSH on that remote server by adding PasswordAuthentication no to /etc/sshd_config and restarting the SSH server. You are now immune to SSH brute force attacks.

Repeat steps 3 and 4 for every remote machine you use, but make sure you have a secure backup of that private key!

I practically never use Linux from the console any more, so I won't contribute a tutorial for that today. PuTTY and Pagent make this pretty damn easy on Windows. No more excuses!

That solves the shell session password issue. For the rest, the solution isn't so neat and tidy. sudo apt-get install pwgen to create super strong (but somewhat memorable) passwords for local accounts and websites. Turn on FileVault or FireFox's master password and let software deal with authentication for you.

"Fun" with Flash

2008-11-29T14:18:00.004-05:00

Here's a high performance Flash policy server I wrote over the long weekend. Just something to stay sharp between the ample meals. I was able to sustain 33,000 requests per minute on my MacBook Pro from localhost with minimal load. Tested on and functional but not quite as fast on OpenBSD (not sure why yet). My Linux test box gave me well over 145,000 requests per minute. This is probably several orders of magnitude faster than anyone needs, but here ya go. GPL, of course.

Update: I just posted a new version which will drop privileges after opening 843.

Rainy Sunday, iPhone, Making Ringtones

2008-11-16T16:55:00.006-05:00

I've gotten some compliments on my Mad World custom iPhone ringtone. It always puts me in a calm mood before dealing with whomever is calling, so I thought I'd share it and a few others I made.

Mad World
Amok Time (All unironic use is prohibited)
MGMT - Kids
MGMT - Time to Pretend

Update: The proper name for that track is probably "Amok Time," nerds.

Miscellaneous discoveries

2008-05-06T10:58:00.006-04:00

Two problems which have been bothering me forever have finally been solved. One (and this should have been obvious) is revealing the total system memory on BSD-like systems from the command line. There is no equivalent of Linux's free -m. The closest approximation is sysctl hw.physmem. This lead to a revelation that one of my OpenBSD firewalls had been provisioned with several gigabytes more than it would ever, ever need. Yay!

The second was, how the heck do I get Dell service tag of an ill server without actually going to the datacenter? sudo dmidecode -s system-serial-number

Yay again. Definitely time to get myself some outside time.

Update: Need the service tag off your OpenBSD box? sysctl hw.serialno

Take Back Your Keyboard!

2008-04-27T20:11:00.010-04:00

A good friend of mine has gone to great lengths in refining the pathway between his brain and his computer. That is to say, he's serious about keyboard design.

I'm not quite ready to invest in an ergonomic (and scary looking) keyboard-as-cereal-bowl, or the ultimate extreme, hacking up my own Dvorak variant.What I am willing to do, however, is repurpose my utterly useless Caps Lock.

Thanks to inspiration from Bill in 2006 and lots of experiments since, I am now able to publish my findings on the subject. VIM users' most frequently used key is probably Escape. For me that means moving my ring finger about three inches diagonally away from the home row, maybe a few hundred times an hour. Plus, the top row is frequently smaller than the other rows. It's actually designed to be hard to hit without effort. Caps Lock, meanwhile, is both enormous and well within pinky range on the home row.

For Windows, it's a registry hack which I've written about before. Create a file called CapsLockEscape.reg and paste this text into it, save it, and double-click. You'll have to log out and log back in (or reboot) for it to take affect.

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Keyboard Layout]
"Scancode Map"=hex:00,00,00,00,00,00,00,00,02,00,00,00,01,00,3a,00,00,00,00,00

For the Debian and Ubuntu consoles, add this line to /etc/console-tools/remap, then either run /etc/init.d/console-screen.sh or reboot.

s/keycode  58 = Caps_Lock/keycode  58 = Escape/;

For X11, create a file called ~/.capslockescape with the following contents, then run xmodmap ~/.capslockescape to try it out. Add that command to your ~/.xinitrc to make it permanent.

remove Lock = Caps_Lock
keysym Escape = Caps_Lock
keysym Caps_Lock = Escape
add Lock = Caps_Lock

The hard part was Mac OS X Leopard. I went so far as to open a RentACoder.com auction for someone willing to write the appropriate keyboard driver. I opened the bidding at $100, and offered it to everyone who had written keyboard macro software for the Mac. One Takayama Fumihiko declined the offer but linked me to something he had already written, PCKeyboardHack. The magic number is 53.

Ubuntu Site-to-Site SSH VPN

2008-04-27T05:50:00.015-04:00

Weekend Project time. Here's how to build a VPN using just SSH and two Ubuntu machines. At least one of these machines should have a static IP or DNS, and they should both be the default gateway for their respective sites.

MachineA is the originator and has a local network on NetworkA/24. MachineB has a static name on the Internet and a local network called NetworkB/24. Our point-to-point network is 10.0.0.0/24, with MachineA at 10.0.0.0.1 and MachineB a 10.0.0.2. I will assume for brevity that you have already established an SSH trust relationship between root@MachineA and root@MachineB.

MachineA:/etc/interfaces:

auto tun0
iface tun0 inet static
      pre-up ssh -f -w 0:0 MachineB "ifdown tun0 ; ifup tun0"
      pre-up sleep 5
      address 10.0.0.2
      pointopoint 10.0.0.1
      netmask 255.255.255.0
      up ip route add NetworkB/24 via 10.0.0.1
      down ip route del NetworkB/24 via 10.0.0.1

MachineB:/etc/network/interfaces:

iface tun0 inet static
      address 10.0.0.1
      pointopoint 10.0.0.2
      netmask 255.255.255.0
      up ip route add NetworkA/24 via 10.0.0.2
      down ip route del NetworkA/24 via 10.0.0.2

(Note that MachineA's interface is set to auto, but MachineB's is not.)

Since these are the gateways, we're probably running a netfilter wrapper such as Shorewall. In that case, we'll have a few more things to add. They'll be the same on both ends.

In /etc/shorewall/zones, add: vpn ipv4
In /etc/shorewall/interfaces, add: vpn tun0 detect
In /etc/shorewall/rules, add:
ACCEPT vpn loc all ACCEPT loc vpn all
And finally, in /etc/modules, add tun

After restarting Shorewal and loading the tun device (modprobe tun && invoke-rc.d shorewall restart), you should now be able to run ifup tun0 on MachineA. Ta-da! You can now connect to your remote networks as if they were a single hop away.

This solution shares some problems with all TCP-only VPNs. Each relayed packet has a reduced payload size, so you'll see more fragmentation. There are also problems related to running Nagle on top of Nagle. Depending on your expected traffic, you might have some luck enabling compression. See ssh_config(5). There are also some problems more or less unique to SSH VPNs. If the MachineB's host key changes, MachineA won't be able to establish a connection until the old key is removed from MachineA:/root/.ssh/known_keys. This also makes failover scenarios pretty ugly. Good luck!

A quine is a quine, right?

2008-04-26T23:18:00.004-04:00

A quine is a computer program which produces its own source code when executed. My brother is learning Python and we've been talking about quines a lot lately. I have a feeling that I'm cheating here, but I present for posterity my own crazy short Perl quine.

seek DATA,0,0;print for<DATA>__END__

Street Art

2008-04-26T23:10:00.006-04:00

I really like this:

But this one is a WTF moment. Metrosexual Movers?

Both found on Sidney Street in Cambridge (Central Square).