wildpointer*

a little abstraction goes a long way (fun with SOAP APIs)

2011-06-19T07:27:40Z

I’ve been working with SOAP services a lot lately. Is that moaning I hear? It is! But you know, as long as I’m provided an API that works, and isn’t *too* slow, I’ll deal! However, it’s nice everyone finally realized XML.. well.. isn’t the best for stuff like this, and are moving away from it. Took them long enough.

This should be a relatively quick post, but I think the point it makes is nice. My colleague Paul once told me my code looks like a 5 year old wrote it. I try to keep things simple, and make things simple. The goal is to take something that could have been complex (or sounds complex), and break it down into little logical chunks you can make sense of the first time through. I’m also a 5 year old, so that might have something to do with it.

In this case, I was working on interacting with a SOAP API for a load balancing product. Their API is actually really nice, but with the amount of settings you can change for certain objects, their free, open code samples definitely wouldn’t fly for an enterprise-class tool. Here it is:

Original Code

 
my $result = $soap->addservice(
    SOAP::Data->name( 'name' => 'test-svc'),
    SOAP::Data->name( 'IP' => '1.2.3.4'),
    SOAP::Data->name( 'servicetype' => 'http'),
    SOAP::Data->name( 'port' => '80'),
    SOAP::Data->name( 'maxreq' => '8'),
)->result();

If this is all you need to do, then it definitely works. But, what happens if you need to set more than 5 values? Maybe you have 10 you need to set? Maybe the number of arguments is going to be dynamic every time you run it (ie: you don’t know if maxreq is going to be set EVERY time). Well, that could be a pain in the bootay.

There are a couple things we’re going to end up doing here, but first thing first: this whole multi-argument thing isn’t going to fly if I want to make things dynamic (ie: loop through a data set to build your input key/val pairs). I also reeeaaally don’t want to have to say SOAP::Data->name for every key/val pair I set. I’d rather just be able to define my arguments as a hash (or dictionary, or whatever your $langauge calls it), and let something take care of the dirty work for me. Simple!

Since our SOAP call takes an array of SOAP::Data objects, all we really need to do is make something that takes a hash, converts it into an array of SOAP::Data objects, then pass to our SOAP call. We’re going to use an arrayref to make things easier. I think pointers are fun, so I tend to just use ref’s for everything. I hope that’s okay.

 
sub hash_to_soap_data {
 
    my($hashref)    = @_;
    my $soap_data = [];
 
    foreach my $key(sort keys %{$hashref}) {
        push(@{$soap_data},SOAP::Data->name( $key => $hashref->{$key} );
    }
 
    return $soap_data;
 
}

Ta da! That wasn’t so bad, was it? Now, we can do something like this:

 
# we could also just define the hash
# as the argument to hash_to_soap_data
# if you wanted to smash it together even
# more.
 
my $hashref = (
    name => "test-svc",
    servicetype => "http",
    port => 80,
    ipaddress => "1.2.3.4",
);
 
my $soap_data = hash_to_soap_data($hashref);
my $result = $soap->addservice(@{$soap_data});

Looking better! But you know, it’s still a little bit too annoying. I don’t want to have to define a hash, pass it to a function X, then take the result of function X and pass it to function Y. The ideal code would just to be to make an API call by passing a hash to a function, and let the underlying code handle making things happen. In short, we’re going to abstract things.

This is where we decide how we want to actually interface with our function. In our case, I think something like this is nice:

my $result = call_api($func_name,$hashref);

With that example, we can just make our API call in one easy line by passing a hashref. Yes, we’ll have a function call_api() to deal with, but that’s part of the abstraction.

Ultimately, this will eventually allow us to easily change the inner-workings of call_api() later on without ever having to change the rest of our code. For example, maybe the API changes from SOAP to a RESTful style, maybe using JSON. By abstracting, you can re-write call_api() to just make RESTful calls and return JSON results.

In any case, let’s write it:

 
sub call_api {
 
    # note: $soap is our SOAP or SOAP::Lite object
    # that we have already defined elsewhere.
 
    my($func,$hashref) = @_;
    my @soap_data;
 
    foreach my $key(sort keys %{$hashref}) {
        push( @soap_data, SOAP::Data->name( $key => $hashref->{$key} );
    }
 
    my $result = $soap->$func(@soap_data);
    return $result;
 
}

With that complete, now we just implement:

Abstracted Code

 
my $result = call_api( "addservice" , {
    name => "test-svc",
    ip => "1.2.3.4",
    servicetype => "http",
    port => 80,
    maxreq => 8,    
});

Your first thought probably is: it looks almost exactly like our original code! What’s the point? You suck? The point is you have now abstracted things just enough to make things a lot easier to manage in the long run. Based on inputs, you can dynamically assign the function (ie: maybe it’s addservice to start, but based on a conditional, it may change to setservice, and all you need to change is $func_name. Your single API line still works as expected, so no need for multiple API lines. Same with your data hash. You can build your input key/val pairs manually, dynamically based on IO, other conditionals, etc, and never have to change that one friendly line:

my $result = call_api($func_name,$hashref);

It also helps with troubleshooting, because you know that all API calls ultimately funnel in to call_api(). And, like I mentioned before, if the server-side ever changes from SOAP to whatever the latest and greatest is, you should have hopefully abstracted things enough to just be able to change your single function without affecting everything else.

In practice, I’ve found this to work pretty well. There will be one-offs now and then, and you’ll have to account for them. However, over time, you’ll start to be able to identify places for abstraction beforehand, which in the long run will just help you to make pretty sweet abstractions for doing anything. This doesn’t just apply to Perl, and it doesn’t just apply to SOAP. I try to apply it to as many places I can. It tends to work best in an object-oriented codebase (yes, even Perl objects), but even abstraction via functions can be good too!

Someone joked once about “hello world” for a new programmer is 1 line, while it’s 250 lines with 5 layers of abstraction and an additional 20 comments for an experienced programmer. While it’s not far off, there’s definitely a sane balance somewhere in the middle, and that’s part of the journey Enjoy!

perl typecast beware! weak typing and the range operator.

2010-11-24T19:29:26Z

Background: I’ve been working on a suite of tools for the past few months to help with some config automation, and one of the config files allows you to put in a range (ie: 1-100, or even 0200-0399). The tool will take the range, look for running jobs with an ID in that range, then do something with it. In some cases, we do have jobs whose ID starts with a zero (or multiple zeroes), so the leading zero can be important. This is where I ran into some interesting discoveries!

Okay, so, to aid in making it easy to iterate over the range, I was using the range operator in Perl, which is the little “..” thing you can do, (or dotdot, or dot-dot, or whatever you might call it). For example:

for(1 .. 100) { 
    printf("%s\n",$_);
}

.. or, being a little more realistic:

my $start = 1;
my $end   = 100;
 
for my $i( $start .. $end ) {
    printf("%s\n",$i);
}

Run those, and you’ll get all the numbers from 1 to 100. Fantastic. Want to get even more crazy? How about letters. Yeah, it does those too. Sweet.

In this example, we’ll get everything from a to z:

my $start = "a";
my $end   = "z";
 
for my $i( $start .. $end ) {
    printf("%s\n",$i);
}

Hrmm, what about if we want the range from a to zzzzz? Yup, it does that too:

my $start = "a";
my $end   = "zzzzz";
 
for my $i( $start .. $end ) {
    printf("%s\n",$i);
}

Of course, I’m not giving you the output, because that would be insane.. but it works. Nice!

There’s actually a bunch more you can do with it, too. There’s even a “…” (dot-dot-dot) that you can do something with (but I forget what). However, it is all documented here:

http://perldoc.perl.org/perlop.html#Range-Operators.

But that’s not what this post is about. It’s about something interesting I ran into. Still interested? Good. Keep reading.

So anyway, in the case of my config parser, I would parse out ranges as tokens (ie: 0200-0300), split, and do a search across the range. For example:

# $range = "0200-0300"
my($start,$end) = $range =~ /^(\d+)\-(\d+)$/;
 
for my $id( $start .. $end ) {
   if(found_job_id($id)) {
      do_something_with($id);
   }
}

This basically allows us to lazily put large ranges into the config, but requires us to be more specific when creating jobs with new IDs (to ensure it falls within our range). In any case, this works exactly as we’d expect, leading zeroes and all.

Now, as part of error checking, I decided I should make sure that the $start value was less than the $end value, since the range operator will not work if $start is greater than $end. It will just return an empty set. This is documented and not an error, so I wanted to throw an error.

To do so, I just modified my code to implement a simple comparison check:

# $range = "0200-0300"
my($start,$end) = $range =~ /^(\d+)\-(\d+)$/;
 
if($start > $end) {
   throw_error_and_exit("Start is greater than end! Fix your range, buddy.");
}
 
for my $id( $start .. $end ) {
   if(found_job_id($id)) {
      do_something_with($id);
   }
}

The change seemed harmless enough, but when I ran the config parser again to make sure things worked, I got nothing. Literally. Nothing. It appeared as if everything broke. Weird. No errors, the range is valid (I was using 0100-0200), but, um, what?

After scratching my head for about 20 seconds, I went, “aha!” Yes, just like that.

Sure enough, when I parse out my range from the config, the values I get back are strings (which makes sense), so in the case of 0200-0300, the range operator is actually performing the operation based on the string value of each, using what they consider the “magical auto-increment algorithm” (per the Perl documentation). I don’t know what type of analysis it does, but it’s probably just based on ASCII values.

In any case, AFTER I added my new comparison operation (to check if $start was greater than $end), Perl had to typecast my variables (which were strings) to integers to compare them, therefore dropping the leading zeroes! Here’s something a little more graphic:

# $range = "0200-0300"
my($start,$end) = $range =~ /^(\d+)\-(\d+)$/; # <--- STRINGS
 
if($start > $end) { # <--- OMG NOW THEY'RE INTEGERS
   throw_error_and_exit("Start is greater than end! Fix that.");
}
 
for my $id( $start .. $end ) { # <--- HEY WHERE'D THE LEADING ZERO GO?
   if(found_job_id($id)) {
      do_something_with($id);
   }
}

This is one of those things with Perl where a mix of dynamic and weak typing features lead to strings magically (and uncontrollably) being casted to integers where needed. In most cases, having a language with both dynamic and weak typing on variables can be nice, but it can also lead to laziness and unexpected results if you’re not careful.

Well, that’s all for today. Just thought I’d share that one.

offload Mac OS X swap with dynamic_pager and an SDXC card

2010-11-24T05:24:40Z

I have an idea.

So, I’ve run into some issues with OS X where it will start swapping, even though there is more than enough physical RAM available. I’m assuming this has to do with the OS deciding certain processes have been idle for long enough; it will pre-emptively swap them out with the assumption of a need for new, soon-to-be “things” requiring memory. This would give the user the perception that their machine loads up things faster when said “things” need to be loaded. That could be total BS on my part, but until I have a full technical explanation of it, that’s the story I’m going to stick to.

Now, I’m used to Linux, where you can set /proc/sys/vm/swappiness=0 and effectively stop swapping altogether. As long as there’s enough physical memory available, it will keep using it. If you run out, then OOM kill (assuming you haven’t disabled it via /proc/sys/vm/oom-kill) will come and start playing whack-a-mole with your processes. Sweet. This is good.

Since OS X doesn’t have this, what *does* it have? Well, it has a process called dynamic_pager, which oversees the management of swap (and swap files). There are two files we’re interested in here:
/sbin/dynamic_pager /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist
The binary is obvious, but the LaunchDaemon plist is what we’re really interested in here. This is what tells dynamic_pager to kick off at launch. Now, the plist is in binary format, so to really get a look (from Terminal, unless you use the plist Editor), you can convert it to XML:
plutil -convert xml1 com.apple.dynamic_pager.plist
From there, the contents of the file will be something like this:
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>EnableTransactions</key> <true/> <key>HopefullyExitsLast</key> <true/> <key>Label</key> <string>com.apple.dynamic_pager</string> <key>OnDemand</key> <false/> <key>ProgramArguments</key> <array> <string>/sbin/dynamic_pager</string> <string>-F</string> <string>/private/var/vm/swapfile</string> </array> </dict> </plist>
Fantastic! By the way, to convert back to binary when you’re done, you can do this:
plutil -convert binary com.apple.dynamic_pager.plist
Now that we’re looking at our plist, you’ll notice the config setting for -F /private/var/vm/swapfile. This is the base name used by dynamic_pager for when it needs to create a new swapfile. If you do a ps, you’ll notice it running:
[chet@cilantro ~]$ ps ax | grep dynamic_pager 39 ?? Ss 0:00.02 /sbin/dynamic_pager -F /private/var/vm/swapfile
And if you check out /private/var/vm:
[chet@cilantro ~]$ ls -lh /private/var/vm/ -rw------- 1 root wheel 64M Jul 19 22:02 swapfile0 -rw------- 1 root wheel 64M Jul 23 21:38 swapfile1 -rw------- 1 root wheel 128M Jul 23 21:38 swapfile2
As you can see, dynamic_pager is alive and well, managing a set of swap files on my local disk. Side note about the listing above: as the need for swap increases (or decreases) with activity, dynamic_pager will take the liberty of dynamically changing the swapfile size for you. You can also specify the -S flag in your plist to specify a static file size.

In any case, here’s where the fun starts: I don’t know about you, but the disk in my MacBook Pro isn’t anything super-amazing. Maybe it’s on a 3Gb/sec SATA channel, but it’s a 5400rpm drive, and random writes usually yield somewhere around 12MB/sec. Sequential writes will hit about 60MB/sec at times, but lets be honest, random is what we should be caring about; they’re nothing spectacular.

So, when I’m in the middle of working on something, have a few Flash widgets running, doing who knows what, and I start to experience the lovely effects of swapping, it’s not always the best experience. On top of slowing things down, it’s also adding a lot of extra physical load to the disk. Poopy. I wish there was a way to decouple swap from my physical root disk, and possibly put it somewhere that might help out swap performance a bit. Oh wait, I have a thought!

Thanks to the designers at Apple, who graciously added an SD card slot to the latest line of MacBook Pro’s, we now have an option for swap: offload it to an SD card! Now, a plain old SD card might not be the best (even a class 10 would only give you 10MB/sec max, and under real world conditions it’s obviously going to be worse.. about 2MB/sec), but with the latest SDXC technology, Kingston, Toshiba (and others) have developed cards hitting between 50-60MB/sec on writes, which is very similar to the sequential writes on our local disk. Again, under real-world, random usage, it’s going to be less, but should be relatively similar to the random writes of our local disk. To try it out, just slide in your SDXC card, and update your plist to something like the following:
<string>/sbin/dynamic_pager</string> <string>-F</string> <string>/Volumes/sdxc_swap/swapfile</string>
After that, restart your dynamic_pager process, and you should be good to go, enjoying an offloaded swap, and totally burn out your SDXC card

That being said, I’d like to mention a couple of things. First, I don’t have a high performance SDXC card of my own for testing, so performance mileage may vary depending on the card. I know Toshiba is targeting their SDXC cards for use in Netbooks, which is great; it gives us an glimmer of hope that their cards are built for some relatively heavy duty lifting- more than your standard camera SD card; maybe you won’t burn it out as quickly after all!

Another thing to point out is that the high performance cards we’re discussing are all relatively larger.. 32GB or greater in size. Chances are you’ll [hopefully] never hit this in swap. However, dynamic_pager will also take care of dumping out a sleepimage file to the swap space, which will be up to however many GB of RAM you may have; take that into consideration when picking a card. You’ll probably still have leftover space, but just a word of advice.

That being said, putting your sleepimage on an SDXC card is actually nice in the sense of security. Put your machine to sleep, pop your SDXC card out, and you basically have locked your laptop down in terms of a thief being able to try and yank whatever you were working on. There are threads out there about people dumping sleep/hibernate images and stealing sensitive data, so this is a nice little side effect of offloading swap to a little card.

In any case, that’s all for now. It was a thought I had the other day, so I figured I would share the idea with others. Maybe some people will think it sucks, but I think it’s kind of fun. If you have a card and would like to dump back some performance data to me, that would be awesome (ie: test via Xbench or the like). And, if I ever get my own card worth testing, I’ll post an update on how it went! If you’d also like to tell me how dumb I am, that’s okay too; I’m used to it. Enjoy!

writing a custom Netscaler health check for NTP

2010-04-11T09:13:23Z

There are lots of load balancing products on the market today. Some are good, some are bad. Opinions aside, one such product is the Citrix Netscaler.

Now, the Netscaler has built in health checking for many different types of backend service protocols: HTTP, FTP, DNS, and more. However, one protocol not included is NTP. If you happen to be load balancing a farm of NTP servers, and want to ensure you’re not routing to a hosed NTP daemon (even if it’s ping-able), it would be nice to be able to health check the service [and take it out of rotation if it's not responding].

Luckily, the Netscaler provides (with documentation) a very flexible way to write your own custom health checks. Using the system, you can test whatever you want, in any way you want, with just a little Perl. Even better, the custom health check subsystem provides a few useful things to simplify the whole process, including:

passing in the service IP, and port, as arguments to your script
passing in a custom argument string (key value pairs, user/pass info, whatever)
handling the timeout for you from a higher level; if your check doesn’t respond within -respTimeout seconds, it will be considered a failure.

Now, for just the basics, this monitor will actually be pretty easy to put together. We are given the IP/port to send an NTP request to, and we don’t need to worry about a timeout, because the underlying subsystem will handle that for us. All we really need to do is send a request and [hope] for a response!

That being said, we’ll need to do a couple things to get to that point. These things are:

Load the health check subsystem module.
Load the Socket module, and create a socket to use to send the request [using the parameters passed in as args].

From that point, we can build and send our request. Since we don’t need to worry about handling a timeout, we can just sit there waiting for a response. First up:

use IO::Socket;
use Netscaler::KAS;

We’ll need both of these modules – one will tap into the health check subsystem, and the other is so we can send our UDP-based request.

Next is our function, which we’ll just call ntp_probe. It assigns a few variables from $_[0] (the service IP) and $_[1] (the service port) as arguments, which get passed in from work within probe().

So, the magic-fu to this is that, later on, we will pass our ntp_probe function as a coderef to the KAS probe() command. That will ultimately result in our function being called with the arguments passed in; it is similar to this:

$custom_function->($host,$port,$args);

If you look at the KAS code (and since it’s perl, you can), you can see the exact line; this just gives you an idea of what’s going on.

In any case, after that, we’ll create a new UDP socket, send the NTP request message down it, and either:

return 1 if something wasn’t defined (ip, port, or sock) – failure!
return 0 if we got a response – success!

Again, we don’t need to worry about timeouts, since the subsystem will handle that for us. So, here’s the function in it’s entirety:

sub ntp_probe {

    my $host    = $_[0];
    my $port    = $_[1];
    my $req     = "\010"."\0"x47;

    if(!$host || !$port) {
        return(1,"Host or port not specified.");
    }

    my $sock = IO::Socket::INET->new(
        Proto     => "udp",
        PeerAddr  => $host,
        PeerPort  => $port,
    );

   if($sock) {
        $sock->send($req);
        $sock->recv($_,1);
        return 0;
    }

    return(1,"Could not create socket");

}

Not bad, was it? Now that we have loaded our modules, and defined a function, the next step is to call the probe() method, with a coderef to our new function, to tell KAS what function to use for probing:

probe(\&ntp_probe);

And you’re off to the races! Just don’t forget to start your script with #!/usr/bin/perl, and it should be good to go

Now, you might be asking “how do I actually configure the Netscaler to use this?” That’s also just as easy. Let’s pretend our new script is called customNTP.pl. Just plop it into the /nsconfig/monitors/ directory, and do something like this from the CLI:

add lb monitor "custom-ntp-mon" USER -scriptName customNTP.pl
   -scriptArgs 1 -dispatcherIP 127.0.0.1 -dispatcherPort 3013
   -interval 20 -respTimeout 3

With that line, you will now have a working NTP healthcheck (20 second intervals, 3 second timeout). Just bind it to your service(s) that need it and enjoy!. Note that the type of monitor is USER; this is the type used for custom health checks.

Well, that’s all for this evening. There are bits and pieces of info like this around the web, on the Citrix AppExpert site, etc. And, even though it wasn’t too difficult to figure out, hopefully this will help someone out some day.

Take care!

apache mod_status: the misleading ReqPerSec metric

2010-04-03T20:07:40Z

To be honest, I had never thought twice about it. At my past employer, we had implemented tracking request rate for the Apache hosts by parsing out Total Accesses (ie: Total Requests) from server-status (thanks to our friendly module mod_status), publish that every minute, and let our metrics collection subsystem do a delta between the current and previous to determine a request rate to publish.

At my new employer, I put something together where the collection subsystem won’t refer to the previous total for deltas: it just takes whatever you give it and relays that to the metrics backend. Most applications provide you with a request/sec metric these days, so I figured I’d just use that. Instead of doing deltas on TotalAccesses, I started just relaying ReqPerSec instead. Simple enough, right? For those of you not familiar, here’s an example of server-status output:

Total Accesses: 400
Total kBytes: 800
CPULoad: 5
Uptime: 12345
ReqPerSec: 20
BytesPerSec: 20000
BytesPerReq: 1024
BusyWorkers: 15
IdleWorkers: 5

Well, amidst my metrics collection testing for the new system, I stumbled upon an interesting detail: I would be nailing the Apache instance with a flood of requests using my friend ab (aka: apachebench). However, I immediately noticed ReqPerSec wasn’t accurate at all. For example, I would be averaging a test of 300-400 req/s, but ReqPerSec would be displaying 21 req/s. It was as if I wasn’t sending it requests.

That being said, the TotalAccesses metric was reflecting the requests I was sending, so I knew the web server was receiving them. Interesting.

It got me thinking: “I wonder how Apache determines ReqPerSec?” Time for some source code action! Sure enough, here’s what I found:

ap_rprintf(r, "ReqPerSec: %g\n",
    (float) count / (float) up_time);

Interesting! All Apache does to calculate ReqPerSec is take the total number of requests throughout the life of the parent (count) and divide that by the total uptime in seconds (up_time). So, really, ReqPerSec is just an *average* requests per second over the parent’s lifetime. That’s NOT what we want. Maybe it’s useful in some scenarios, but when you need a live metric to let you know what your server is ACTUALLY doing, this isn’t the one.

If you want an accurate request rate, your best bet is going to be to collect Total Accesses and do deltas off of the previous collection. I’m glad that’s how we did it before!

To a point, this goes into the expense of metrics. For Apache, Total Accesses is just a simple counter. It returns an accurate value live, on-the-fly, and makes it very easy for the sys admin to collect and process. Very little work is done on it’s end. Even the current implementation of ReqPerSec has very little cost: it just does some simple division on-demand, using counters it already tracks anyway.

To keep an actual average request rate, there would be some more resource involved. It would need to set a timer (ie: every 60 seconds), have a callback for the timer (to actually handle the calculation), and store counters in memory to calculate deltas. However, in reality, this will really have a very minimal impact on server performance and system load (a few extra CPU hits every minute, storing two little 64-bit counters, etc).

So what did I end up doing? I hacked up my own mod_status using shared memory!

Using the existing mod_status code, I pulled some code from a shared memory module example and smashed them together to calculate an accurate ReqPerSec metric on the fly, returned when you request server-status.

How does it work?

When you request server-status, it will go into shared memory, pull out the total accesses and uptime from the previous server-status request, do deltas on the current total accesses and uptime, then return the rate. It will then put the current total accesses and uptime back into shared memory for the next poll (base is our shared memory baseline).

ap_rprintf(r, "ReqPerSec: %g\n", (
    ( (float)count - (float)base->prev_req_count ) /
    ( (float)up_time - (float)base->prev_uptime ) )
);

There is of course all of the shared memory code, but the end result is the calculation above. We poll every 60 seconds, so we incur a small cost every minute, but, like I mentioned before, it’s not noticeable, and it gives us the view we need into the environment.

Some of you may also wonder what happens to the first request, since there isn’t any data in our shared memory space yet. For now, it will be inaccurate, but only for the first request in the lifetime of the parent, which I can deal with until I make it better.

Well, that’s all for this one. It was an interesting discovery in the world of Apache metrics collection, and an interesting adventure to hack together the shared memory module. I’m sure there are other scenarios out there like this one; if you come across one, feel free to drop a comment!

I’m off to the Apple Store now- it’s iPad launch day

use perl: pack() to send floats between intel/sparc with proper endian-ness.

2010-02-22T08:17:51Z

So I like using Perl, if anyone hasn’t realized yet. Python seems awesome and all, but I haven’t had the time to start writing things in Python. I’m still at the stage when, if a tool needs to be put together, it needs to be done ASAP; I can’t delay trying to figure out Python particulars.

In any case, one of our metrics publishing systems is wrapped by a few different object classes I put together in Perl (follows a certain protocol created by another group, etc). Of the many data types supported by our metrics receivers, we can do things like 32-bit unsigned integers, single precision (32-bit) floating points, and a few others.

Now, when I would publish the 32-bit uint’s, it would work great- no issues. The receivers would expect a byte stream, so I would use pack() to send my values as binary. For example:

$val=29;
print $sock pack('N',$val);

Easy, whatever. However, when I would send values as a single precision float, it would totally be hosed on the other end. For example:

$val="30.0";
print $sock pack('f',$val);

On the other end, they would get some massive negative number. I don’t deal with this too much, things usually just work. Thankfully, my mind was jogged in regards to standards- there are lots of standards around byte order for integers (16 bit, 32 bit, 64 bit), but not as much for float; one architecture may use big-endian, and another architecture might use little-endian.

In my searches, I found that Intel stores floating points in little-endian byte order, and Sparc stores them in big-endian byte order. Sure enough, I was trying to publish from an Intel-based BSD host to a Sparc based Solaris host. Aha!

Luckily, pack() can *also* handle this for us! With a simple carat added to the templates section of pack(), I can easily pack the value in big-endian byte order and resolve our issue. I do that like so:

$val="30.0";
print $sock pack('f>',$val);

And, ta da! Worked like a charm. Was it difficult to get it working? No. But, it was fun to have to deal with something like this. On any given day, you don’t really worry about endian-ness; things just work. This was one of those days where I actually had to use the old noggin, and that was refreshing.

calculating dequeue time to tune load balancer queue handling

2010-05-03T04:51:23Z

Maybe there’s already some obvious answer to this that I just ignored sometime in years past, but at some point, you may use a load balancer that will allow you to queue requests up in the event the backend apps are running slower than normal. Instead of killing the apps, the load balancer sets requests aside until the app is ready for more, then it sends it off.

However, what happens if the app is REALLY in trouble? Your connection queue could very easily grow into thousands of piled up requests all waiting to be serviced. At some point, you’re going to have to start dropping requests from the queue: it will get so large, that even after the app comes back, you may never recover from the pile of requests (or if you do, it might be 60 seconds worth, at which point your user will probably have given up anyway).

So, here we go.

First: how many requests per second (max_requests) can your application slice handle? In this case, your application slice will be however many services are being load balanced. Depending on the max_requests, that will allow us to know how much headroom will be available for a given slice utilization; we will use that headroom to determine how many requests over and above the current utilization are available to help dequeue.

Second: What is our average utilization? If on a given day we are 60% utilized, then we should in theory have 40% capacity available to help dequeue in the event of a surge/backup/whatever. That is, once we begin to dequeue, we will actually be running at a full 100% until we’re back from being under, then we’ll be back at the usual 40%.

Third: How long were we backing up for, ie: what is the worst-case scenario we want to prepare for? If underlying apps rely on a database, and the database locks up for 10 seconds, then that will be 10 seconds of queueing we will get to enjoy. If we can’t afford 10 seconds, then what is the most we’re willing to deal with?

Equation time! Here are the three variables we have to deal with:

max_requests = number of requests/sec that our load balanced slice can handle (ie: 400 .. as in 400 req/s)

utilization = percentage of our average utilization that we want to plan around (ie: 0.6 .. as in 60% avg utilization)

queue_time = number of seconds we were queueing for (ie: 10 .. in which the db locked up)

And here are some things we can derive:

current_requests = ( max_requests * utilization) <--- ie: req/s utilized

available_requests = ( max_requests * ( 1 – utilization ) ) <--- req/s available ... this is also what you should consider headroom

queue_size = ( current_requests * queue_time )

At this point, it’s simple:

time_to_dequeue = ( queue_size / available_requests )

Basically, all we’re saying is, take the number of current requests/sec we’re handling, and multiply it by the number of seconds we were queueing. This will give us the number of requests that are backed up.

Once our queuing has stopped, and we can begin to dequeue, we will still be handling the usual rate of requests (ie: 60% of our max_requests), but we have that 40% of headroom (ie: available_requests) to handle the queue, so we divide our total queue_size up by the available_requests rate, and that will give us the number of seconds our load balancer will take to dequeue.

Expanded, the equation is this:

time_to_dequeue = ( ( ( max_requests * utilization ) * queue_time ) / ( max_requests * ( 1- utilization ) ) )

But wait! The beauty of this is, the time_to_dequeue is going to be the same for any given max_requests (you can see that from the equation). Regardless of our request rate, the rate at which we dequeue will be same relative to the percentage of utilization and a given queue_time.

So, we can simplify, removing anything having to do with request rates, allowing us to place time calculations for anything where we know the utilization and time we were queueing.

time_to_dequeue = ( ( utilization * queue_time ) / ( 1 – utilization ) )

Now, a lot of the times, we need to know how LARGE the queue will be at the given peak (to configure the load balancer accordingly), and in that case, we will need to know request rates.

Either way, I found this to be an interesting little exercise- got my mind thinking about math, and having an equation was much easier to help me build a table of data and pump it into gnuplot.

Enjoy!

adventures in perl: foreach returns pointers to elements

2009-12-17T18:54:08Z

I’m not sure how I’ve never run into this issue before. In some work I was doing recently, I ran into what I thought was a nasty bug [in my code] but couldn’t explain it. Without thinking, I started trying to undo this, fix that, hack this, and ignore that.

After the meeting I was in finished, I was able to sit down and actually put some thought into what was happening. The one thing that stood out was interesting, but until then, I had no clue it was how Perl interpreted [in that scenario]. So, I wrote a short test script, sure enough proving the theory, and was able to fix my error.

Take a look at this:

my @animals = ("cat","dog","emu","frog");
 
foreach my $animal(@animals) {
    printf("do not eat the %s\n",$animal);
}

which dumps out this:

do not eat the cat do not eat the dog do not eat the emu do not eat the frog

In my head, the code above would work like this: for each element in the array @animals, copy the data from that element into a new scalar $animal, then print it out. Simple enough. Now, consider this sample:

my @animals = ("cat","dog","emu","lemur");
 
foreach my $animal(@animals) {
    printf("do not eat the %s\n",$animal);
    $animal = sprintf("rabid %s",$animal);
}
 
printf("\n");
 
foreach my $animal(@animals) {
    printf("do not eat the %s\n",$animal);
}

I had *EXPECTED* it to output this:

do not eat the cat do not eat the dog do not eat the emu do not eat the frog

For the first loop, again, my thinking was that we copy each array element’s data into the new scalar $animal, print it, modify it (by adding ‘rabid’ to it), but do nothing with the modification (we are just modifying $animal, which should be assigned by copy, of which would be lost when we iterate to the next element). Then, in our next loop, we iterate over @animals again, initializing $animal yet again for each element, so we just hit the un-modified @animals array and see the same thing.

[un]Surprisingly, I was totally wrong. This was the output I got:

do not eat the cat do not eat the dog do not eat the emu do not eat the frog

do not eat the rabid cat do not eat the rabid dog do not eat the rabid emu do not eat the rabid frog

Wow! What happened? Turns out, like the title suggests, when you use foreach in Perl, it actually assigns $animal to be a pointer (reference, whatever) to that array element, and not a copy. When you make any changes, the change applies directly to the element the array. As another example, it’s for-loop equivalent would be this:

my @animals = ("cat","dog","emu","lemur");
 
for( my $index=0 ; $index < @animals ; $index++ ) {
    printf("do not eat the %s\n", $animals[$index] );
    $animals[$index] = sprintf( "rabid %s", $animals[$index] );
}

Fun! Doing some research after the fact, it turns out there has been some mild discussion on the topic. Some consider it a bug, but in reality, it works by design. What does this mean for you? Well, if you happen to be iterating over an array using foreach, and want to modify each element, and plan on looping over the array multiple times, make sure you understand what’s going on behind the scenes, or else you’ll run into the same issue I did. You can create a temporary variable (which will assign by copy), ie:

my $new_animal = $animal;

or you can iterate using a for-loop and just do it like this:

my $animal = $animals[$index];

There of course may be some cases where you want to take advantage of the referencing feature, and if you do, all the more power to you.

Well, that’s all I have for now. Hopefully this helps someone out some day. Enjoy!

nxdomain redirection with powerdns and lua

2009-11-25T11:33:35Z

For those of you unfamiliar, NXDOMAIN is a DNS query response for non-existent domain. An example would be if you start typing google.com, but sneeze halfway through, and end up going to gegooele.com. That domain doesn’t exist, and you will ultimately receive an NXDOMAIN response from your upstream DNS server instead of an A record with the IP of the domain.

These days, a lot of ISPs have been doing NXDOMAIN redirection. Instead of handing back the NXDOMAIN response, they will intercept it and send you back an A record with an IP of one of their web servers (lets say 1.2.3.4 as an example). Your machine will ultimately think that gegooele.com has the IP 1.2.3.4, connect to that IP, and do a GET request with the HTTP host header of gegooele.com. The ISP’s web server, which specifically handles NXDOMAIN “redirected” HTTP requests, will then take that host header, pass it through some type of dynamic script, and display a nice custom error page with possible corrections, advertisements, etc.

Now, here’s where PowerDNS comes in: there are companies out there whose specialty is to sell DNS appliances and software solutions, solely for the purpose of doing NXDOMAIN redirection. However, if you are in the market for doing something like this (which is a whole separate discussion altogether, since there are strong opinions about this subject), you can save yourself a chunk of change by instead using PowerDNS (pdns-recursor, to be exact) with built-in Lua scripting. PowerDNS is open source, free, awesome.

We’re going to skip the build/compile details for now, and assume you have the latest PowerDNS recursor installed, and that Lua scripting has been compiled in. Chances are, it has. If not, check out http://www.powerdns.com and download the latest source. Once it’s installed, you just need to write a Lua script to modify DNS results, configure your recursor to use the script, and start up pdns-recursor. Let’s get to it.

First, you need to define a Lua script to handle DNS responses. Currently, Lua will only handle two types of responses: nxdomain, and preresolve. Preresolve is a response to give before any resolution has taken place. This is to basically hijack requests with a static response.

To define, we’ll add this line to our recursor.conf:

lua-dns-script=/opt/pdns/bin/nxdomain.lua

Now, let’s write our script. At it’s most basic level, it’s really easy:

function nxdomain (ip,domain,query_type) ips={} if query_type ~= pdns.A then return -1, ret end ips[1]={ query_type=pdns.A, content="1.2.3.4" } ips[2]={ query_type=pdns.A, content="5.6.7.8" } return 0, ips end

This script will first make sure our NXDOMAIN response is the result of a request for an A record. As long as it is, we will build an A record response to return instead of an NXDOMAIN. Per this code, the client will ultimately receive an A rotor response with two records.

Now that our script has been written and added to our recursor.conf, let’s start up the recursor and try it out!

$ /opt/pdns/sbin/pdns_recursor –config-dir=/opt/pdns/etc

Assuming the script is loaded successfully, you will see the following from the startup output:

Nov 25 03:09:36 Loaded ‘lua’ script from ‘/opt/pdns/bin/nxdomain.lua’

Looks good. Now, let’s see what happens when we try to hit our non-existent gegooele.com:

$ host gegooele.com gegooele.com has address 1.2.3.4 gegooele.com has address 5.6.7.8

And there we go! NXDOMAIN redirection without the need for a commercial solution. Of course, there are more things you can do [with the Lua script]. WIth the proper log level, you can have it log every time the handler is called. For example, you can add a simple print line:

print ("nxdomain handler received: ", ip, domain, query_type)

You can also have it make decisions based on incoming IP, hostname/domain, and query type. However, the point of this was just to show the basics with what you can do. For more Lua syntax, and PowerDNS capabilities with Lua, check out the Lua website and PowerDNS wiki. Enjoy!

efficient reads on perl file handles: while vs. foreach

2009-11-17T05:32:13Z

A while back, I had put a Perl script together to read in some large files, do some data mining, and dump out the results I wanted to see (well, maybe they weren’t the results I was hoping for, but results nonetheless). Now, when the script ran, it would go through it’s first few routines nice and quick-like, but when it went to the following routine, it would become unbearably slow. The first couple of times, I didn’t think much of it; I liked feeling that the script was hard at work (of course).

However, after a few more runs, it was getting kind of ridiculous: the routine in question was the one that did reads from the file handle. Now, I can understand on a multi-gigabyte file it might take a while (read in each line, parse it through a regex, and populate a data structure). However, it was causing the entire system to hang, and that made no sense at all given it’s relatively easy instructions. Here’s an example of what I was doing:

my $fd = IO::File->new("data.out",O_RDONLY);
 
foreach my $line(<$fd>) {
   @data = $line =~ /regex_pattern/;
}

Simple enough. So, on the next run, I started tracking the PID. Now, I saw two things: 1) the CPU was pinned, and 2) memory utilization was going through the roof. Bingo. Basically, my multi-gigabyte file was being read into memory, eating up all the remaining free physical memory, which was forcing the box to start swapping (which was eating up the CPU). Now it made sense why things were going ridiculously slow.

Now, what didn’t make sense was the fact this was happening at all. For each line of data, I was only pulling about 10% of it into my data structure (bytes parsed vs bytes per line), so I shouldn’t see memory utilization match that of the file. Additionally, I had worked with large files like this before, on systems with less memory, and never had an issue. What was different?

I took lunch, thought about it, and somewhere in there thought to myself, “hrmm, I wonder if it has to do with the foreach loop.” In the past, I had always used a while-loop, but didn’t really consider there might be an extreme difference between how each was interpreted when reading a file handle. I went back to my desk and modified the code to use a while loop instead:

my $fd = IO::File->new("data.out",O_RDONLY);
 
while(my $line = <$fd>) {
   @data = $line =~ /regex_pattern/;
}

Sure enough, it tore right through the file, did it’s parsing, and was done very shortly after. Interesting! So, what happened?

Basically, it’s a major difference in how Perl handles while loops vs. foreach loops.

In a while-loop, Perl will blindly shift lines one-by-one off of the array/fd you pass until it reaches an EOF. Once it does, it breaks out and you’re done. This is how it should work (and does exactly what we expect it to do), so that makes sense.

Now, in a foreach-loop, the Perl interpreter needs to know the end of the array before it begins iterating over it (whereas with a while loop we just go until EOF). If the data is already an array, that’s not a problem. However, if the data is a file handle, then the only way it can know the end is by reading the entire file into memory and ultimately create an array of the data to return back to the foreach iterator.

Most of the time, people aren’t running through multi-gigabyte files, so doing a foreach or a while on a file handle won’t really be noticeable, even though there are two completely different logic paths being followed behind the scenes.

Ultimately, use a while loop when reading off of a file handle. It’s cleaner, uses less overhead, and has obviously proven itself to be more fit for this task.

There are a lot of Perl hackers out there (many of whom I have worked with) who may end up commenting or shed some deeper technical insight. By all means, I would love to hear it! If you think I’ve done a good job summarizing and detailing the issue, that works just as well! Enjoy!