Dave Goodell's Blog

Debugging Wily Stack Overflows in MPI Programs

2011-04-15T00:05:00.001-05:00

I recently was debugging a problem in MPICH2 with the new MPI-3 nonblocking collectives implementation that involved a stack overflow. The code in question was usually blowing up with a SIGABRT that came from a stack trace ending in __stack_chk_fail. This bug was particularly nasty because it was a Heisenbug: it kept changing the failure location and mode whenever I would insert debugging code in order to track it down (hence the aforementioned "usually").

For reasons I will explain later, my usual trick for this sort of thing did not help. Adding in those macros would cause the bug to vanish, or to reappear in a different function that had not been instrumented with MPIU_SG_ macros.

Luckily, I ran across this very helpful blog post that eventually helped me track down the real problem. My basic approach was the same as the one outlined there, but it needed modification due to the pain inherent in debugging a parallel MPI application. This particular bug required running at least 4 processes in order to manifest itself, and it would only appear after several iterations within the test program. So I used mpiexec + xterm + gdb + a gdb script file to handle the job.

First, here's an example stack trace of the failure (using valgrind to easily obtain stack traces):

% mpiexec -l -n 4 valgrind -q ./icbarrier |& sort_labeled_output

=====================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= EXIT CODE: 6
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
=====================================================================================
APPLICATION TERMINATED WITH THE EXIT STRING: Abort trap (signal 6)
[1] ==47313==
[1] ==47313== Process terminating with default action of signal 6 (SIGABRT)
[1] ==47313== at 0x10027D616: __kill (in /usr/lib/libSystem.B.dylib)
[1] ==47313== by 0x1003022CB: __stack_chk_fail (in /usr/lib/libSystem.B.dylib)
[1] ==47313== by 0x10005D9D3: MPIDI_CH3I_Progress (ch3_progress.c:539)
[1] ==47313== by 0x100109B0C: MPIR_Wait_impl (wait.c:67)
[1] ==47313== by 0x100109FFA: PMPI_Wait (wait.c:171)
[1] ==47313== by 0x100004506: MPI_Barrier (nbc_pmpi_adapter.c:24)
[1] ==47313== by 0x10000141D: main (icbarrier.c:37)
[2] ==47314==
[2] ==47314== Process terminating with default action of signal 6 (SIGABRT)
[2] ==47314== at 0x10027D616: __kill (in /usr/lib/libSystem.B.dylib)
[2] ==47314== by 0x1003022CB: __stack_chk_fail (in /usr/lib/libSystem.B.dylib)
[2] ==47314== by 0x10005D9D3: MPIDI_CH3I_Progress (ch3_progress.c:539)
[2] ==47314== by 0x100109B0C: MPIR_Wait_impl (wait.c:67)
[2] ==47314== by 0x100109FFA: PMPI_Wait (wait.c:171)
[2] ==47314== by 0x100004506: MPI_Barrier (nbc_pmpi_adapter.c:24)
[2] ==47314== by 0x10000143B: main (icbarrier.c:45)
[3] ==47315==
[3] ==47315== Process terminating with default action of signal 6 (SIGABRT)
[3] ==47315== at 0x10027D616: __kill (in /usr/lib/libSystem.B.dylib)
[3] ==47315== by 0x1003022CB: __stack_chk_fail (in /usr/lib/libSystem.B.dylib)
[3] ==47315== by 0x10005D9D3: MPIDI_CH3I_Progress (ch3_progress.c:539)
[3] ==47315== by 0x100109B0C: MPIR_Wait_impl (wait.c:67)
[3] ==47315== by 0x100109FFA: PMPI_Wait (wait.c:171)
[3] ==47315== by 0x100004506: MPI_Barrier (nbc_pmpi_adapter.c:24)
[3] ==47315== by 0x10000143B: main (icbarrier.c:45)

So I took a quick gander at the assembly for MPIDI_CH3I_Progress to find the canary value initialization and comparison code:

% objdump -d icbarrier > icbarrier.s
% sed -n -e '/MPIDI_CH3I_Progress>:/,/^$/ p' icbarrier.s | head -n 15

000000010005c7f0 <_MPIDI_CH3I_Progress>:
   10005c7f0: 55 push %rbp
   10005c7f1: 48 89 e5 mov %rsp,%rbp
   10005c7f4: 41 57 push %r15
   10005c7f6: 41 56 push %r14
   10005c7f8: 41 55 push %r13
   10005c7fa: 41 54 push %r12
   10005c7fc: 53 push %rbx
   10005c7fd: 48 81 ec 08 02 00 00 sub $0x208,%rsp
   10005c804: 48 89 bd 10 fe ff ff mov %rdi,-0x1f0(%rbp)
   10005c80b: 89 b5 0c fe ff ff mov %esi,-0x1f4(%rbp)
   10005c811: 48 8b 05 60 c8 10 00 mov 0x10c860(%rip),%rax # 100169078 <_pvars+0x78>
   10005c818: 48 8b 10 mov (%rax),%rdx
   10005c81b: 48 89 55 c8 mov %rdx,-0x38(%rbp)
   10005c81f: 31 d2 xor %edx,%edx

So in this code it looks like the offset is -0x38 rather than -0x14. Let's find the two addresses at which we will set our breakpoints:

% sed -n -e '/MPIDI_CH3I_Progress>:/,/^$/ p' icbarrier.s | grep -- '-0x38(%rbp)'

   10005c81b: 48 89 55 c8 mov %rdx,-0x38(%rbp)
   10005ce61: 48 8b 55 c8 mov -0x38(%rbp),%rdx

And now for our gdb script:

% cat gdb.script

break *0x10005c81b
break *0x10005ce61
commands 1
silent
set variable $cow = (unsigned long*)($rbp - 0x38)
watch *$cow
continue
end
set variable $count = 3
commands 2
silent
delete $count
set variable $count = $count + 1
continue
end
run

% mpiexec -n 4 xterm -e gdb -x gdb.script ./icbarrier

That command launches four xterm windows, each containing a single gdb instance that was running icbarrier as the inferior. Here's the resulting xterm+gdb screenshot for the process that hit the watchpoint first:

Unlike in many situations, this backtrace alone was not sufficient for me to figure out what the real bug was. But, given that I changed some high level code (the nonblocking version of barrier, MPIX_Ibarrier), I knew it was extremely unlikely that I had tickled some long-latent bug in the frequently used routine MPIDI_CH3U_Request_unpack_uebuf. So after a few "WTFs" and some further head scratching, I took a hard look at the upper level code (error checking and boring code omitted for clarity):

int MPIR_Ibarrier_inter(MPID_Comm *comm_ptr, MPID_Sched_t s)
{
   int mpi_errno = MPI_SUCCESS;
   int size, rank, root;
   int buf = 0;

   /* do a barrier on the local intracommunicator */
   mpi_errno = MPIR_Ibarrier_intra(comm_ptr->local_comm, s);
   MPID_SCHED_BARRIER(s);

   /* rank 0 on each group does an intercommunicator broadcast to the
   remote group to indicate that all processes in the local group
   have reached the barrier. We do a 1-byte bcast because a 0-byte
   bcast will just return without doing anything. */

   /* first broadcast from left to right group, then from right to
   left group */
   if (comm_ptr->is_low_group) {
   root = (rank == 0) ? MPI_ROOT : MPI_PROC_NULL;
   mpi_errno = MPIR_Ibcast_inter(&buf, 1, MPI_BYTE, root, comm_ptr, s);

MPID_SCHED_BARRIER(s);

   /* receive bcast from right */
   root = 0;
   mpi_errno = MPIR_Ibcast_inter(&buf, 1, MPI_BYTE, root, comm_ptr, s);
   }
   else {
   /* receive bcast from left */
   root = 0;

        mpi_errno = MPIR_Ibcast_inter(&buf, 1, MPI_BYTE, root, comm_ptr, s);

        MPID_SCHED_BARRIER(s);

        /* bcast to left */

   root = (rank == 0) ? MPI_ROOT : MPI_PROC_NULL;
   mpi_errno = MPIR_Ibcast_inter(&buf, 1, MPI_BYTE, root, comm_ptr, s);
   }

fn_exit:
return mpi_errno;
fn_fail:
goto fn_exit;
}

This function is called, adding operations to a communication schedule, then an MPI_Wait call is made later to actually force the bulk of the schedule to complete.

Do you see the bug? I'd be surprised if you did, since there are few people familiar with the style of MPICH2 code, and even fewer people who know how the NBC implementation works. But if you do, give yourself a pat on the back, and then please send me bug reports and patches for my other bugs :)

Take a closer look at the buffer used for the intercomm broadcast. It's a stack allocated variable, which worked great in the blocking version of this routine. Unfortunately, this is a nonblocking barrier, so this routine exits before the ibcast is actually ever performed. So now we've managed to schedule a broadcast to occur that will overwrite a location on the stack that no longer actually refers to int buf;. Instead, if we are unlucky it will refer to some random piece of a different function's stack frame. In the backtraces above, this random piece was the -fstack-protector/SSP canary memory. The bug moved around when I would insert debugging variables because they would affect the specific location of the canary value in memory, sometimes yielding a corrupted parameter or local variable instead.

Changing that buffer to be malloced instead fixed the problem.

Roku stands behind their product

2011-04-03T21:31:00.000-05:00

My older Roku player (this model, N1000-B) died a couple of months ago. The wireless NIC went out to lunch for some reason, showing 00:00:00:00:00:00 as its MAC address and totally unable to see any wireless networks, just as described here. I was busy and lazy at the time, so we just switched over to watching Netflix streaming content on our Wii instead.

Last Friday I finally got around to calling Roku about the issue and the technician that I got was very helpful. He didn't give me the "reboot runaround"; he just asked me what troubleshooting steps I had taken, and ensured that my MAC address was showing all zeroes. After a bit more discussion, he offered to replace the unit for free, even though it was nearly a past its warranty expiration. They mailed the replacement unit (looks like it might be factory refurb?) via FedEx and 3 business days later the new unit was in my hands. The new unit setup was painless (as expected), and a return shipping label was included so even the round-trip shipping was free.

I was extremely pleased that Roku stood behind their product, even after it was out of warranty. They did exactly the right thing to change my mind from "I'll never buy another one of these" to "I'd happily buy something from them again."

(full disclosure: I have no affiliation with Roku, other than being a fairly satisfied customer)

m4_include and aclocal

2011-03-05T10:50:00.000-06:00

I ran into some surprising behavior a little while back while hacking on the MPICH2 build system, that I thought I should mention here. But first, some background info is in order. We have numerous subsystems that are currently configured (in the autoconf sense) by running an autoconf-generated configure script from various points in the top level configure. Think AC_CONFIG_SUBDIRS, but the subconfig happens at the place where the macro is used, rather than being deferred to config.status-time. There are some advantages to this scheme, but there are two fairly severe disadvantages:

The subconfigure scripts tend to re-test a whole bunch of information about the build environment (e.g., six repeated checks for "how to run the C preprocessor"). Not only is this wasteful in terms of time, but there is always unlikely possibility that two configure scripts will come to subtly different conclusions about the build environment, which could cause wacky bugs later in the build.
There's no easy mechanism to utilize the results of subconfigure tests that should be global. That is, if we add -I/path/to/includes to CPPFLAGS that should be used for all compilation, then this won't be seen in the top-level configure without some additional work. We currently "pass" these values back via a "localdefs" file that is listed in the subconfigure's AC_CONFIG_FILES and contains some @FOO@ substitutions that are either AC_SUBSTed or "precious" (which implies AC_SUBST for that var). This stinks because it's a fairly manual process, and it's impenetrable to outsiders who are unfamiliar with our build system because the sourcing of "localdefs" is hidden inside a custom autoconf macro.

So I'm taking a stab at converting these configure files to be configure.in fragments that are then m4_included. This will cause those tests to be run in the same shell as the top-level configure and will only run the additional tests that are needed by the subsystem. In the process of that conversion, I ran into some odd error messages from the autotools that are represented by the following example program.

Quick Quiz: The code in Listing A will successfully autoreconf and configure. The code in Listing B will fail during autoreconf's invocation of aclocal. Why?

Common code:

## contents of configure.in
AC_INIT([foo],[1.0])
AM_INIT_AUTOMAKE([-Wall -Werror foreign 1.11 subdir-objects])
LT_INIT([])
AC_PROG_CC
var="yes"
AM_CONDITIONAL([COND],[test "$var" = "yes"])
dnl this makes m4 bail, claiming that AM_COND_IF is undefined
m4_foreach([subsys_i],[[subconfigure]],[m4_include(subsys_i[.m4])])
AC_OUTPUT([Makefile])

Listing A

## contents of subconfigure.m4
AC_PROG_GREP

Listing B

## contents of subconfigure.m4
AM_COND_IF([COND],[echo COND is true],[echo COND is false])

Do you see the problem here? I sure didn't, even after some moderate debugging. Go ahead and think about it for a minute, I'll wait...
.
.
.
.
.
.
.
.
.
Luckily, the nice folks on the autoconf list helped me understand what was going on here. There are two things happening here:

Not all macros are created equal. In my original world view, AC_PROG_GREP and AM_COND_IF are both just AC_DEFUNed m4 macros and should behave more or less identically. However this assumption is totally false. AC_PROG_GREP is a macro that is built-in to autoconf, and therefore does not get distributed with your package via aclocal.m4, while AM_COND_IF comes with automake and is managed by aclocal, and must be placed into your package's aclocal.m4 in order to be properly expanded.
It turns out that aclocal isn't as smart as I thought it was. When it traces the m4 files looking for macro definitions that it must add to aclocal.m4, it doesn't perform real m4_includes at that time. Instead it uses some regex heuristics in order to determine what file is being included and then traces that file separately. Since the name of the file that is being included is being computed via an m4 macro in the example above, aclocal gets rather confused and just doesn't trace the subconfigure.m4 file.

The upshot of all of this is that I can't use m4 code to programmatically include subsystem m4 fragments. They must either be hard-coded m4 file names or be autogenerated by some other non-autotools step, such as a quick shell script run before autoreconf.

Midwest Ice Climbers Ice Festival, 2011

2011-01-21T22:46:00.002-06:00

Last weekend I got to try ice climbing out at the Ice Pit near Green Bay, Wisconsin. It was a very cool setup, sort of an outdoor ice climbing gym in an active quarry. Normally you must be a member to climb there, but for the festival it was just $20 for the whole weekend. I did the trip with five other guys, all rock climbers, three of which also work at the Lab.

I found the climbing itself surprisingly difficult, despite my (at least modest) rock climbing abilities. According to others that I spoke with at the festival, most of the routes that we were climbing were in the WI3-WI4 range, all about 100 feet high. The hardest part was getting accustomed to the crampons and really trusting my feet. When I didn't trust my feet enough, I would put too much weight on my hands which led to totally pumped out forearms after 100 feet of ice. We were each able to get three climbs in each of the two days, which was enough to tire me out pretty effectively.

There was a fair amount of demo gear to borrow from various vendors, including boots, crampons, and ice tools. However, we got in touch with Down Wind Sports (in Michigan's UP) via email before the festival and arranged to rent gear from them on Friday evening instead. That way we were able to save a lot of time each morning and were certain that we would have gear that fit. Bill @ Down Wind Sports was extremely nice and helpful, I'd do business with them again without hesitation. For $40/person we had boots and crampons for the whole weekend. Ice tools were easy to come by as demo gear, and someone in our party already had a set as well.

Overall it was a very enjoyable trip. I might do a little easy ice climbing in the future as the opportunity presents itself, but for now I'm not planning on picking it up as a regular hobby. I already have too many hobbies as it is, and this isn't the cheapest or most accessible hobby around. A brand new set of top-rope ice equipment (boots, crampons, tools) adds up to $800-$1000, while a used set is probably half of that plus more time hunting for a decent deal. And northern Illinois isn't exactly teeming with waterfall ice to climb.

You can find a few more photos of the event on my Smugmug page.

sort_labeled_output script

2011-01-20T23:38:00.001-06:00

Quick MPICH2 tip: passing the "-l" option to MPICH2's mpiexec (hydra) yields rank-labled output like so:

% mpiexec -l -n 2 ./examples/cpi

[0] Process 0 of 2 is on g2-2.local

[1] Process 1 of 2 is on g2-2.local

[0] pi is approximately 3.1415926544231318, Error is 0.0000000008333387

[0] wall clock time = 0.000210

This is handy for figuring out which process is printing what without reaching in and modifying all of your printf statements to explicitly print the rank as well.

For more complicated output, and especially multiline output that you would get from running valgrind, this interleaved output can be hard to follow. Enter sort_labeled_output:

#!/bin/zsh

# assumes output that matches the following regex: /^\[\d+\] .*$/

exec sort --stable --key=1.2,1.0 -n

The script just stably sorts the output by rank prefix, which makes it much easier to read multiline output.

Given that this is just a thin wrapper around GNU sort, I actually didn't build this script for a long time. But it turns out that this is a task that I do all the time and reconstructing the sort arguments always distracts me a bit from the debugging task at hand. So the script has been a big productivity win.

Climbing in Agio Farango

2011-01-20T23:25:00.000-06:00

Dave following on "Το ονειρο και ο φοβος" (Google trans. "The dream and the fear"), 5b (approx. 5.8+), in Agio Farango, Crete, Greece.

I had business on Crete, Greece and was fortunate enough to be able to stay an extra day for some sport climbing. We were staying east of Heraklion, near Hersonissos, so Agio Farango, on the south side of the island seemed the best option for a day trip. We picked up a guide book online ahead of time, rented a car, and headed southwest about 100 km or so, probably 2.5 hours each way, to Kali Limenes.

We took the approach from Kali Limenes, a hike from the east of the gorge a few hundred yards inland from the Mediterranean. Despite online directions and directions in the guide book, it was extremely unclear where we should park and where the actual marked path (with blazes and cairns) started. Below I've included a map based on my several-months-old recollection of the route and my interpretation of the satellite photos:

View Agio Farango approach from Kali Limenes in a larger map

If I were to do it all again, I would probably try the alternative approach from Moni Odigitrias at the north end of the gorge, even though this website advises the Kali Limenes route. We burned up a bit of time just trying to figure out where the trail was.

The climbs were fun, although I found the difficulty a bit stiff in some cases. Also, many of the routes had sections of very sharp limestone rock, which was difficult to hold onto. Good footwork was essential for such routes. While most routes were well bolted, I also followed on my first trad route, an easy 4c (French scale, approx. 5.7 YDS, even though it felt easier than that) named Ξωτικο that started right on the beach and had a great view of the water and beach from the top.

Goats are all over the area, especially down in the gorge itself, as well as soloing up on the cliffs above you. They are amazing climbers, able to scale walls and routes that look difficult for us humans with little fear or difficulty. However, watch out for other tourists and their dogs. While I was top roping one of the routes someone's dog chased a goat between the rock and my belayer 30 feet below me, which was a rather unsettling experience.

Overall, a great trip. It's a pretty intense day trip from the Heraklion area, and the driving and trail finding can really eat into your climbing time. If you have the time, I'd recommend bringing a tent and some food and staying overnight on or near the beach in order to fit more climbing into a more relaxed outing.

More photos from the trip can be found here on Smugmug.

reviving posting...

2011-01-11T23:25:00.001-06:00

I looked at my blog again today and it turns out that my last post was in August of last year. That's around the time that the Fall travel season picked up, followed immediately by Thanksgiving and Christmas, so things have been pretty busy for me. Here are the highlights from my period of radio silence:

the annual Seattle trip for "Birthstravaganza," a joint birthday party for two of my old friends from college
New-timers / Old Climbers event with the Chicago Mountaineering Club at Devil's Lake, WI
EuroMPI 2010 in Stuttgart, Germany...
... immediately followed by IEEE Cluster 2010 in Crete, Greece
one day of sport climbing in Agio Farango, Crete, Greece
Heather and I went to Spain for two weeks on vacation in November
SC10 in New Orleans, LA
hosting Thanksgiving for my parents and siblings at our home for the first time
I finished my long-overdue bookcase (no pics yet).

In retrospect I had a ton to blog about but a lack of time and motivation. I'll try to put up a few pics and details about some of these items in catch-up posts, but I'm not making any promises. In the unlikely event that anyone wants details on any particular event, please post a comment to let me know.

In forward-looking news, I'm going ice climbing this weekend at the Ice Pit in Green Bay, WI, which I'm really excited about. I also was lucky enough to snag tickets to the Chicago Beer Society's Brewpub Shootout, which is an excellent event. Lots of great beers that are brewed just for the event, in addition to some amazing food that is served.

disable animations in Skim to avoid using the MacBook Pro external (power hogging) GPU

2010-08-01T19:01:00.001-05:00

My work laptop is a 15" MacBook Pro from the current generation (first batch with Nehalems). This laptop includes two GPUs: an integrated one from Intel and an external one from Nvidia. The integrated unit has generally lower performance but generally lower power consumption as well. The external unit has generally higher performance but generally higher power consumption.

OS X has a pretty cool feature where it will seamlessly switch between the two cards based on current software performance requirements. However, as with any automagical switching, the switching software sometimes makes a bad decision, or simply doesn't have enough information available to make a good decision. So I use gfxCardStatus to figure out which unit is currently in use. It shows me which unit is in use, as well as which applications (or the external display) are forcing the Nvidia card to be used at any given time. It also lets me force one unit or the other to be used instead of dynamically switching between the two.

Here's a (trivially incomplete) list of applications that force external graphics usage. Among them is Skim, my favorite PDF reader. Oddly, it only uses the higher-powered graphics after searching the document. This annoyed me greatly because I tend to leave the app open for long periods of time, and I also tend to search fairly often.

Luckily, a recent version of Skim added a new hidden preference to disable animations. This seems to eliminate the dependency on the external display, and thereby save my battery power. Just type the following command into your terminal and restart Skim:

defaults write -app Skim SKDisableAnimations -boolean true

fix codetrunk diff/patch colors in Firefox

2010-06-03T09:11:00.010-05:00

At work I often use codetrunk.com to share patches for code review with other developers on my team. I used to use pastebin.com, but ever since it changed ownership a few months ago it has been far less useful. They broke the major pastebin command line tools and the syntax highlighting for diff/patch types is very poor.

The codetrunk site was simple enough that I quickly wrote up my own command line script for uploading content.

Unfortunately though, the codetrunk highlighting for diff/patch files (the primary type that I use there) leaves something to be desired. By default additions are blue and deletions are green, which is hard to read if you are familiar with the green/red highlighting used by git and other programs. Fortunately, this is pretty easy to fix with some CSS in your Firefox userContent.css file in your profile. Just stick the snippet below into that file (not your userChrome.css file) and restart your browser.


@-moz-document domain(codetrunk.com) {
  .syntaxhighlighter .diff.comments, .syntaxhighlighter .diff.comments a {
      color: red !important;
  }
  .syntaxhighlighter .diff.string, .syntaxhighlighter .diff.string a {
      color: green !important;
  }
}

ltrace + pthreads

2010-05-27T22:39:00.009-05:00

I was recently debugging some threading issues in MPICH2 and tried using ltrace to tell me the return code from pthread_mutex_lock and a few related functions. Tools like strace, ltrace, and valgrind are essential for debugging MPI programs because you can't always strap a debugger on them, and even when you can it tends to be a heavyweight procedure.

Unfortunately, when I did use ltrace I got a bunch of output that looked like this:

% mpiexec -n 1 ltrace ./examples/cpi |& grep pthread
pthread_mutex_init(0x810543c, 0, 0xbfe8d6c8, 0x80c44a9, 0x80eb4ef) = 0
pthread_mutex_init(0x810540c, 0, 0xbfe8d6c8, 0x80c44a9, 0x80eb4ef) = 0
pthread_mutex_init(0x8105424, 0, 0xbfe8d6c8, 0x80c44a9, 0x80eb4ef) = 0
pthread_self(0x8105424, 0, 0xbfe8d6c8, 0x80c44a9, 0x80eb4ef) = 0xb7da1ae0
pthread_mutex_init(0x81052a0, 0, 0, 0, 0xbfe8d6f4) = 0
pthread_mutex_lock(0x810540c, 0x80eb50d, 128, 0x80c44a9, 0x80eb4ef) = 0
pthread_mutex_unlock(0x810540c, 0xbfe8d73c, 0, 0xbfe8d6b4, 0xbfe8d6bc) = 0
pthread_mutex_destroy(0x810540c, 0, 0xbfe8d708, 0x80597bb, 0xb7eedff4) = 0

Of course, that doesn't make any sense. Everyone knows that pthread_mutex_lock only takes one argument, and pthread_self doesn't take any arguments. After a few minutes of studying /usr/include/pthread.h and /etc/ltrace.conf, I whipped up this output instead:

% mpiexec -n 1 ltrace ./examples/cpi |& grep pthread
pthread_mutex_init(0x810543c, NULL)              = 0
pthread_mutex_init(0x810540c, NULL)              = 0
pthread_mutex_init(0x8105424, NULL)              = 0
pthread_self()                                   = 3084118752
pthread_mutex_init(0x81052a0, NULL)              = 0
pthread_mutex_lock(0x810540c)                    = 0
pthread_mutex_unlock(0x810540c)                  = 0
pthread_mutex_destroy(0x810540c)                 = 0

The "secret" sauce goes in ~/.ltrace.conf:


; pthread.h
; - misc
ulong pthread_self(void);
int pthread_equal(ulong, ulong);
int pthread_create(addr, addr, addr, addr);
int pthread_join(ulong, addr);
int pthread_exit(addr);

; - mutex functions
int pthread_mutex_init(addr, addr);
int pthread_mutex_lock(addr);
int pthread_mutex_trylock(addr);
int pthread_mutex_unlock(addr);
int pthread_mutex_destroy(addr);

; - condition variable functions
int pthread_cond_init(addr, addr);
int pthread_cond_destroy(addr);
int pthread_cond_signal(addr);
int pthread_cond_broadcast(addr);
int pthread_cond_wait(addr, addr);

The system that I was on was a 32-bit Linux box running Ubuntu Hardy. For some reason, the --library option didn't work (I still got output from all libraries), which is why I have the grep bit above. If you are sure you only care about a particular function, the -e pthread_mutex_lock approach seemed to work OK. You may have to adjust this configuration snippet a bit in order to match your platform. In particular, the ulong types might change on 64-bit (I haven't checked though).

I'm not sure why pthreads wasn't supported out of the box by ltrace. In fact, given the limited range of supported types for ltrace, I'm not sure why this can't be almost completely automated. I think a few hours with pycparser and a little python scripting and you could generate 90% of the ltrace configuration.

cyclone update

2010-05-27T21:47:00.004-05:00

I re-read my recent post on my new dust collector recently and realized that I only wrote down about half of what I had in mind.

I only mentioned the Oneida system in comparison, but I should point out that I did look at a few other big names. I have a Penn State mini lathe, that I am generally happy with (post a comment if you want to hear my full opinion on it). So I did consider the PSI cyclones, in particular because they had a very attractive price. However, at the time they didn't have much in the way of technical detail. Also, the inlet size was 7 inches, which always concerned me when planning a 6-inch system. My understanding of air engineering is pretty weak, but a 7-inch pipe has 36% more area than a 6-inch pipe. So I think there's some translation needed when interpreting the CFM vs. static pressure loss plot for a 6-inch system.

The JDS machines seemed decent enough, but just too pricey for the same level of power.

And finally, a status update:

the filter clean-out box has been constructed and installed
the filter stack is attached, reusing the gasket from one of the filters between the clean-out and the stack
ducting is about half finished, including the transition from the cyclone to the main ceiling duct, and the vertical stack leading to the table saw
tool hoods still need to be constructed for all the tools (table saw, lathe, sander, drill press, plus a floor sweep)

Also, if you want a Clearvue cyclone of your own, you are too late (for now). The website currently has an announcement that they are trying to sell the business so that the main owner can retire. They've ceased sales for the time being.

dust collector installation

2010-04-25T21:44:00.003-05:00

The past couple of months have been pretty busy because we've been putting together papers for the Fall conference season. However, little by little I've been working on installing a new CV1800 cyclone dust collector in my garage shop (pardon the noisy iPhone photo):

Powering this thing is a serious 5 HP, US made, Leeson motor. It also has two large 9L300BL pleated cartridge filters from Wynn Environmental (approx. 3' tall x 1' diameter each). The goal with this purchase is to have a dust collector that I basically can't outgrow at the hobbyist level. I was having lots of trouble with wood dust, taking lots of the fun out of my hobby and worrying me about the long-term health issues.

Once again, I have to thank my father-in-law for some help with the planning and assembly on this project. He came over one Sunday and we assembled the bulk of the cyclone itself and roughed-in the electrical conduit feeding the cyclone. During a second (shorter) visit later on, we hung 3 10-foot pieces of 6-inch diameter SDR-35 sewer pipe for use as ducting.

Current status:

electrical - done
mounting bracket - built and attached to wall
motor and impeller - mounted
main cyclone body, blower housing, and transition - assembled, hung, and leveled
trash can - lid attached to cyclone, weather stripping under lid
filter stack - assembled but not attached to cyclone
filter clean-out box - to be built
ducting - WIP

Today I ran my first actual power-on test. I could only run it for a brief time (< 30 s) because w/o any ducting or filters attached the motor will draw very high current and runs some risk of burning up. It was much louder than I thought it would be, although it should be quieter with ducting and filters attached. I'm only planning on running it when I would normally wear ear protection anyway, so this shouldn't be too big of a deal. But if it's too loud for neighbors or others inside my house, I might need to look at some sort of muffling setup.

I went with the Clearvue instead of some of the other options out there after much research and deliberation. I went with it because I felt like it would deliver the best performance for a reasonable price and actually had the largest amount of trustworthy technical information available on the web. The system is setup to work with 6" pipe throughout the system, simplifying design and purchasing decisions. The Oneida product looked a little easier to assemble and very polished, but the website is short on details in some cases and more expensive for a roughly equivalent amount of power. Also, this comparison page was way too aggressive and painted an inaccurate picture of the Clearvue. Distorted information like that makes them look desperate, rather than letting a supposedly superior product stand for itself. Also, the CV06 patent dispute makes them look like bullies, although I am not familiar with any specifics of the case.

(FWIW, I've never seen an Oneida system in person, and I have no relationship with either company other than my sole purchase from Clearvue. This is just my personal opinion as a consumer in the market for this kind of product.)

The kit nature of the Clearvue meant a little bit more work, but it's been a fun project so far. I need to wrap it up soon though so that my garage can return to normalcy and I can get to work on actual woodworking projects. I also am planning on building an ambient air filtration device using a spare attic fan, some 1/2" plywood, and some other (rectangular) filters from Wynn Environmental.

Midnight Plane to Georgia

2010-04-18T00:01:00.003-05:00

Well, not really midnight at all. But I am packing for my ~11:45am flight to Georgia at around midnight tonight. On Friday I found out that my co-authors on our paper at CAC are stranded in Europe by the Icelandic volcanic ash. I was not originally planning on attending and Stéphanie was going to present the paper. But without anyone to present, I booked a flight on Friday and will be at CAC to present the paper instead.

The next few days are going to be pretty busy, between travel, presentation, and finishing up papers for the EuroMPI deadline on Tuesday morning.

(photo from NASA Goddard via Flickr)

Electrical Master Plan Phase 4 - Complete! (or the demise of mega-circuit)

2010-01-10T17:32:00.007-06:00

Today my father-in-law came over to help me with my electrical master plan and we had an extremely productive day. We finally managed to break up the circuit known as "mega-circuit" into 4 total pieces:

The rear exterior outlets and garden shed (including motion sensor light).
The freezer receptacle in the northwest corner of the kitchen.
The front exterior lights.
The remainder of mega-circuit, which includes all of the living room, one kitchen receptacle on the west wall, one kitchen light fixture (over the table), and the front hall lights.

This circuit was problematic for three reasons:

It covered a broad area of the house, both spatially and logically. Exterior lights and interior light should rarely affect each other, and a problem in the back yard shouldn't shut down the front hall light.
There are one or more subtle and infrequent issues somewhere on this circuit that were causing the breaker to trip occasionally. We currently suspect an issue somewhere in the exterior of the house (probably the front coach lamp line), but it's difficult to say for sure where it is.
The chest freezer was on this circuit. The aforementioned problem typically happened when we were on vacation, jeopardizing all the frozen food in our freezer.

We ran a 14/3 wire in the crawlspace from the breaker box to the northwest corner of the house and used that to power the freezer receptacle and the back yard lights/receptacles. It required some minor rewiring in the freezer receptacle and in one of the junction boxes of the crawlspace lights, but that was pretty easy. The worst part about that was actually pulling and stapling the wire all the way across the house because working in the crawlspace is hard on your knees and back. We lucked out that there was already a 14/2 wire running up from the crawlspace so that we didn't have to fish anything up to the box.

The front exterior lighting required a different trick. The garage interior lights (four fixtures) were three-way switched at the front and back of the garage. The exterior lights all run into the box in the front of the garage as well, but are fed from a switched line (on mega-circuit) coming from elsewhere in the house. This setup let us move the front exterior lights from mega-circuit to the garage light circuit by converting the garage light to a two-way switch, converting one of the travelers to a hot conductor, and then feeding that out to the remote switch before coming back to the exterior lights at the front of the garage. This was awesome because we didn't have to pull any new wire to make it happen, it just required reworking a few of the connections in the three boxes. Hopefully this is the last time (for a while anyway) that I'll need to mess around in the box by the front door, which is flimsy plastic and very crowded with a mix of 14AWG and 12AWG conductors.

Phase 3 should hopefully be done in the next few weeks. We have all the pipe, boxes, and receptacles for augmenting the electrical situation in the garage itself. I even have a radiant quartz heater from Lee Valley that we are going to install to take the edge off of the frigid Illinois winters. Phase 2 (splitting and replacing the tandem breakers) will get done whenever I get around to it, so maybe never ;)

It also sets me up for a new phase, phase 5, which is to add a circuit to support a new dust collector that I hope to buy and install this year.

Santa's workshop 2009

2010-01-03T17:16:00.003-06:00

For Xmas 2008, I received a midi-lathe. So for Xmas 2009 I used it to make several of my gifts to others. In the photo above you can see two pens in Honduran rosewood and some mystery wood (from the Woodcraft Classic American kit), a 4-in-1 screwdriver in maple (from the Rockler kit), a two screwdriver set in cocobolo, and a 6-inch pepper mill in walnut (from a Woodcraft kit). I also made a matching salt shaker for the walnut pepper mill, a 4-inch pepper mill and salt shaker combo in jobillo, and a pair of chrome wine stoppers in maple and European pear (from this Woodcraft kit).

The screwdriver set is unique because it is probably the only cocobolo project I'll ever turn. Many people experience varying levels of allergic reaction to the wood dust from cocobolo, and I'm no exception. Luckily my allergies only result in a mild skin rash that fades after a day, but I don't need to take any risks for a hobby like this. Plus allergic reactions like that usually increase in severity with repeated exposure.

The shanks on the two screwdrivers are 6-inch screwdriver bits from Lee Valley. For some reason, they only sell Phillips-head and square-head bits in that length, so the slotted end in the photo is actually the back end of a square drive bit that has been ground down on the bench grinder. The copper ferrules come from a single 3/4" copper pipe coupling that I cut in half. To clean up the rough-sawn edges and polish up the outside of the ferrules, I made a mandrel that fits in my scroll chuck. I don't have a photo of it handy right now, so I'll just try to describe it. It has a tenon whose diameter tapers from slightly smaller than the inner diameter of the ferrule to slightly larger at the headstock. I cut an "X" in the end of the tenon with a handsaw, then tapped the rough ferrule on the tenon with a mallet, and wedged the tenon tight by hammering in a small Phillips head bit at the center of the "X". Then I turned the edge smooth with a round nose scraper and sanded the outside of the ferrule with various grits of sandpaper to clean up the finish.

Additional photos can be found on our smugmug page.

If anyone wants more details on how I turned any of these items, please let me know in the comments. I'm happy to elaborate.

Samsung DLP and the "white dots" issue

2010-01-03T16:51:00.003-06:00

When Heather and I got married back in October 2006, some of our friends gave us a very generous wedding gift: a Samsung 46-inch DLP high-definition television. We use it pretty much every day that we are at home (it's even on in the background while I write this post). Unfortunately, early October the TV started having a problem. It developed a small white dot in the upper left hand part of the screen. Then a second dot, and a third, and so on. Over the course of the following two months the TV screen developed dots on a substantial fraction of the screen, as you can see from the picture below:

Googling turned up this forum thread, which basically said that this was not an uncommon issue with Samsung DLPs of this era. Unfortunately, back in early October the latest update on the issue from Samsung was that this was a "normal" dead pixel issue, not any particular manufacturing defect, and that we would need to pay a repair shop to fix our out-of-warranty TV. A call to Samsung's support line yielded the same story. The forum thread suggested replacing the DLP chip yourself for about $150-200, and a few hours of figuring out how to actually perform the surgery on the TV. I nearly went down this road in late October, but that time of year is the travel season for me. So I punted until the end of November when all of my travel was finished.

Well, come late November it turns out that Samsung came around and initiated a recall program for their DLP TVs with this issue. A different forum explained the procedure, which was really simple. Basically just call Samsung and tell them that you are having these symptoms, and they will send out a technician from a nearby authorized service center to replace the DLP chip at your house.

So I called Samsung, told them my problem, and they said that someone would get in touch with me within a couple of days. The ASC called me the next day to schedule an appointment. We had to wait 4 or 5 days for them to get the part in, but otherwise it was painless. The technician showed up at our house on time and replaced the part in about 30 minutes. I watched for part of the time while he disassembled/reassembled the TV and I was glad that he was there to do it. I'm pretty sure that I could have done it, but it would have been pretty nerve wracking and would have taken me a few hours at a minimum.

Overall, I have to applaud Samsung for owning up to the problem and fixing it at no cost to us. They could have been a little faster and proactive (I nearly replaced the part myself), but the eventual outcome was exactly what it should have been. If things had gone differently, I might have sworn off Samsung products forever instead.

MPI_C_BOOL officially 1 byte in external32 representation

2010-01-03T00:44:00.000-06:00

A very minor announcement, the errata ticket for the size of MPI_C_BOOL in the external32 representation has been passed:

https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/171

MPI-2.2 Is Now Official

2009-09-04T14:52:00.007-05:00

Earlier today (~11am GMT+3) the MPI Forum ratified the final PDF of the MPI Standard Version 2.2. Other than a "No" vote from NEC the vote was unanimous in favor of accepting it. As I mentioned before this version of the MPI Standard contains many clarifications and corrections as well as several new features. The Change-Log (Annex B) contains a concise list of the relevant functionality changes.

Bill Gropp (the MPI-2.2 chair) is currently out of internet range, so the official document isn't yet posted on the normal MPI Documents website. In the mean time I've posted an unofficial version here that you can download. I imagine that the official document should be up by Monday.

(Update (2009-09-05): The official MPI-2.2 document has been posted.)

MPICH2 should support these new clarifications and features in a release before the end of October (possibly sooner). All but one of them (ticket #33, the MPI_Dist_graph_* functions) are implemented already on the SVN trunk if you are so inclined to play with MPI-2.2.

Off To Helsinki

2009-08-30T20:42:00.010-05:00

Tomorrow I'm headed to Helsinki, Finland for a couple of weeks to attend the next MPI Forum meeting and EuroPVM/MPI 2009. Although I'm unsure how productive the Forum meeting will be because many participants cannot attend this particular meeting, it should be a noteworthy meeting. Barring any unforeseen problems we should ratify the new version of the MPI Standard, MPI-2.2. This version of the document corrects many problems with the 2.1 Standard and adds several new features. I could list them all here but Torsten has done a pretty good job of it already, so I'll send you over to his post on the matter.

Most of my MPI Forum work since the 2.1 effort ended has been on ticket #18, which adds predefined named MPI datatypes for many C99 types like int32_t, _Bool, and double _Complex. The lesson to be learned from this ticket is to always separate sufficiently separable changes, even if they are very related. There were several points at which this ticket was nearly derailed and postponed until MPI-3.0. The voting process used by the MPI Forum has only a single path by which tickets can make it into the Standard but many ways they can be removed from consideration. Everything is fine with the ticket now, but if I had separated it into the three logically distinct changes that it makes (fixed-width integers, C99 complex types, and MPI_Aint/Offset types) there would have been a better chance of getting something rather than nothing in for MPI-2.2.

My understanding is that Rolf will be printing up hardbound copies of the MPI-2.2 standard for purchase at SC09, much like he did with the MPI-2.1 standard. It makes for a handy reference when debating standardization issues or implementing the Standard, so I'm planning on buying a copy. It's also one of the few dead-tree publications to actually have my name in it somewhere, so that doesn't hurt either ;)

EuroPVM/MPI 2009 should also be enjoyable, since it is a good opportunity to get together with others in the fairly small world of MPI implementation development. I have a paper there (together with several other authors) entitled "MPI on a Million Processors" that explores scalability issues inherent in the MPI Standard. I'll post a copy online somewhere after the conference.

If I feel especially motivated I might post some updates and photos from the trip here. We'll see how busy I am at the conferences and how bad the jet lag hits me...

-fstack-protector + valgrind ==> stack array overflow debugging?

2009-07-15T07:45:00.012-05:00

stack array overflow background

I've recently been playing with valgrind to help find/debug stack overflow bugs in MPICH2. That is, places where someone made a programming mistake and scribbled past the end of an array that lives on the stack, such as in this MPICH2 ticket (and associated fix). I'm not really concerned about finding malicious buffer-overflow attacks in this post, more just the general program bugs from stack array overflows.

Of course, valgrind won't normally detect these sorts of errors, because whenever the stack pointer is moved down (that is, the stack grows) everything between the old SP position and the new SP position is marked as addressable. This is entirely reasonable, since this memory is actually addressable and it means that valgrind works on code that doesn't perfectly follow a particular calling convention, such as that generated by an experimental/broken compiler or from hand-written assembly.

In case you don't feel like trying to grok the changes I referenced above, here's a simple program that has a similar problem that should illustrate what's going on.


1 static int foo(void)
2 {
3     int a = 1;
4     char b[32];
5
6     printf("before memset\n");
7     memset(b, 0, 64); /* stomps the stack */
8     printf("after memset\n");
9
10     return 0;
11 }
12
13 int main(int argc, char **argv)
14 {
15     int x;
16
17     printf("before foo()\n");
18     x = foo();
19     printf("after foo()\n");
20     return 0;
21 }
22

Line 7 is the line that causes the badness here. Most of you at home know why immediately, but I'll write it out just in case so that the main topic of this post is clear to everyone. When main() calls foo(), the stack ends up looking like this (as of line 5). Earlier lines in the diagram are higher addresses (the stack grows down on most architectures and operating systems).


- last address in caller's stack frame
- any arguments in reverse order (none in the case of foo)
- return address                       <-- frame pointer register (EBP) points here
- old frame pointer value
- storage for "int a" (4 bytes)
- storage for "char b[32]" (32 bytes)  <-- ESP points here

This is an x86 cdecl calling-convention view of the world, but besides possibly the frame pointer and the int size, things will basically look this way on most systems. The important thing to observe here is that the storage for b actually has a lower address than both the storage for a and the storage for the return address for the function. The problem with our sample code is that the memset operation on line 7 sets all of b to zero (expected), but keeps on going and clobbers the values for a, old frame pointer, the return address, and probably part of the caller's stack frame.

When the RET instruction at line 10 is executed, the processor will pop the return address off of the stack and jump there. In our case, that value isn't correct anymore, it's 0 instead and will result in a general protection fault. And of course, if we had logic in foo() that depended on the value of a making sense then we might have had some other very strange behavior manifest instead. And the worst part is that valgrind won't catch this for you. You'll get the crash just the same as if you had run int outside of valgrind. In a large-ish code base this can be tricky to track down sometimes, and having a tool catch the problem for you would be really handy.

enter the MPICH2 Stack Guard macros

So, I've implemented some macros that can help catch this:


#define MPIU_SG_GUARD_VALUE_ (0xdeadbeef)
#define MPIU_SG_FIRST_DECL int MPIU_SG_upper_guard = MPIU_SG_GUARD_VALUE_
#define MPIU_SG_LAST_DECL \
    int MPIU_SG_upper_guard_blk_handle; \
    do { \
        MPIU_SG_upper_guard_blk_handle = VALGRIND_CREATE_BLOCK(&MPIU_SG_upper_guard, sizeof(MPIU_SG_upper_guard), "stack guard variable"); \
        VALGRIND_MAKE_MEM_NOACCESS(&MPIU_SG_upper_guard, sizeof(MPIU_SG_upper_guard)); \
    } while (0)
#define MPIU_SG_AT_RETURN() \
    do { \
        VALGRIND_MAKE_MEM_DEFINED(&MPIU_SG_upper_guard, sizeof(MPIU_SG_upper_guard)); \
        VALGRIND_DISCARD(MPIU_SG_upper_guard_blk_handle); \
        MPIU_Assert(MPIU_SG_upper_guard == MPIU_SG_GUARD_VALUE_); \
    } while (0)

And here's an updated version of our foo() function:


  1 static int foo(void)
  2 {
  3     MPIU_SG_FIRST_DECL;
  4     int a = 1;
  5     char b[32];
  6     MPIU_SG_LAST_DECL;
  7 
  8     printf("before memset\n");
  9     memset(b, 0, 64); /* stomps the stack */
 10     printf("after memset\n");
 11    
 12     MPIU_SG_AT_RETURN();
 13     return 0;
 14 }

If someone overflows the b array by a sufficient amount, they will end up scribbling over the guard variable. The various VALGRIND_ macros tell valgrind that the guard variable shouldn't be touched during the main body of the function. Furthermore, we check the value via the MPIU_Assert to see if it has been scribbled over by some bad code in the function so that we get failures even when we aren't running under valgrind. So running this updated code under valgrind will give you a warning message indicating the line where the bad access occurred, rather than having to try to work backwards from the GPF that you get at return time or from any wonky behavior that was predicated on a corrupted value of a:


% valgrind -q ./foo
before foo()
before memset
==49581== Invalid write of size 4
==49581==    at 0x1E2B: foo (foo.c:36)
==49581==    by 0x1F68: main (foo.c:48)
==49581==  Address 0xbfffedd4 is 0 bytes inside a stack guard variable of size 4 client-defined
==49581==    at 0x1DB2: foo (foo.c:33)
==49581==    by 0x1F68: main (foo.c:48)
after memset
Assertion failed: (MPIU_SG_upper_guard == (0xdeadbeef)), function foo, file foo.c, line 39.

This technique is a little too ugly and cumbersome for blanket use in the code base. However if you are poking around with a debugger or printf() and you see memory corruption that valgrind can't seem to find, you might be able to insert these macros into any suspicious functions and find your bug rather quickly. It also will only flag an error when the array access overflows far enough to clobber the guard. In our example valgrind won't flag an overflow of 4 bytes or less because it will corrupt a but not the guard variable. Intermediate guards can help with this if you have strong suspicions about a particular function, but it takes a slightly unwieldy tool and makes it even more cumbersome.

I can't possibly be the first person to do this, but I haven't seen it anywhere else yet to cite as another example of this. Please add links in the comments if you know of of something/someone else that does this.

stack smashing protection

Various techniques have been developed over the years to protect against malicious buffer-overflow attacks, which are very similar to this type of bug. One of them is the Stack Smashing Protector for GCC (the -fstack-protector option). This technique basically involves a modification to the compiler to insert a guard variable similar to the one above on any function that allocates an array on the stack. Then prior to any return from the function the guard value is checked to ensure that it hasn't been modified. This prevents malicious code from overwriting the return address and hijacking execution on function return, which is the standard approach for a buffer-overflow attack.

All you have to do to get this protection from a modern version of GCC is to compile with -fstack-protector. I believe that there is a slight performance penalty associated with enabling this feature, but it's probably a good trade-off for network-facing software.

valgrind and -fstack-protector: two great flavors that could taste better together

Of course, everything earlier was just setup for the main point of this post. In my mind these two techniques are begging to be combined in some way. I'm not familiar enough yet with the details of the -fstack-protector implementation to know how technically feasible/difficult this is. I know that the exact mechanics of the guard depend heavily on OS support and the architecture's capabilities. For example, the availability of thread-local-storage (TLS) changes the implementation. On top of that, I'm not familiar enough with any of the debugging symbol information (like DWARF) to know how suitable it is for annotating information about the guard variable. Maybe this would be a good opportunity to play with LLVM a bit more, since I have used that a little in the past.

Anyway, if someone (maybe me) could work out the technical issues, this could be a really handy tool for debugging in some situations. If you suspected stack corruption issues at all, you would just need to recompile with -fstack-protector (if not already so-compiled) and maybe the currently-non-existent -fstack-protector-annotate-for-valgrind and then run valgrind to help locate your bug.

Updated (2009-06-16):
There's a relevant GCC feature that I just learned about called _FORTIFY_SOURCE. I haven't had the time yet to check whether or not this would have caught our actual MPICH2 bug (I don't think it would have, unfortunately), but it definitely catches the error in the sample code above. As written, the it catches at compile time even, which is far better than waiting until you experience a bug to run valgrind on it, although the sample code is admittedly trivial.

handy "git log" options and the nowrap script

2009-06-28T16:26:00.011-05:00

I recently settled on two sets of options to "git log" that I greatly prefer over the defaults. The first is --pretty=tformat:"%C(yellow)%h%Creset %Cblue%aN%Creset %Cred%d%Creset %s". This is basically an improved version of --pretty=oneline that gives output that looks like the following:

I won't go into the details of that format string, the git-log(1) manpage ought to be sufficient to pick it apart. If something seems confusing about it, contact me and I can explain it further. You can put that in your ~/.gitconfig file as format.pretty=tformat:"..." to make git log use that style by default.

That, combined with another cool feature of git makes for the other git-log style that I find myself using often. git log --graph --decorate --abbrev-commit --all --pretty="tformat:.../oneline" is the other cool command. It looks like this:

It provides a pretty good approximation of the commit trees that programs like GitX and gitk show you, only on the command line instead. Obviously the --pretty part above can be whatever you want, although I usually prefer either "oneline" or the custom tformat string mentioned earlier. I also alias it in my .gitconfig to "git graph", so that I don't have to type all that in.

Since I work almost exclusively with git-svn repositories where the short subject with a long message style of commit messages isn't the standard practice, both of these single line formats can be hard to read. Many of the commit subjects from SVN will wrap, even in a fairly wide (230 character) terminal. To work around this problem, especially when viewing graph output, I've written a small perl script that will truncate output at the width of the window (or any other specified width). I know that you are thinking "why not just use cut -c1-$COLUMNS?" but that won't work very well with output that has color escape sequences in it. The program is called "nowrap" and is available here on github.

By comparison, here's pretty much the same git graph command as before but with wrapping disabled:

otx: Objective-C disassembly

2009-05-26T21:54:00.011-05:00

Let's say that you have a closed-source Objective-C program on your Mac that needs a minor modification. Maybe the company is delayed in sending you the license file that you purchased, or the company went out of business and there's no way that the code will ever get fixed.

I'm not very sophisticated at this sort of thing, but I've twiddled a few bits here and there in the past. Usually simple stuff like replacing a set of instructions with no-ops or altering a constant in a move or arithmetic instruction. However, it was always in regular C programs, so the standard GNU binutils like nm and objdump, together with xxd, have been sufficient to get an idea of what was going on in the program. Unfortunately, these tools aren't sufficient by themselves for an Objective-C program.

Here is an excerpt from the normal objdump -d from the binary that lives inside of GitX:


4018:       89 74 24 04             mov    %esi,0x4(%esp)
401c:       89 04 24                mov    %eax,(%esp)
401f:       e8 88 21 02 00          call   261ac <.objc_class_name_PBNSURLPathUserDefaultsTransfomer+0x3b0c>
4024:       83 c4 20                add    $0x20,%esp
4027:       31 c0                   xor    %eax,%eax
4029:       5b                      pop    %ebx
402a:       5e                      pop    %esi
402b:       c9                      leave
402c:       c3                      ret
402d:       55                      push   %ebp
402e:       89 e5                   mov    %esp,%ebp
4030:       83 ec 38                sub    $0x38,%esp
4033:       89 5d f4                mov    %ebx,0xfffffff4(%ebp)
4036:       89 75 f8                mov    %esi,0xfffffff8(%ebp)
4039:       8b 75 10                mov    0x10(%ebp),%esi
403c:       89 7d fc                mov    %edi,0xfffffffc(%ebp)
403f:       8b 3d 30 1b 02 00       mov    0x21b30,%edi
4045:       c7 44 24 10 00 00 00    movl   $0x0,0x10(%esp)

Here is the same range of bytes interpreted by otx (otx -e < /Applications/GitX.app/Contents/MacOS/GitX):

  +100  00004018  89742404                movl        %esi,0x04(%esp)
+104  0000401c  890424                  movl        %eax,(%esp)
+107  0000401f  e888210200              calll       0x000261ac                    _objc_assign_strongCast
+112  00004024  83c420                  addl        $0x20,%esp
+115  00004027  31c0                    xorl        %eax,%eax
+117  00004029  5b                      popl        %ebx
+118  0000402a  5e                      popl        %esi
+119  0000402b  c9                      leave
+120  0000402c  c3                      ret

+(BOOL)[PBGitRepository isBareRepository:]
+0  0000402d  55                      pushl       %ebp
+1  0000402e  89e5                    movl        %esp,%ebp
+3  00004030  83ec38                  subl        $0x38,%esp
+6  00004033  895df4                  movl        %ebx,0xf4(%ebp)
+9  00004036  8975f8                  movl        %esi,0xf8(%ebp)
+12  00004039  8b7510                  movl        0x10(%ebp),%esi
+15  0000403c  897dfc                  movl        %edi,0xfc(%ebp)
+18  0000403f  8b3d301b0200            movl        0x00021b30,%edi               PBEasyPipe
+24  00004045  c744241000000000        movl        $0x00000000,0x10(%esp)

As you can see, it makes a big difference in terms of readability. otx decodes the function names and boundaries from the text and decodes the function calls whenever possible. The dynamic dispatch makes it tricky to read a raw disassembly otherwise.

So you just take the byte offsets of the instructions you want to replace and write the appropriate opcodes in place of them with your hex editor. You can write up a little bit of assembly and assemble it with as and then disassemble that if you don't know the opcodes for the replacement instructions off the top of your head. On x86/x86_64, 0x90 is the opcode for nop, which is the most common instruction that I use as filler. The filler is pretty important, since you can't easily change the byte offsets of any of the other instructions or else your jump locations and possibly your variable references will be wrong.

minor tab sweep

2009-05-18T22:58:00.003-05:00

A little while back I posted about some SVN permissions problems that we had at work. While cleaning up my many open tabs today I came across this page that I read when trying to figure out what was going on. It pretty well describes what we ran in to, which is yet another one of the many lousy SVN shortcomings. We basically went with the fourth solution as our fix.

helpful options for less

2009-05-18T22:02:00.011-05:00

"less" the unix command, that is. The default behavior for less isn't quite as good as it could be. I use a case insensitive search (actually 'smartcase') in vim, but less is case sensitive by default. You can pass "-i" on the command line or once less is started to get smartcase-like behavior in searches. Or you can put that in the $LESS variable in your .zshenv. My .zshenv has the following:


export LESS='-R -i --quit-if-one-screen'

From the less(1) manpage:


       -F or --quit-if-one-screen
              Causes less to automatically exit if the entire file  can  be  dis-
              played on the first screen.
...
       -i or --ignore-case
              Causes  searches  to  ignore case; that is, uppercase and lowercase
              are considered identical.  This option is ignored if any  uppercase
              letters  appear in the search pattern; in other words, if a pattern
              contains uppercase letters, then that search does not ignore  case.
...
       -R or --RAW-CONTROL-CHARS
              Like  -r,  but  tries  to keep track of the screen appearance where
              possible.  This works only if the input consists of normal text and
              possibly some ANSI "color" escape sequences, which are sequences of
              the form:

                   ESC [ ... m

              where the "..." is zero or more characters other than "m".  For the
              purpose  of keeping track of screen appearance, all control charac-
              ters and all ANSI color escape sequences are assumed  to  not  move
              the cursor.  You can make less think that characters other than "m"
              can end ANSI color escape  sequences  by  setting  the  environment
              variable LESSANSIENDCHARS to the list of characters which can end a
              color escape sequence.

The other two options are primarily git related. The "-R" option is so that the wonderful colored output from various git commands gets displayed properly. The "--quit-if-one-screen" option prevents bits of short output from commands like "git log -1" from interrupting the normal flow of work. On some terminals you might need to pass "-X" in order to prevent the "--quit-if-one-screen" and the terminal's alternate screen from hiding any output that takes up less than one screen.

Also, to echo Andrew, never use "more", there's absolutely no point on any system where "less" is available. Seriously, just don't do it.

Open Portable Atomics v1.0.0 released

2009-05-12T22:02:00.002-05:00

We just released the first official release of the OpenPA project, v1.0.0. If you ever are thinking about writing inline assembly to perform atomic operations (compare-and-swap, fetch-and-increment, etc), then think about using OPA instead. It provides a consistent interface to primitive atomic operations via C functions and macros across multiple compilers and architectures. We have a solid range of platforms at this point, but we intend to add more in the future and we are always willing to accept contributions if you have a particular platform you would like to support.

You can download openpa-1.0.0 here if you are interested in playing around with it.