semicomplete.com - Jordan Sissel

new xdotool version available (20090612)

Sat, 13 Jun 2009 06:26 GMT

Hop on over to the xdotool project page and download the new version.

The changelist from the previous announced release is as follows:

20090612:
  * Fixed bug where shift modifier was not reset when 'xdotool type' used.
  http://code.google.com/p/semicomplete/issues/detail?id=5

new xdotool version available (20090609)

Tue, 09 Jun 2009 07:01 GMT

Hop on over to the xdotool project page and download the new version.

The changelist from the previous announced release is as follows:

20090609:
  * Add '--delay ' to xdotool type. Sets the delay between keys.
  * Add '--window ' to xdotool type, key, keyup, and keydown.
    This feature (key events with --window ) only works if the
    application does not reject X events that have 'send_event' set to true.

    Special notes:
    * Firefox appears to ignore all input when it does not have focus.
    * xterm ignores sendevent by default, ctrl+leftclick menu will let you
      enable it.
    * gnome-terminal appears to accept send_event input by default

XSendEvent + LD_PRELOAD == win

Sun, 07 Jun 2009 05:28 GMT

As far as feature requests come, for xdotool, one of the more common ones is to have the ability to send key or mouse events to a specific window, not just the active one. XTEST (what xdotool uses for key/mouse currently) doesn't let you specify a window to send events. XSendEvent(3) lets you send hand-crafted events to a specific window, but most applications ignore these sent events.

The XEvent struct has a member 'send_event' which is true if the event came from an XSendEvent call and false otherwise. Programs like firefox and xterm (by default) ignore many events that have 'send_event' set to true.

Enter LD_PRELOAD.

Writing a custom shared library that overrides the default XNextEvent and XPeekEvent functions allows us to force 'send_event' to always be false, so an application with this library loaded will happily handle keyboard/mouse events generated with XSendEvent. I already have a helpful project that lets me write such a shared library: liboverride.

#include <stdio.h>
#include <X11/Xlib.h>

void hack_send_event(XEvent *ev) {
  switch (ev->type) {
    case KeyPress: case KeyRelease: case ButtonPress: case ButtonRelease:
      ev->xany.send_event = False;
      break;
  }
}

override(`XNextEvent', `
  {
    real_func(display, event_return);
    hack_send_event(event_return);
    return;
  }
')

This small bit of liboverride code will give me a shared library I can preload with LD_PRELOAD. Doing so will ensure that send_event is false for any key or mouse button events.

Works well. Now that we have a reliable way to allow XSendEvent I think it's worth putting this into xdotool.

Ruby Net::IMAP and Exchange

Tue, 26 May 2009 08:03 GMT

Exchange's server-side filters are pretty weak, so I decided to work around them by writing a tool that will fix my inbox and filter mail appropriately so that any client I use to view mail with (OWA, whatever) has the same view with no client-local filters. It's likely/possible there's already a tool that does this; let's ignore that possibility for now.

Ruby comes with Net::IMAP, but it doesn't come with an authenticator that supports 'PLAIN' auth, so we have to provide one:

# Learned the 'PLAIN' expected format from imapsync.
class PlainAuthenticator
  def process(data)
    # Net::IMAP takes care of base64 encoding the result of this...
    return "#{@user}\0#{@user}\0#{@password}"
  end
  
  def initialize(user, password)
    @user = user
    @password = password
  end
end

Net::IMAP::add_authenticator('PLAIN', PlainAuthenticator)

Now that we have that, let's try connecting.

imap = Net::IMAP.new("exchange.example.com", "imaps", usessl=true)
imap.authenticate("PLAIN", user, passwd)

This fails, because Exchange's IMAP server ignores the RFC:

/usr/lib/ruby/1.8/net/imap.rb:3122:in `parse_error': unexpected token CRLF (expected SPACE) (Net::IMAP::ResponseParseError)
        from /usr/lib/ruby/1.8/net/imap.rb:2974:in `match'
        from /usr/lib/ruby/1.8/net/imap.rb:1959:in `continue_req'
        from /usr/lib/ruby/1.8/net/imap.rb:1946:in `response'
...

Expected a space, not a crlf. The failure is in continue_req, which expects what the RFC says:

continue_req    ::= "+" SPACE (resp_text / base64)

However, Exchange's IMAP server doesn't send a space after the plus. Great, let's fix that by overriding the continue_req method:

# Copied/modified from net/imap.rb, don't modify that file, put this
# in your own code to override the continue_req method
module Net
  class IMAP
    class ResponseParser
      def continue_req
        match(T_PLUS)
        #match(T_SPACE)   # Comment this line out to not expect a space.
        return ContinuationRequest.new(resp_text, @str)
      end
    end
  end
end

Once you've done that, everything else seems to work normally. I have only tested listing mail folders thus far, but the hacks above allow you to get this far.

More rmagick/rvg playtime

Wed, 20 May 2009 09:27 GMT

While working on graphs tonight, I found that calculation and labelling of ticks should be provided by special 'tick' classes. The iterator (the 'each' method) takes a min and max value and yields ticks in that range. This allows you to:

A 'tick' provider should just be an iterable class (foo.each, etc) which provides the tick position and optional label.
A graph can have multiple ticks per axis, allowing you to have 'major' ticks labeled while 'minor' ticks are not labeled, and even more than two layers of ticks on each axis.
The same tick classes can easily be used to draw both the graph ticks and the grid.
Trivially have 'time format' tickers
Have a 'smart time ticker' that looks at the min/max and determines the correct set of time ticks to display (display format, tick distance, tick alignment, etc). Can use multiple 'time ticker' instances internally (code reuse!)

I'm sure this has all been though of before, but it's a research experience for me :)

At any rate, I'm finding myself wondering if RMagick/rvg is really the right tool. It certainly makes doing graphics trivial, but even for what I see as a simple graph it takes a little over a second to render, which would hurt usability if multiple graphs needed rendering simultaneously.

The bottleneck seems to be with text rendering. If I disable text display in the graph (tick labels, etc), graph rendering drops by 0.5 seconds (from 1.1). Switching from 'gif' to 'png' output shaved 0.2 seconds on average of rendering, which is interesting.

Today's results, with real data:

graph = RPlot::Graph.new(400, 200, "Ping results for www.google.com")
pingsource = RPlot::ArrayDataSource.new
File.open("/b/pingdata").each do |line|
  time,latency = line.split
  pingsource.points << [time.to_f, latency.to_f]
end
pingsource.points = pingsource.points[-300..-1]
graph.sources << pingsource
graph.xtickers << RPlot::SmartTimeTicker.new
graph.ytickers << RPlot::LabeledTicker.new(alignment=0, step=25)
graph.render("test.png")

Graphs in Ruby with RMagick

Tue, 19 May 2009 07:11 GMT

I'm always finding myself wanting to graph random data. Gnuplot is nice, but not enjoyably scriptable. Matplotlib in python is too matlab-ish, or was when I looked at it last (though it looks much improved now). Some ruby options exist (even ruby+gnuplot), but none were much to my tastes.

I started fiddling around with RMagick and stumbled across what it calls "RVG" (ruby vector graphics). From the site:

RVG (Ruby Vector Graphics) is a facade for RMagick's Draw class that supplies a drawing API based on the Scalable Vector Graphics W3C recommendation.

The API is pretty reasonable and hasn't hindered me yet and feels good after having hacked with it for a few hours: Simple operations like point translation, scaling, rotating, flipping, etc are simple in code; the api is well documented; images can be embedded easily into another which allows for easily writing modular code.

Anyway, the goal of this adventure was to come up with something that would produce non-crappy plots. Main emphasis on having a means to apply axis labels and ticks that wasn't painful. The result is below: (x-axis ticks are hour-aligned and have 12 hour steps, y-axis ticks are single-value aligned)

Here's the code that generates the above graph (using rplot.rb). A lot of things (like axis label tick alignment and stepping) are hardcoded right now, but that will obviously change if I decide this project needs attention (and I don't find something that does the same thing but better).

# graph some random stuff, like log(x) and sin(x)
# use time for the 'x' to demo time formatting
# each point is an hour (i * 3600)
graph = RPlot.new(400, 200, "Happy Graph")

points = 60
axis = GraphAxis.new
(1..points).each do |i|
  axis.points << [Time.now.to_f + i*3600, Math.log(i)]
end

axis2 = GraphAxis.new
(1..points).each do |i|
  axis2.points << [Time.now.to_f + i*3600, Math.sin((i / 2.0).to_f) + 1]
end

graph.axes << axis
graph.axes << axis2

graph.render("/home/jls/public_html/test.gif")

jps output not correct

Sun, 17 May 2009 03:46 GMT

A nagios alert checking for some java processes started firing because it couldn't find those processes. This check used 'jps' to look for those processes.

% sudo /usr/java/jdk1.6.0_04/bin/jps
15071 Jps
% ps -u root | grep -c java
15

I espected lots of output from jps, but there was only the jps process itself. Confusing. What does jps use to track java processes?

Your old strace (truss, whatever) friend will help you here:

# Always use 'strace -f' on java processes as they spawn new processes/threads
% sudo strace -f /usr/java/jdk1.6.0_04/bin/jps |& grep -F '("/' \
  | fex '"2/{1:2}' | sort | uniq -c | sort -n | tail -5
      5 proc/self 
      5 proc/stat 
     12 usr 
     17 tmp/hsperfdata_root 
    283 usr/java

It referenced /tmp/hsperfdata_root multiple times. Weird, checking it out:

% ls /tmp/hsperfdata_root | wc -l
0

This directory is empty. Looking further around the strace and confirming by looking at the classes jps invokes (sun.jvmstat.perfdata.monitor.protocol.local.MonitoredHostProvider) shows that /tmp/hsperfdata_<user> is used by each jvm instance. It stores a file named by processes' pid.

So the question is, why is this directory empty?

Of the hosts I know run java, it only seems like long-running instances of java are disappearing from jps, making me think we have a cron job removing files from /tmp. I found this while looking through cron jobs:

% cat /etc/cron.daily/tmpwatch 
/usr/sbin/tmpwatch -x /tmp/.X11-unix -x /tmp/.XIM-unix -x /tmp/.font-unix \
        -x /tmp/.ICE-unix -x /tmp/.Test-unix 240 /tmp
/usr/sbin/tmpwatch 720 /var/tmp
for d in /var/{cache/man,catman}/{cat?,X11R6/cat?,local/cat?}; do
    if [ -d "$d" ]; then
        /usr/sbin/tmpwatch -f 720 "$d"
    fi
done

This file comes from the tmpwatch rpm, which appears to come base installed on CentOS. This means that for every file in /tmp (except those specified by '-x dir') are being deleted if they are older than 240 hours (10 days). As an FYI, the default time value inspected is the file's atime, so if you mount noatime, the accesstime is not reliable.

Ultimately, we need to add a new set of flags to the cronjob that excludes /tmp/hsperfdata_*. This should keep me from being paged when a java process lives for more than 10 days ;)

Additionally, it makes me think that the people who use CentOS don't use Java or don't monitor their java processes with jps.

Shebang (#!) fix.

Wed, 01 Apr 2009 07:40 GMT

Most shebang implementations seem to behave contrary to my expectations.

As an example, prior to today, I would have expected the following script to output 'debug: true'

#!/usr/bin/env ruby -d
puts "debug: #{$DEBUG}"

Running it, I get this:

% ./test.rb
/usr/bin/env: ruby -d: No such file or directory

This is because the 'program' executed is '/usr/bin/env' and the first argument passed is 'ruby -d', exactly as if you had run: /usr/bin/env "ruby -d"

My expectation was that the above would behave exactly like this:

% /usr/bin/env ruby -d test.rb
debug: true

It doesn't. The workaround, however, is pretty trivial. It's only a few lines of C to get me a program that works as I want. I call the program 'shebang'. Why is it C and not a script? Because most platforms have a requirement that the program being executed from the shebang line be a binary, not another script.

#!/usr/local/bin/shebang ruby -d
puts "debug: #{$DEBUG}"

Now run the script again, with our new shebang line:

% ./test.rb
debug: true

Simple and works perfectly.

Where have I been?

Wed, 25 Mar 2009 17:22 GMT

I've been pretty silent on this site since January. This is because I've been busy at work and haven't had the time and energy to throw into writing here, unfortunately. Though, each time I think about my lack of recent writing, I remind myself that I 'm still well over average for posting since writing 23 of the 25 articles on sysadvent.

Fortunately, I can now open up about the company I work for since we surfaced two days ago. I work for OnLive.

As things calm down I should be able to dump more time into writing here.

xdotool 20090126 available

Mon, 26 Jan 2009 11:15 GMT

A new release of xdotool is available. It fixes two bugs that were moderately annoying.

20090126:
  * Change the default behavior of 'getwindowfocus' to get the first
    ancestor-or-self window that has WM_CLASS set. WM_CLASS will be set on
    (all?) top-level windows and it's possible that the currently focused window
    according to X is not a top-level window. To use the previous behavior, use
    'getwindowfocus -f'
  * Make 'xdotool key at' work correctly. 'at' is usually Shift+2, for example.
    Now all shifted characters should work, but I've only tested on a US
    keyboard.
  * Minor Makefile fixes for package maintainers.

I've tested it in GNOME and Ion-3 with success.

Download: xdotool-20090126.tar.gz