The Wallaby git repository now contains a command to export versioned configurations from a database to flat text files. Clone the repository or just download the file and then place cmd_versioned_config_export.rb somewhere in your WALLABY_SHELL_COMMAND_PATH, and you’ll be able to invoke it like this:

Usage:  wallaby vc-export DBFILE
exports versioned configurations from DBFILE to plain text files
    -h, --help                       displays this message
        --verbose                    outputs information about command progress
    -o, --output-dir DIR             set output directory to DIR
        --earliest NUM               output only the earliest NUM configurations
        --latest NUM                 output only the most recent NUM configurations
        --since DATE                 output only configurations since the given date

It will create a directory (called snapshots by default) with subdirectories for each versioned configuration; each of these will be timestamped with the time of the configuration. Within each version directory, there will be directories for node configurations and stored group configurations. (If you’re using an older version of Wallaby, the only stored group configuration will be for the default group. Versioned configurations generated with a fairly recent version of Wallaby, on the other hand, will have stored configurations for more groups and should also have useful information about node memberships in the configuration file.)

wallaby vc-export allows you both to back up versioned configurations and simply to inspect them. Let us know if you find a new application for it, or if you find it useful in other ways!

(Cross-posted from Chapeau.)

The Wallaby source repository now includes a small Python module that patches the Wallaby python client library with idiomatic tagging support: namely, you can ask the store for the name of the group that represents the partition between tags and groups, and you can call getTags() and modifyTags() methods on Node objects, as well as inspect the tags property of a Node. This will appear in Wallaby packages in the future, but you don’t have to be that adventurous to try it out now! To use it, clone the repository and put extensions/tagging.py and schema/wallaby.py somewhere in your PYTHONPATH. Then follow along with this transcript:

I appreciate any feedback or comments on this library.

(Cross-posted from Chapeau.)

The Wallaby API is designed to be sufficiently general to allow developers to do just about anything with configuration data, not to unnecessarily restrict users to a few use cases that we thought of. Because of this generality, some tasks might require adopting application-level conventions. In this article, we’ll cover one such convention and see how the Wallaby API is flexible enough to handle an interesting use case — namely, tagging nodes with various keywords, perhaps as supplied by a user or generated by an agent like sesame or matahari.

First, though, we’ll review how configurations are generated and applied to nodes. Recall that a node is a member of several groups. These memberships are ordered: a node’s lowest-priority membership is always in the special default group (which includes every node), and its highest-priority membership is always in a special, node-specific identity group (which only includes a single node). Zero or more memberships in explicit groups may occupy the priority space between the default group and a node’s identity group. In the illustration below, node.local is a member of two explicit groups, which have blue backgrounds: “EC2 submit nodes,” and “Execute nodes.”

When Wallaby calculates the configuration for node.local., it will begin with a copy of the default group’s configuration. It will then repeatedly apply the configurations for the explicit groups and identity group in order of increasing priority, so that parameter-value mappings from higher-priority groups take precedence over lower-priority ones (and thus either replace these or are appended to them, depending on the mapping kind). The condor_configd, which takes a node’s configuration from Wallaby and installs it on a node, will also cause the Wallaby groups a node is a member of (as well as the Wallaby features it has installed) to be advertised for use in Condor matchmaking. So a Condor job could specify that it wanted to match against a node that was a member of the “EC2 submit nodes” group; this would translate into a preference for node.local.

Because group names appear in machine ClassAd attributes, the list of explicit node memberships is a natural place to put tag information. Nodes would simply be members of “dummy” groups like “Desktop workstations” or “Machines with more than 8GB RAM,” and these keywords could be used in matchmaking or in searching for particular nodes. However, the list of explicit groups is not necessarily suitable for automatic manipulation: users will not necessarily expect their changes to be overridden, other API clients won’t necessarily expect users to rearrange or remove groups corresponding to tags, and in general it is impossible to determine whether a group membership should be interpreted as a tag or as an explicit membership.

We can adopt a convention to partition the space of group memberships. Say we create a special sentinel group to partition the membership space: every node will be a member of the sentinel group. All memberships that are of a lower priority than the sentinel group will be managed by tagging agents, and all memberships that are of a higher priority will be explicitly managed by the user. In the example below, “—EXPLICIT GROUPS” is the sentinel group, and groups in yellow correspond to tags.

This approach demonstrates the generality of the Wallaby API, and allows users to supply tag-specific configurations by installing parameters or features on tag groups, for example, ensuring that every desktop workstation has a policy that favors its owner, or overprovisioning high-memory nodes. However, it is not free of shortcomings. Firstly, Wallaby will present all of these groups — sentinel, tag, and explicit groups — so a user-facing tool that puts a friendly interface on node memberships and tagging will need to use the sentinel group to partition which groups it displays to the user. Secondly, the sentinel groups must be added to all nodes (in practice, we can assume that the absence of a sentinel group implies the absence of tags); in general, we are relying on clients to enforce invariants in this case. Finally, altering a node’s membership list to insert a tag will cause that node’s configuration to be validated at the next activation. If the node has a complex configuration, this could be expensive.

(Cross-posted from Chapeau.)

(Cross-posted from Chapeau.)

The wallaby source repository now includes a Python client library to interact with a wallaby service. This library does a few things to make your life easier:

1. Raw QMF object proxies are wrapped in Python client objects.
2. Object references returned by QMF methods are automatically wrapped by client objects.
3. QMF methods that return error statuses will raise exceptions in the client library.
4. Client classes include docstrings for QMF methods.

See below for a transcript illustrating these features. Note that you’ll need to have the following packages installed to use the Python wallaby client library: Red Hat Enterprise MRG 1.3 (or the python-qmf package from a recent Fedora release) and Wallaby 0.9.18 or later (0.10.0 to access all of the features exposed through the client library).

(Cross-posted from Chapeau.)

First, make sure you’re running wallaby 0.9.23 or later. If you’re running Fedora, you can simply git clone the repository, check out the v0.9.23 tag, and run rake rpms to generate installable packages.

Next, you’ll want to have a directory in which to put your custom wallaby shell commands. I recommend putting them somewhere in your home directory. (For the purposes of this discussion, we’ll assume that you’re using ~/.wallaby.) The wallaby command will find and attempt to load every ruby file in this directory that begins with cmd_, so you probably don’t want it to be writable by anyone other than you. Once you have this directory set up, set WALLABY_COMMAND_DIR in your environment to this directory:

export WALLABY_COMMAND_DIR=${HOME}/.wallaby

We’ll start by making a fairly straightforward command that simply prints out the API version supported by the wallaby service. To create a new wallaby command, use wallaby new-command:

The -d option allows us to provide a description for the new command (viz., what would show up if we typed wallaby help commands). The -D option allows us to specify a directory in which to create the new command file. (wallaby new-command supports several other options; you can use wallaby help new-command for additional documentation.) After executing this command, we’ll have a documented template file for a new wallaby command, called cmd_api_version.rb, installed in our WALLABY_COMMAND_DIR. It will look something like this:

We can now run wallaby help commands and verify that our command shows up. Sure enough, it does:

We can even run wallaby api-version, although it won’t do anything yet. To make it do something useful, we’re going to edit the act method defined for us in the ApiVersion class. We could do something very simple, like insert puts store.apiVersionNumber before return 0, but it would be nice to allow a user to format the API version number as he or she sees fit, in order to use the result of our command in other scripts.

To let the user supply a format string, we’ll need to add a command-line option, which requires us to edit the init_option_parser method. This method must return an OptionParser object; if you haven’t used Ruby’s option parser class, read up on its documentation. For this example, though, the changes we have to make are pretty minor.

After we’ve added a command-line option and a body for the act method, our cmd_api_version.rb file will look more like this:

Running this command without arguments will return a string like The wallaby service is running version 20100915 of the Wallaby API. If you supply a format string, you can get simply the raw number (with --format '%s') or something more suited to your taste. (Note that this example script doesn’t do any sanity- or error-checking of the format string, which you’d certainly want to do if you were going to put your command into production.)

We’ll look at a more interesting example wallaby shell command and some of the issues that more interesting commands raise in a future post.

(Cross-posted from Chapeau.)

Recently, I added constraint support to wallaby inventory. That is, instead of listing all nodes, you can choose to list only nodes for which a given Ruby expression is true. These constraints can be simple, like last_checkin < 1.hour_ago (which will find all nodes that have checked in more than one hour ago), or more complex. They are only limited by Ruby’s $SAFE execution: among other things, they cannot modify running classes, perform file I/O, fork new processes, or, because of how QMF is implemented, invoke QMF methods on the node object. However, every QMF property of the node object is available to wallaby inventory constraints. The transcript below provides some examples running against a wallaby agent loaded up with test data.

(Cross-posted from Chapeau.)

wallaby http-server requires that the Sinatra framework is installed. If you are building RPMs for wallaby, wallaby-http-server will be built as a separate package, and only if you’re building from Fedora 12 or later. If you’re installing from source, simply gem install sinatra before running the HTTP server; if you’ve already installed a wallaby package on a non-Fedora system, you can simply install cmd_http_server.rb from the source repository into your wallaby shell commands directory. (In the future, we expect to package wallaby-http-server for Red Hat Enterprise Linux as well, thus simplifying this process.)

The transcript below shows the HTTP server’s built-in help, listing all of the methods it supports (as of wallaby-0.9.20, which will be released later today). Please let us know if you can think of other API methods that would be useful to expose over the Wallaby HTTP server.

(Cross-posted from chapeau.)

If you’re writing a Wallaby API client in Ruby, you can use a couple of nice features to ease development: the Wallaby client library, which presents a more polished interface than dealing with raw QMF queries and calls, and the wallaby console utility, which connects to a specified broker and gives you a Ruby REPL that has a global variable pointing to a store client — perfect for casual, interactive experimentation.

Use the same global command line options for wallaby console as you’d use for any other wallaby subcommand: specifying host, port, and authentication information for the broker. Once you’ve started it up, you’ll be dropped into a Ruby prompt where Wallaby::store is a reference to a client proxy object for your Wallaby store. You can then inspect or modify any entity that the store knows about; the store object has accessors called nodes, features, parameters, etc., that return arrays of each kind of object. Then you can invoke methods on each object. The transcript below shows a session in which the user inspects some details of a node in a trivial pool:

>> Wallaby::store.nodes.map {|n| n.name}
=> ["frotz"]
>> frotz = Wallaby::store.nodes[0] ; nil
=> nil
>> Wallaby::store.groups.map {|g| g.name}
=> ["+++DEFAULT", "+++af413ebf1de0f9d4253eb89717a4e13b"]
>> Wallaby::store.groups[1].features
=> ["DisablePreemption"]
>> frotz.getConfig
=> {"PREEMPT"=>"FALSE", "WALLABY_CONFIG_VERSION"=>"1284180448517726"}

In the future, it will be easier to extend the wallaby command with your own subcommands. For now, however, wallaby console provides a great way to interact with the Wallaby API.

(Cross-posted from Chapeau.)