semicomplete.com - Jordan Sissel

Disabling battery/power management in Ubuntu

Mon, 05 Nov 2012 20:04 GMT

I have some ubuntu VMs. VirtualBox's guest tools provide a way for guest OSs to see battery and other power states.

The problem is that Ubuntu 12.04 will always suspend itself when battery is low. What is "low"? Something not tunable. Can you change the 'low battery' action? No, the setting panel for it says "Perform action on critical battery" or something and the dropdown box is blank.

When Ubuntu decides it's 'critical' time, OSX (the host os, here) claims usually 20% battery remaining. Critical, eh? :\

So, the next step is to get Ubuntu/Gnome to not even see a battery.

The utility providing this data is 'upowerd'. You can try to kill it, but dbus will just start it back up again. It's not a service you can disable with upstart, since it's managed by dbus itself.

You can't just uninstall the 'upower' package, no, because it will remove gnome-session and other probably important packages for your workstation.

To disable upowerd, you have to remove this file:

/etc/dbus-1/system.d/org.freedesktop.UPower.conf

Then reboot. Alternately, you could 'pkill upower; restart dbus' but I found that restarting dbus made gnome/xorg freak out (blink, apps restarting frequently, etc) as well as it killing the network (because networkmanager), so I had to reboot anyway.

And now I have a vm that doesn't foolishly suspend itself. Also, dbus is shitty.

TF2 performance on wine+linux

Mon, 05 Nov 2012 05:11 GMT

I recently gave up windows 8 (which is horrible, by the way) to install Fedora on my workstation at home.

I wanted to still play TF2, so while I wait for the steam linux/tf2 beta, I figured wine would work.

I used the fedora wine packages as well as 'winetricks' to install steam (winetricks is awesomesauce.). Basically, with winetricks, you just do 'winetricks steam' to install steam. Bam! Done.

To run steam after winetricks installs it, you'll want to do this crazy business, because winetricks installs steam to it's own "wine prefix":

WINEPREFIX=$HOME/.local/share/wineprefixes/steam/ wine "C:\\Program Files (x86)\\Steam\\Steam.exe"

When running TF2, I noticed framerates were pretty crappy. Most googling I did found practically no useful information except for one or two forum posts that indicate CPU affinity being the most likely cause. This made sense given frame rates were completely the same regardless of graphical settings used (resolution or features like model quality, etc).

First step is to enable 'multicore rendering'

Then, when tf2 is up, run this script below. It looks for the most cpu-hungry threads in hl2.exe (tf2) as well as for the steam/wine/xorg processes. It then pins them to specific CPUs.

(
  # Get the top cpu-using threads for the 'hl2.exe' process (tf2)
  # then pin each to a separate CPUs and give them elevated scheduling priority
  top -bn1 -Hp $(pgrep hl2.exe) \
  | awk '$NF == "hl2.exe" { split($(NF - 1), t, ":"); cputime = t[1] * 60.0 + t[2]; print cputime, $1 }'  \
  | sort -n | tail -3 | awk '{print $2}'

  pgrep 'Steam|wineserver|Xorg'
) \
| awk '{print NR, $1}' \
| sudo xargs -tn2 sh -c 'taskset -p $((1 << ($1 - 1))) $2; renice -n -2 $2' -

Once I run this command while tf2 is up, frame rate doubles (to around 60) and is much more consistent (bursty drops in framerates)

MITRE's CEE is a failure for profit.

Wed, 12 Sep 2012 05:34 GMT

I wrote this post a few months ago, but never got around to publishing it.

Anyway, someone mentioned 'project lumberjack' and I found it was based on CEE: Common Event Expression. CEE is a sort of comedic tragedy of design.

The effort is owned by a "non-profit" (MITRE), but the complexity and obfuscation in CEE can only drive towards one thing: consultant profits. I had a go at explaining what I describe in this post on the 'project lumberjack' mailing list, but I did it quite poorly and got a few foot-stomps in response, so I gave up.

CEE is a failure because, while claiming to be a standards effort, it maximizes incompatibility between implementations by doing the following:

poorly describes multiple serialization formats, requires none of them. This ensures maximum incompatibility in event serialization
defines requirements for log transport protocols, but does not describe an actual protocol. This ensures maximum protocol incompatibility
naming style inconsistencies This ensures confusion

In general, the goal of CEE sounds like, but is actually not, creating a standard for common event expression. Instead, CEE is aimed at ensuring consulting dollars through obfuscation, complexity, and inconsistency.

Inconsistency.

No consistency in naming. Some names are abbreviations like ‘crit’, some are prefixed abbreviations (“p_” prefix), some are full english words like ‘action’.

If the goal was to be inconsistent, CEE has succeeded.

Mysterious ‘p_’ prefix. Base fields are abbreviated names like “crit”, “id”, “pri”, yet others are called “p_app”, “p_proc”, and “p_proc_id”.
Some fields are abbreviated, like “crit” and “pri”, but others are full english words, like “action” and “domain”
there’s “id” which marks the “event type” by identifier, but “uuid” which marks the event instance identifier. You are confusing. I’m still not sure what the purpose of ‘uuid’ is.

ETOOMANYPROTOCOLS

Serializations: JSON, RFC3164, RFC5424, and XML serializations
4 conformance levels.

Pick one serialization and one transport (“conformance”) requirements list. Describe those two. Drop all the others.

If I pick JSON, and you pick XML, we can both be CEE-conforming and have zero interoperability between our tools. Nice!

Serialization underspecified

JSON for event serialization is fine, but no message framing is defined. Newline terminated? Length prefixed? You don’t know :(

JSON

“Reserved Characters” - I don’t think you have read the JSON spec. Most (all?) of the ‘escaping’ detailed in CEE JSON is already specified in JSON: http://www.json.org/string.gif

Specific comments on the ‘json’ format inline as comments, this example was copied verbatim from the CEE :

{
    # Forget this "Event" crap, just move everything up a level.
    "Event": {
        "p_proc": "proc1",
        "p_sys": "host.vendor.com",
        "time": "2012-01-18T05:55:12.4321-05:00",
        "Type": {
            # Action is like, edge-trigger information.
            # Status is like, line-trigger information.
            # You don't usually have both edge and line triggers in the
            # same event. Confusing.
            "action": "login",
            "status": "ongoing",

            # Custom tag values also MUST start with a colon? It's silly to make
            # the common case (custom tags) the longer form.
            # Also, tags is a plural word, but the value is a string. What?
            "tags": ":hipaa"
        },
        "Profile": {
            # This is a *huge* bloat, seriously. Stop making JSON into XML, guys.
            "CustomProfile": {
                "schema": "http://vendor.com/events/cee-profile.xsd",
                "new_field": "a string value",
                "new_val": 1234,
                "product_host": "source.example.com"
            }
        }
    }
}

If you include the schema (CEE calls it a Profile) in every message, you’re just funding companies who’s business model relies on you paying them per-byte of log transit.

Prior art here for sharing schema can be observed in the Apache Avro project, for example.

CLT Protocol

Just define the protocol, all these requirements and conformance levels make things more confusing and harder to write tooling for.

If you don’t define the protocol, there’s no incentive for vendors to use prior art and are just as likely to invent their own protocols.

Combining this incentive-to-invent with the whole of CEE that underspecifies just about every feature, this guarantees incompatibilities in implementations.

Growing logstash's value

Mon, 30 Jul 2012 06:00 GMT

I spent a while today thinking about nerdy stuff - logstash, etc. I want to grow logstash in terms of performance, use case, deployment instances, happy users, and community.

While musing about on my mental roadmap of logstash, I found most things boil down to costs and returns on investment, even with open source software. Money, time, energy, and patience are all costs. Just because something doesn't cost any money doesn't mean it won't consume any time or energy.

I see two distinct groups of users, with respect to cost. New users and current users. In terms of inputs and outputs, the phrase 'return on investment' comes to mind. New users are likely in the evaluation and investigation phase, mainly estimating ROI or judging "is this solution useful to me?" Existing users are likely in the maintenance and integration phases, mainly trying to improve or maintain ROI or pushing towards improving value provided by a tool.

These two user groups are, in my observation and from a seller's perspective, quite distinct in terms of strategy. How you acquire happy new users is not necessarily how you maintain and energize existing users.

Targeting New Users

New users easily stumble on bad documentation, complex architectures, and excessive steps.

I have a few goals regarding minimization for new users: reducing mean time to demo and mean time to deploy are critical. Reducing 'time to demo' requires that I focus on minimizing steps required to answer 'is logstash useful?'. Additionally, ensuring each required step in 'time to demo' is as simple as possible. Reducing time to deploy means publishing high quality init scripts (upstart, systemd, sysv init) as well as high quality puppet, chef, and cfengine modules.

The time someone spends as what I consider a 'new user' is actually quite short. My goal with users in this stage are to help them quickly and accurately answer, "will this tool benefit me?"

Targeting Existing Users

As an existing user of a tool, I'm often looking for how to reduce operational, maintenance, and integration costs. Operational costs appear as physical resource usage (servers, or fuel-like resources). Maintenance costs appear as bugs and related investigations, monitoring, etc. Integration costs appear as time and energy spent making the tool work well with your infrastructure.

These kind of users are usually a bit more invested in use of the tool, but I want to avoid abusing that fact. Time and energy investments in a free tool can cause as much vendor trappings as monetary investments. Don't treat your existing users like dirt, right?

My goal for the next few months is to become one of these existing users of logstash (to date, as I've confessed, I've never run logstash in production). I'll be able to do that at DreamHost (starting tomorrow!).

Additionally, I will be focusing strongly on improving logstash new user experience. Happier new users should reflect nicely on community growth and activity. That's my goal, anyway!

Installing Windows 8 Consumer Preview

Thu, 10 May 2012 06:50 GMT

I have a fresh workstation and am running through the windows 8 installer on USB. When choosing the drive to install to, I get an error:

We couldn't create a new partition or locate an existing one

Lots of googling and I didn't find any hints for windows 8, but windows 7 has a similar error and folks pointed at diskpart nonsense to fix it. So let's do that -

At the installer, choose "Repair your computer"
Choose "troubleshoot"
Choose "advanced options"
Choose "command prompt"
Run diskpart.

In diskpart, you'll want to make sure your target drive is formatted and active.

list disk

# now pick your disk
select disk 0
clear
create partition
format fs=ntfs compress quick
active

Now reboot and try the installer again, it worked for me.

When all you have is a hammer, make your own tools?

Wed, 29 Feb 2012 06:01 GMT

Clarifying my position from this post:

The "ops folks need coding skills" groupthink is lame. Software requires extra coding because it is shitty, not because people are unskilled

I will lead with this: I want more people who use technology to grow and learn better skills for bending that technology to their needs. An ops guy with programming skills is, to me, more valuable than one who cannot - programming in any language or platform lets you extend an otherwise static system.

Anyway, back at the post in question, I'm not trying to say people (ops or otherwise) shouldn't want stronger programming skills. I'm saying the equipment we use is pretty shitty.

I am part of the generation raised near devices ever blinking "12:00". Devices which have no business caring what time it is, nor any sane reason to make the state of "I don't know what time it is" a high priority alert worth blinking forever.

It's 2012, and this problem persists - my microwave refuses to cook food unless it has the time *and* date from user input. Now I have to program it every time it has a power disruption (which is has frequently due to some bug in the hardware causing it to power off randomly with certain dishes at home).

Now I have to learn to program or configure these devices before they'll stop irritating me. And, damn it I hate that. If, instead, this were enterprise software, I could report these irritations to the vendor who would kindly offer me training and consulting for extortionate piles of money.

I love coding. It's fun, and many times lets me solve problems I couldn't otherwise. Allowing me to abuse an analogy, "When all you have is a hammer, you can sit down and build whatever tool you need to repair the delusion that everything is a nail."

But despite being able to solve my own problems in software, I don't think this is a great pattern of work. I write code, most of the time, because the solutions available are terrible or don't meet my requirements. With a new software popping up every day, I see a strong correlation between software availability and people asking for more programmers.

So, the more software we have, the more programmers we need to work around limitations in the available body of software. I think that's pretty lame :(

And regarding my microwave problems, I want some confidence that the problems being solved are meaningful problems, not programming learned for the sake of working around bugs and misfeatures in software we're suffering with.

Goodbye, 2011.

Sun, 01 Jan 2012 00:29 GMT

This year's been pretty good, but the last two months were pretty lame.

In the last six weeks, I found out Caramel has lymphoma, got unemployed, and had emergency surgery to remove my appendix on Christmas Day. The unemployment caused me to lose an in-progress mortgage refinance.

I'll pick up the mortgage thing once I remedy the employment problem, but I'm staying quite happily unemployed until after my kid is born - should be any day now!

Most of my career-growing moves were outside of work: at meetups, in open source efforts, or in networking with folks on IRC or twitter. Lots of awesome folks out there, so go introduce yourself. Don't be a dick. :)

I didn't write much on this site, but mainly, that was due to an increase in my activities on IRC and twitter. Most of what I published this year was code and was less writing about said code. I'd like to fix that, though.

This years successes were topped by two new major projects, fpm and logstash. I also released some major improvements to xdotool and other tools.

The current implementation of logstash isn't very old, but prototypes, hacks, and other incarnations of pretty much the same thing date back to at least 2005 and probably earlier. This project has been a long-time-coming, and Pete Fritchman and I have been talking about logstash for years, so it's nice to finally have some code shipped and a community building around it.

FPM had a crazy positive response. I wrote it as a hack, and it's used all over the place now. Bonus that people are contributing patches and other improvements as well.

Sysadvent was another excellent success, the end of which marked the 4th year and 100th article posted to the project. It is awesome seeing such community involvement from so many different authors.

This year also cemented my move to git from svn. Why? Github, mostly, and not really the features of git itself. Sharing code and patches is so much easier on github than it is with other services.

I went to CarolinaCon and OSCON to talk about logstash. I also went to DevOps Days Mountain View and gave a lightning talk on logstash.

My OSCON talk was overflowing with people standing at the back of the room, etc; it went awesomely. I've also been able to do lunchtime logstash presentations at places like Square and others. I also gave talks at BayLISA meetings. It was a good year for getting out of the house and talking about code.

I tried to get a count of how much code I'd written this year, but I had lots of web-based projects that included third-party stuff like jquery, and I'm too lazy to pick through the results and trim that stuff out. I'm up to about 70 different projects on github now, some useful; some not; all fun!

Looking forward to 2012 :)

Insist on better asserts

Sun, 09 Oct 2011 05:25 GMT

I never really liked C's assert() feature. If an assertion is violated, it'll tell you what assertion failed but completely lacks any context:

example: example.c:9: main: Assertion `i == 3' failed.

This is better:

Assertion failed insist.c:7 in main(), insist(i == 3): Something went wrong, wanted i == 3, got 4

The main difference here is that there's context about what failed. A message for humans looking to debug this. This is especially important on Linux these days because every distro I've used recently hates sysadmins and hates debugging - all libraries are stripped of debug symbols and coredumps are disabled by default.

What's the usage look like?

#include 
#include 
#include "insist.h"

int main() {
  int i = 4;
  //assert(i == 3);
  insist(i == 3, "Something went wrong, wanted i == 3, got %d", i);
  return 0;
}

I also added a special 'return' version of this, 'insist_return' that lets you do error checking and early aborting like this:

insist_return(fd >= 0, START_FAILURE,
              "socket() returned %d, an error: %s", fd, strerror(errno));

Works just like insist() except returns START_FAILURE if 'fd > 0' is false and additionally logs the error formatted above.

Code here: insist.h

xdotool 2.20110530release

Mon, 30 May 2011 08:34 GMT

It's been about 8 months since the last xdotool release, and I think it's long overdue! This release has a ton of new feature and fixes.

Download: xdotool-2.20110530.1.tar.gz

As usual, if you find problems or have feature requests, please file bugs or send an email to the list.

Changelist since previous announcement:

2.20110530.*
  - New set_window feature: --urgency. This lets you set the urgency flag on a
    window Window managers will interpret this as something about your window
    needing attention. It might flash in the taskbar, pop up, or other.
    Original patch and suggestion by ervandew.
    Issue: http://code.google.com/p/semicomplete/issues/detail?id=39
  - New function: xdo_window_seturgency (see above)
  - Hack in OS X support as it is missing a proper clock_gettime.
    Should fix http://code.google.com/p/semicomplete/issues/detail?id=37
    Reported by ryandesign.com
  - Add support for typing UTF-8 characters. Patch from Joseph Krahn.
  - Make all output call fflush to send data immediately (for pipes). Reported
    by Andreas Wagner on the mailing list.
  - Make 'get_desktop_viewport' output usable with 'set_desktop_viewport'
    http://code.google.com/p/semicomplete/issues/detail?id=47
  - You can now make 'libxdo.a' for embedding libxdo into your binary
    (Requested by psc on the mailing list).
  - Fixed a typing bug where the keymap changes unnecessarily 
  - Should now build cleanly in C++ environments (Reported by psc on the
    mailing list)
  - bugfix: xdotool should use command names first before trying file scripts. 
    See https://bbs.archlinux.org/viewtopic.php?pid=938309 for original report.
  - Add a 'sleep' command. (Requested by Joseph Krahn via mailing list)
  - Add --relative flag to windowmve. (Requested by Anthony Thyssen via mailing
  - Add --desktop flag to the search command. This lets you search for windows
    on specific desktop. Requires a window manager that supports multiple
    desktops in a way that EWMH supports.
    Fixes http://code.google.com/p/semicomplete/issues/detail?id=38
  - Add --limit flag to search. This allows you to break the search early after
    a certain number of matches. (Requested by Anthony Thyssen)
  - New command 'getwindowgeometry' for fetching window position and size
    (Requested by Anthony Thyssen via mailing list)
  - Add --sync flag to search command; blocks until results are found.
    xdotool will search every 0.5 seconds for results.
    http://code.google.com/p/semicomplete/issues/detail?id=42
  - windowmove can now move windows along an axis. Give literal 'x' or 'y'
    instead of a coordinate and it uses the current position. (Requested by
    etnlIcarus via mailing list)
  - Add '--args N' and '--terminator TERMINATOR' to the 'exec' command.
    Default terminator unless specified (or --args is) is ':' (Requested by
    Joseph Krahn and Henning Bekel via mailing list)
  - set_desktop now supports --relative flag (+N or -N to move relative)
    (Requested by Anthony Thyssen)
  - The mouse cursor now changes during 'xdotool selectwindow' (Requested by
    Anthony Thyssen via mailing list)
  - Added '--args N' and '--terminator TERMINATOR' to the 'type' command.
  - Add 'getdisplaygeometry' command for querying the size of your screen.
    (Requested by @rrwo via twitter)
  - Add xdo_get_viewport_dimensions function.

logstash's first major release - 1.0.0

Sat, 07 May 2011 08:00 GMT

Ready for log and event management that doesn't suck or drain your budget? It's time to logstash.

After lots of refactoring and improvements to logstash since the first minor release last November, logstash is ready for wider usage now.

Read my announcement here.

The logstash site is also online and has docs, intros, slides, and videos.

http://logstash.net

Happy logstashing!