Non-volatile memory

Request logs rolling with suffixes from Jetty

2010-08-25T16:14:00.003-04:00

It seemed simple enough. The goal: get jetty to log requests to a file such as request.log and rotate that file on a daily basis to something with a suffix.

By default Jetty wants to rename request logs something like mm_dd_yy_request.log, this is all well and good, but it complicates the logic of any tool you might use to sync and remove log files. For example log4j's RollingFileAppender will append numeric suffixes like .1 to the end of logs it has closed and rotated out, but with Jetty's default configuration, one has to get a little tricky to figure out what request log is still open. It's not as easy as grabbing request.log.* to grab the out-of-rotation logs.

I discovered the solution in the code of RolloverFileOutputStream, the class in jetty-util that Jetty's NCSARequestLog class uses internally to write the log files. When logfile appending is turned off and RolloverFileOutputStream attempts to open a new log where a file with the same name already exists, RolloverFileOuputStream will rename the existing file by appending the date, format specified by the system property ROLLOVERFILE_BACKUP_FORMAT, to the existing logfile name. This means that the request.log is the live, open log and files named like request.log.yyyyMMdd-HHmmss are the logs that are out of rotation.

So to achieve this, modify the RequestLog entry in jetty.xml to look like this:

    <Ref id="RequestLog">
      <Set name="requestLog">
        <New id="RequestLogImpl" class="org.mortbay.jetty.NCSARequestLog">
          <Arg><SystemProperty name="jetty.logs" default="./logs"/>/request.log</Arg>
          <Set name="retainDays">31</Set>
          <Set name="append">false</Set>
          <Set name="extended">false</Set>
          <Set name="LogTimeZone">GMT</Set>
          <Set name="ignorePaths">
            <Array type="java.lang.String">
              <Item>/solr/ping</Item>
            </Array>
          </Set>
        </New>
      </Set>
    </Ref>

And at starup of your jetty instance, do something like:

java -DROLLOVERFILE_BACKUP_FORMAT=yyyyMMdd-HHmmss -jar start.jar

And Viola! New log messages are written to request.log and logs are rolled out to request.log.somenicesuffixhere.

As for the ignorePaths stanze in the configuration above, you probably don't need it in your configuration. It's handy for me to skip logging health checks.

Also as for syncing the logs off the individual machines they're written to, rsync's --remove-sent-files option is pretty handy.

Changing Laptop Screen Brightness from the command-line

2010-08-01T13:33:00.001-04:00

My new Dell Studio 15's laptop screen is great. It is bright, and by bright I mean that on full brightness I can use it outside on a sunny day with my sunglasses on and still read the screen comfortably. This brightness is great except for cases when I don't want it so bright, like when I'm working in a darker room. One night I discovered that the lower third of my vision had a dark blind spot after doing some work while watching a movie. That can't be good.

Problem is, I don't always remember to adjust the brightness, and from what I can tell, it always boots into Ubuntu af full brightness, so I dug around for a way to change the brightness automatically. Thanks to google I found a post over on LinuxScrew that talks about changing the brightness for a dell inspiron 1501. Turns out the Studio 15 has a similar method using /proc. Cat'ing the file shows me the acceptable brightness levels:

# cat /proc/acpi/video/M86/LCD/brightness 
levels:  6 12 18 24 30 36 42 48 54 60 66 72 78 84 90 100
current: 100

Echo'ing something to the file (as root) sets the value:

# echo -n 18 > /proc/acpi/video/M86/LCD/brightness

There's lots of places to go with this, but for now I'm going to pop it into rc.local and run with it.

No more blind spots, excellent.

Serving content with Solr's Jetty instance.

2010-07-27T16:47:00.001-04:00

When prototyping with Solr 1.4.x, I've found it useful to serve webpages, javascript and images using the Jetty server used by Solr's example instance. Doing so is straightforward:

Create a directory named contexts in the Solr examples directory which contains the etc and start.jar files. Create a context definition file inside this directory. It can be named anything, I commonly call it root-context.xml

The file contains the following:

<Configure class="org.mortbay.jetty.webapp.WebAppContext">
  <Set name="contextPath">/</Set>
  <Set name="resourceBase"><SystemProperty name="jetty.home" default="."/>/contexts/root</Set>
  <Set name="defaultsDescriptor"><SystemProperty name="jetty.home" default="."/>/etc/webdefault.xml</Set>
</Configure>

In the contexts directory, create a directory named root and place your content there. Voila -- once you launch the solr instance with java -jar start.jar, everything placed in the root directory is available under http://yourhostname:8983, while solr is availble at http://yourhostname:8983/solr

A Useful Hint to Young Engineers

2010-05-21T08:52:00.001-04:00

When I visited Thomas Edison's lab in Orange, NJ last winter, I spotted a clipping in one of the displays:

"A Useful Hint to Young Engineers -- 'The man for whom every employer of men is searching, everywhere and always is the man who will accept the responsibility for the work he has to do -- who will not lean at every point upon his superior for additional instructions, advice or encouragement.

There is no more valuable subordinate than the man to whom you can give a piece of work and then forget about it, in the confident expectation that the next time it is brought to your attention it will come in the form of a report that the thing has been done. When this master quality is joined to the executive power, loyalty and common sense, the result is a man whom you can trust. On the other hand, there is no greater nuisance to a man heavily burdened with the direction of affairs than the weak-backed assistant who is continually trying to get his chief to do his work for him, on the feeble idea that he thought the chief would like to decide this or [that]."

Gender bias aside, a useful hint indeed.

Ubuntu 9.10 and sticky java dialog boxes.

2010-01-09T16:25:00.002-05:00

I'm running Ubuntu 9.10 in my laptop and have been suffering from an unusual problem. The buttons on dialog boxes created by the eclipse ide 'stick'. When I click on any button in a dialog, it gets stuck in the down position and never activates. The only solution is to tab over to the button if possible and activate it with the return key. Very irritating.

I finally tracked down the fix today. The problem is caused by something related to the way in which GDK renders its windows in the latest version. The solution is to set the environment variable GDK_NATIVE_WINDOWS=1 prior to launching eclipse. Apparently this is also a problem on newer versions of Fedora as well. Indeed it is probably an issue with anything running gdk 2.18 or newer, where the switch to 'client side windows' was made.

I've fixed this by renaming eclipse to eclipse.bin and placing the following script named eclipse in the same directory:

#!/bin/bash
BASE=$(dirname $0)
export GDK_NATIVE_WINDOWS=1
exec $BASE/eclipse.bin

don't forget to chmod +x it so that it is executable. You'll also have to rename eclipse.ini to eclipse.bin.ini -- eclipse finds its ini based on the binary name.

sheesh.

For an explanation of what GDK_NATIVE_WINDOWS is all about, check out the post "What GDK_NATIVE_WINDOWS=1" means over on Dax's blog.

GUID vs. Auto-Increment Primary Keys.

2008-09-03T11:48:00.001-04:00

This all started off from reading yet another wonderful post over at Better Explained, which is currently my favorite blog of the moment. My Google reader starred items list overflows with entries to-read from this blog. I wanted to brush up on the semantics of GUIDs/UUIDs because I'm currently implementing RDF output for one of our products from work and keeping the object URIs unique and consistent is a good thing. I'm using a simple MD5 hash for that.

The interesting aspect that cem out of this is the discussion of using GUIDs as database primary keys. We'd tried that a couple years ago but abandoned the implementation for a number of reasons.

I'll come back and finish this post some day, but I really just want to capture a bunch of links in one place. It would be cool if those social bookmarking services would allow you to cluster a bunch of links together into a single posting. At any rate, I'll use a blog for this instead and see how it works.

Coding Horror: Primary Keys: IDs versus GUIDs
Brian "Krow" Aker's Idle Thoughts - Myths, GUID vs Autoincrement: Excellent discussion here, need to go back and find out if he ever did benchmarks for secondary keys.
MySQL Performance Blog: To UUID or not to UUID?

The general sentiment here is is that using UUID's instead of auto-incrementing ints decentralizes id assignment, enabling scale and avoiding having to re-key data when it's merged. The tradeoffs related to space are worth it and the performance impact is likely to be minimal

Earthquake in NJ

2008-08-17T22:44:00.001-04:00

Growing up in NJ, earthquakes were unheard of. Back when I was a kid in Bergen County I felt one. It was a Saturday morning, and I was at a friend's house after sleeping there the night before. It lasted long enough for us to realize we didn't know what was going on and to run outside of the house and witness everything moving.

I was thinking about the quake lately and finally managed to track down the event. According to the Lamont-Doherty Earth Observatory of Columbia University, the earthquake occured about 6:04am on October 19th, 1985 and was measured at 4.0 on the Richter scale. Its epicenter was in Ardsley, NY. (40.98N, 73.83W), referred to as the "Ardsley Quake".

Color Grep

2008-03-16T14:23:00.001-04:00

I was grep-ing through some RSS feeds today when it struck me that there's got to be a way to highlight the text that grep matches much like the matching text in vi or less is when searching. Well, it turns out there is.

Just like ls, grep supports the --color option. This means with --color=auto, grep will colorize the portion of the line that matches when it sends output to a terminal, but avoid adding the color escape sequences when you're sending the output to a file or pipe. Unlike ls, which relies on some setup to get it displaying colors properly (see: /etc/profile.d/colorls.sh if you don't believe me), setting up grep to colorize matches by default is as easy as defining the environment variable GREP_OPTIONS='--color=auto' in either your local or system-wide shell startup script.

A typical use-case for me is to pipe the output of grep output to less. If you do this too, keep in mind that the right invocation to get colorized output in less is:

grep --color=always foo bar.txt | less -R

The -R argument to less allows it to pass ANSI color codes unchanged, and --color=always forces grep to colorize its output even though it is going to a pipe.

If you're interested in customizing the color of the highlighted matches, you can set the GREP_COLOR environment variable. A good source of example colors is available in the adequately-commented /etc/DIR_COLORS that's used for setting up color ls. I like GREP_COLOR='43;30;01' which gives you a nice black on yellow highlight for the matched text.

del.icio.us and xhtml parsing in java.

2007-11-04T19:16:00.004-05:00

To find the information that del.icio.us knows about any url use http://del.icio.us/url/(url-md5sum). The md5 is a standard 32-character md5 string with downcased hex digits. URLs should be properly slash terminated prior to generating the md5, for example, http://www.slashdot.org/ but not http://www.slashdot.org. In some cases people have tagged the urls without the trailing slash.

The html (xhtml really) served by del.icio.us is really clean, so occured to me that the various entries I'm interested in extracting could be retrieved using the same strategy used to identify microformats. That also started me thinking about using xpath expressions to identify and exract the right xhtml nodes.

I didn't have much luck. xhtml is an interesting beast. When following the java Xpath api usage examples, the parser complains about the contents of script tags containing illegal characters, and del.icio.us doesn't wrap their script tag contents contents in a CDATA, which doesn't seem to be required or common practice anyways.

It turns out that a combination of jTidy and dom4j seems to work reasonably well.

InputStream in = new FileInputStream("target/out.xhtml");

Tidy tidy = new Tidy();
tidy.setShowWarnings(false);
tidy.setQuiet(true);
tidy.setXHTML(true);
tidy.setXmlOut(true);
org.w3c.dom.Document domDocument = tidy.parseDOM(in, null);

DOMReader domReader = new DOMReader();
Document doc = domReader.read(domDocument);

List list = doc.selectNodes("//div");

for (int i=0; i < list.size(); i++) {
System.out.println(list.get(i).toString());
}

Crawling

2007-10-27T22:04:00.001-04:00

The twins are almost nine months old now, and we've got our hands full. They're both on the edge of mobility. Owen actually gets around pretty well these days, happilly terrorizing stacks of magazines, cd, photo frames, while Audrey still uses the steamroller technique and often gives up in a show of arm and leg flapping amongst other protesting.

Nevertheless, we've only begun to child-proof our house. The long and winding road towards child-safety remains to be fully traveled.

A Picture of the Twins

2007-08-26T13:57:00.001-04:00

I've been cleaning out the memory cards on various devices.

Here are Audrey and Owen lying in their crib. They're about 5 months old in this picture. Yes, they're holding hands. How cute is that? Now that they're more mobile, if you were to put them into a crib together one will eventually end up steamrolling the they would start trying to chew on each other's closest extremity. This leads to unhappy babes, which is not a good thing, especially since Owen has his first two teeth -- and they're sharp!

I'm also using this as an excuse to test out posting to a blogger blog via Picassa.

Verizon Bluetooth EVDO on Ubuntu

2007-08-08T10:49:00.000-04:00

I managed to get my Verizon KRZR k1m connected to my ubuntu laptop via bluetooth for dialup EVDO access. The BluetoothDialup help page on help.ubuntu.com was essential to figuring it out, along with the carrier specific information for Verizon

I did not have any luck getting EVDO dialup to work with the k1m connected to the laptop via USB. Apparently the usb configuration on Verizon's k1m doesn't cause the cdc-acm usb driver to load for the device and I ended up getting stuck with uhci instead. So, it appears that I'm out of luck as far as using it over a usb connection is concerned.

The interesting thing is that the speeds I got via Bluetooth in linux vs. USB using Verizon's access manager software in windows were quite similar. At my location, in the suburbs of the Washington DC area, I managed to get ~480k down and ~128k up. I wonder how that compares to the speeds that others have received elsewhere.

Now the only minor annoyance is that this doesn't work via NetworkManager at all, and I'm stuck with using pon and poff from the command line. I can set up the call via NetworkManager 's ppp control, but nothing is notified that the ppp connection has succeeded, so I can't for instance use the NetworkManager applet to start my vpn connection once the connection is set up.

FiosTV and Network changes.

2007-05-11T09:59:00.001-04:00

Finally, there's decent tv service in our area.

The incumbent provider's service in this portion of Montgomery County, MD was very poor -- static/poor picture on some channels, often unwatchable video-on-demand despite a couple service visits and more than a couple new cable runs from the curb to my house. It got to a point where I just wasn't willing to invest any more time into working with them to solve the problem.

I never made the jump to satellite. There's too many trees in the south-facing sky on our property, and after having FIOS internet installed about a year ago now, I knew the tv service was around the corner. I kept checking the threads in the broadband reports forum for updates on the status of the local verizon cable franchise agreement and rollout of installs across the county.

Enter Verizon's FIOS tv service, delivered to the house via fiber. The poor picture is gone, video-on-demand works well, all together a much better experience. I still have to deal with IR Blasters from my Tivo's to the Verizon supplied Motorola STB's but that annoyance is nothing new. Apparently the serial controls are working for some areas, but I haven't gotten around to trying that here yet.

Getting the new hardware that comes with the service working with my home network just the way I wanted was a bit tricky. I don't really have what you'd call a typical network setup and plus, I like to fiddle around to see what's possible.

With the install came 2 motorola set top boxes and a brand new ActionTec MI424-WR router. The thing that sets this router apart from your run-of-the mill hardware is that it hooks up to both the ethernet and coax drop from the FIOS ONT and provides a 4 port switch to hook your home network into. The set top boxes provided for TV service get their TV signal directly from the ONT, but get guide information and video-on-demand over the IP network via the router, which serves as a bridge between the wan ethernet connection and the coax running through my house. Data over coax is really referred to as MoCA, a spec from the Multimedia Over Coax Alliance (new to me). Apparently, some new FIOS installs get their internet connectivity over a wan coax connection, and the router can handle bridging between that and a lan ethernet connection as well. Pretty cool. Finally, the router also acts as a wireless access point, supporting the G standard. I've got to give kudos to Verizon/ActionTec for having wireless security configured out of the box with the default wep keys printed on the bottom of the router. It's WEP 64, but better than nothing.

By default this thing is set up to run your network -- assign full on dhcp addresses to your machines, dns, etc you name it. Perfect for the person who wants an easy network install and doesn't want to worry about configuring a thing.

If you dig around on the net for information about this router, you'll see alot of hate tossed in its general direction. There's been some discussion that the ActionTec has a puny NAT table that ruins the performance of some P2P apps. Also, people love their Linksys WRT54G's. The level of customization you can perform on these really makes them your 'own' router. I recently discovered the incredible Tomato Firmware and I was pretty dismayed at having to move the thing behind this new ActionTec thing as the FIOS installer required, that and the quick and dirty setup I had in place to make sure everything was working before the installer left was doing double NAT. ugh. As soon as the guy had left the house, I was switching things around to get that WRT54G out in front again.

Getting everything configured properly to do that was a breeze. Initially, the WRT54G was connected to the WAN via PPPoE and doing NAT for everything behind it. The wan port of the ActionTech was plugged into a LAN port on the WRT54G and getting an IP using dhcp. The STB's were connected to the ActionTech and had no problem getting IP addresses and guide data, but video-on-demand was hosed. Changing the ActionTech into a bridge (I roughly followed that guide) was trivial as well but didn't solve the video-on-demand problem either. I found a posts reporting that video-on-demand works with certain firmware on the WRT54G v5, but others that suggested it might be a that video-on-demand doesn't work with the Tomato Firmware. I suspect it has something to do with the QoS support in that firmware -- in the ActionTech's default config, specific QoS settings that are specified for the Verizon STB's.

So, I'm back to having the ActionTech out in front, handling pppoe, nat and bridging the ethernet and coax connection. The wrt54g is still behind this, acting more as a simple router than anything else right now -- it's still the default route for most of the machines on my network (other than the STB's a voip phone, tivos and a couple other devices). DHCP, DNS, VPN connections, among other tings, are still handled via my trusty linux box -- that's not going away any time soon.

[UPDATE: I flashed my wrt54g with DD-WRT and I have been running fine with the ActionTech in bridged mode for a couple years. I've added the Verizon DVR and continue to receive all services without the need to reconfigure or any problems. YMMV as my install was performed a long time ago and Verizon may install things differently these days.]

Anti-aliased fonts in gnome-terminal on Centos 5

2007-05-07T19:24:00.001-04:00

I use vncserver / vncviewer for more than you can imagine, mostly as a reasonable replacement for screen when working off of spotty wireless networks. I was pretty bummed out after upgrading a number of machines to Centos 5, that the fonts in gnome-terminal weren't anti-aliased anymore. I'm a sucker for a smooth-edged Bitsteam Vera Sans Mono.

After digging around a bit, and following a couple red herrings in the form of ~/.fonts.config, or /etc/fonts/.., I found a couple references to anti-aliasing being slow only with xservers that supported XRender, of course vncserver doesn't implement this extension, so that was a potential clue that something along these lines was the issue. Hmm, I never noticed gnome-terminal being slow unless I was tail -f'ing a webserver log or something else spouting prodigious amounts of output.

Finally, I remembered the gconf tree and finally the tool gconf-editor, which hadn't been installed by default on any of my Centos 5 installs. One invocation of yum later, I had gconf-editor installed and was poking around in the apps/gnome-terminal folder. It turns out that under there, in the profiles/default folder there's the no_aa_without_render option, which was enabled by default. Unchecking this fixed the problem -- ahh, a nice smooth Bitstream Vera Mono in my gnome-terminal again.

As an aside, I was pretty surprised to see the number of people asking how to turn off anti-aliasing instead of turning it on -- I can understand a certain affection for Misc-fixed, but fonts without jagged edges are just so much easier on my eyes. I guess all things being equal, this would probably work on Redhat Enterprise Linux / rhel 5 as well.