AsciiArmor

Lessons Learned from the GitHub Recommender Contest

Fri, 14 Aug 2009 23:42:00 -0400

I spent a few evenings last week working on a contest that GitHub is running to create a recommender engine for their site. Think Netflix Prize but much smaller scale. Their description:

The 2009 GitHub contest is a simple recommendation engine contest. We have released a dataset of our watched repositories data and want to provide a list of recommended repositories for each of our users. Removed from the sample dataset are 4,788 watches - write an open source program to guess the highest percentage of those removed watches and you win our prize.

My objective wasn’t to place first, but to understand the problem more deeply. I distilled down a few observations and presented them at BarCampRDU on Saturday. The discussion was lively and the feedback was positive:

A few people asked for my notes, so here they are…

Lesson 1: Simple is Best

I started off my preparation for the contest by reading a few papers on recommender systems that seemed relevant. Armed with this info, I formulated an approach based on co-occurrence of repositories: Create a matrix of repositories and calculate the number of times a pair of repositories is followed by the same user. Then to generate a recommendation, walk through a users follows accumulating the most frequently co-occuring repositories that he was not following already. Merge the results together for each repository a user follows, based on highest co-occurrence to provide the top 10 list. This took a while to get working right and execution was very, very slow.

It turns out this method wasn’t very effective. I was let down when my first score was just over 900 ( the top score at the time was over 2000). I then decided to take a closer look at both the data and at what everyone else was doing. Not all teams have their source checked in; this is particularly true of the top ranking teams. After some simple analysis on the data, I could see a clear problem: the data is sparse. As the plot below illustrates, most people follow a couple repositories and most repositories have just one or two followers.
R Code for above chart

I also noticed that some submissions using very simple techniques ranked more than twice as high as my submission. One of the most effective methods was simply to walk up the repository fork history and add everything encountered. Another technique made use of the fact that repository owner can be derived from the repository name. If a user follows a single repository, all other repositories sharing the same owner are added. I altered my approach; using these simpler techniques.

R Code for above chart

My entry, including code, can be found here.

Lesson 2: Collaborate

Participants are collaborating both implicitly, through checked in code, and explicitly. The most clear example of this can be seen in @chebuctonian and @xlvector who are working closely together.

This article in the NYTimes suggests that the lesson of collaboration was perhaps the most important that was learned in the Netflix Prize.

The biggest lesson learned, according to members of the two top teams, was the power of collaboration. It was not a single insight, algorithm or concept that allowed both teams to surpass the goal Netflix, the movie rental company, set nearly three years ago: to improve the movie recommendations made by its internal software by at least 10 percent, as measured by predicted versus actual one-through-five-star ratings by customers.

Lesson 3: Have a method

GitHub has done a nice job creating an automated scoring mechanism. As soon as results are committed via git, they are automatically scored and posted on the leaderboard. Top placing teams have created mechanisms to do similar tests offline without the commit phase, thus speeding up the testing of hypotheses. This allows for many ideas to be formulated and then systematically checked and ranked against previous findings.

A related point that we discussed at BarCampRDU was the efficacy of off-the shelf recommender systems. Directed Edge, for example, recently received funding.
Directed Edge provides an API that can be called over HTTP to provide recommendations. However, a general-pupose system like this allows for no domain-specific parameters to be included in the analysis. In the case of the GitHub contest, there is no way to factor in fork-history or repository owners. These were far and away the most effective parametrs to include in the recommendation. I applied for a develpoper key to their web-service. If I get a key, I will run the GitHub dataset and see how it ranks.

If I find time before the end of the month to revise my entry, I would like to refactor my code a more formalized KNN approach where each parameter has a boost co-efficient. Then I would let the computer walk through various boost co-efficients using simulated annealing. I realize this goes counter to lesson #1, but it seems like the simplest competitive approach given what I know now. It also has the benefit of letting the computer do the work for you.

Once completed at the end of the month, it will be interesting to see both the methods used by different teams as well as how the code is put into use by GitHub.

Minifig photo credits: kennymatic and Craig Rodway

Excellent new book in the making: Computational Modeling and Complexity Science

Sat, 23 Aug 2008 18:12:00 -0400

Recently, in a post about his terrific, Programming Collective Intelligence, Toby Segaran calls out the “excluded middle” of technical books. His goal with with PCI was to cover that middle ground - “showing concepts, implementation and applications all at once”. Its a truly great book that will stand the test of time.

I’ve recently stumbled across another title that belongs in this middle ground: Computational Modeling and Complexity Science by Allen B. Downey. He’s also the author The Little Book of Semaphores, Think Python: An Introduction to Software Design, Learning Perl the Hard Way and a few other titles; all self published and released under the GPL.

According to Downey, CMCS, is about “algorithms, intermediate Python programming and complexity science”.

The book has a lot in common with PCI. Both:

Use Python as a basis for all examples
Span academia and pragmatism with working examples
Maintain an easy to read narrative
Are just the right length ( so far! )

CMCS gives just enough theory to provide context. The readability of the text is helped out with anecdotes from the lives of the referenced mathematicians. CMCS takes on analysis of social networks, queuing theory, stochastic modeling, cellular automata and a bunch of other great subjects.

In Chapter 7, Self-organized Criticality, there is some coverage of modeling forest fires:

In 1992 Drossel and Schwabl proposed a cellular automaton that is an abstract model of a forest fire. Each cell is in one of three states: empty, occupied by a tree, or on fire.
The rules of the CA are:

An empty cell becomes occupied with probability p.

A cell with a tree will burn if any of its neighbors is on fire.

A cell with a tree will spontaneously burn, with probability f, even if none of its neighbors is on fire.

A cell with a burning tree becomes an empty cell in the next time step.

I couldn’t help pulling out PIL and gluing some Python together to try this out. Here’s what I came up with (converted to flash to keep filesize down):

source

Much of the book us still under construction and none of the referenced source URLs I tried were functional. That said, one suggestion I do have is to make more use of existing Python libraries. As an a example, the chapter on graphs includes the implementation of basic graph datastructures. This is understandable since this will no doubt be used in undergraduate classrooms, but there should at least be a pointer to NetworkX; a battle-tested warhorse that the reader should be aware of.

It’s a great read so far and I’m looking forward to seeing how the book evolves.

Beneath the Bare Metal: Using dynamic profiling tools like DTrace and VProbes

Sat, 02 Aug 2008 18:04:00 -0400

Just back from BarCampRDU. Lots of great sessions. I especially enjoyed Grant Ingersoll’s presentations on Mahout and Solr.

I did a talk on use of dynamic tracing tools such as VProbes and DTrace. FWIW: Here are some links I referenced in the talk.

DTrace

DTrace Toolkit

VProbes

VProbe Toolkit

Greg Kroah Hartman did a techtalk at Google recently. Some...

Wed, 11 Jun 2008 12:18:00 -0400

Greg Kroah Hartman did a techtalk at Google recently. Some interesting stats related to the Linux 2.6.25 kernel development effort:

2399 contributers over last 1.5 years
1/2 of these have done one patch, 1/4 did 2
9.2M Lines of Code
10% year over year lines of code growth
~55% of code in drivers - majority by a small bit, but not all drivers.
Change rate of kernel code to driver code is proportional. Translation: the core of OS is changing at a crazy rate.
New releases every 2.75 months or so
9000 symbols and 12000 data structures inside the kernel (below the syscall layer)
Rate of change is insane

4300 lines added
1800 lines removed
1500 lines modified
PER DAY!!!

Pythonic API for Tumblr's web API checked in

Sun, 08 Jun 2008 15:55:00 -0400

When my Wordpress blog got hacked, I decided it was time to port over to Tumblr and let them deal with hosting, patching, etc… I put together a python script to import my exiting content via RSS. My original script grew into python-tumblr. Several samples are checked into SVN including one that uses Google Charts to visualize post type frequency.

Comments, feedback, patches welcome…

Google Charts Gets Invader'd

Sun, 16 Dec 2007 16:38:00 -0500

Playing with Google Chart API and PyGoogleChart this afternoon after reading the announcement last week.

A couple things to think about when using the API:

You will want to use a wrapper - Take one look at the link that generated the above graphic, and you’ll quickly see that this is not meant for humans. Maybe write your own wrapper around the bits that you are using. Maybe use one like PyGoogleChart.
It’s not just for web apps - Charts can be cached for later use using curl or equivilent. NB: IANAL so review the Terms of Service and usage policy and make your own decision.
You will have to normalize your data - The chart api docs talk quite a bit about reducing number of elements in your datasets. Another important thing to think about is the range. For instance if I wanted to chart the last few days closing values for the CAC 40 Index which varies from 5,547.21 to 5,750.92. I would have to first normalize the values to something under 4096 for Extended Encoding, under 100 for Text encoding and less than 62 for Simple Encoding. Then the y-axis labels would have to be adjusted to reflect the normalization. Example

It’s pretty clear we’ve only seen the beginning. People outside will find new and unanticipated ways of using the API and Google will continue to crank out new features.

Inspiration for above:

Luke 1.7 is out

Sat, 03 Mar 2007 20:18:00 -0500

After a year or so of shut-eye, Luke, the Lucene query tool, is back. New features listed below. Time to dust off some patches I’ve got for an earlier version and send them along for 1.8.

Luke 1.7 features:

Added pagination of results, especially useful for very large result sets.
Added support for new Field flags. They are now displayed in the Document details, and also can be set on edited documents.
Added a function to add new documents to the index.
Low-level index operations (such as detecting unused files, index directory cleanup) use the newly exposed Lucene classes instead of duplicating their internals in Luke.
A side-effect of the above is the ability to properly cleanup all supported index formats, including the new lockless and single-norm indexes.
Added a function to copy the list of top terms to clipboard.
Added a function to copy the term vector to clipboard.
Added a function to close and/or re-open the current index.
In the Documents tab, pressing “First Term” now positions the term enumeration at the first term for the selected field.
Added a field vocabulary analysis plugin by Mark Harwood, with some modifications.
Overall UI cleanup - improved layout in some places, added graphics instead of ASCII art, etc.

Nicely done Andrzej.

In other Lucene related news, it looks like some interesting things are brewing in the Hadoop project.

Hadoop is a framework for running applications on large clusters built of commodity hardware. The Hadoop framework transparently provides applications both reliability and data motion. Hadoop implements a computational paradigm named Map/Reduce, where the application is divided into many small fragments of work, each of which may be executed or reexecuted on any node in the cluster. In addition, it provides a distributed file system (HDFS) that stores data on the compute nodes, providing very high aggregate bandwidth across the cluster. Both Map/Reduce and the distributed file system are designed so that node failures are automatically handled by the framework.

The Definitive ANTLR Reference

Sat, 24 Feb 2007 12:54:00 -0500

As reported elsewhere, the Pragmatic guys have shepherded another great book through their process: The Definitive ANTLR Reference. Though not intended as a how-to, I’m finding it very readable.

Last time I looked at ANTLR, the ANTLRWorks tool ( pictured above ) didn’t exist. It’s got some quirky aspects and rough edges, but it makes exploring and learning by example a reality. Recommended!

VIM Plugin for JavascriptLint

Mon, 01 Jan 2007 14:21:00 -0500

I’ve put together a vim compiler plugin to provide quick access to JavaScriptLint output. It will identify file, line and column of warnings and errors generated by jsl.

Requirements:

vim 7+
jsl in path

Installation:

Download vimball - jsl.vba
Open vimball - vim jsl.vba

Verify contents - :VimballList
Should see:

would extract : 1 lines
would extract : 22 lines

Install - :so %

Usage:

Open file - vim jsl-test.js
Run JavaScriptLit - :make
View output - :cope

Feedback welcome…

Raleigh NFJS 2006

Wed, 21 Jun 2006 08:19:00 -0400

Recently Jay brought his current lineup of speakers to town for the 2006 NFJS Symposium Tour . Lots of great speakers. Packed with nuts and bolts details. It was buzzword compliant, with plenty of talks on AJAX and Ruby. Justin Gehtland’s talks on hibernate were excellent and can be found on codecite . These are polished speakers who have given these sessions many times prior. Dave Thomas could make event the most prosaic topic engaging and entertaining. The rare bit of wit, humor and humanity go a long way in this field.Points that stood out:

Keep Track of Your Ideas

Andy Hunt talked at length about keeping track of the ideas that occur to you throughout the day. Method doesn’t really matter. Some techniques he proposed: spacepen and pad, voice memos on your cell phone, wiki or wiki modes in your editor. I’ve installed potwiki to give it a try. Elsewhere on the web Michael Michalko seems to have a few things to say on this topic.

I like Michalko’s suggestions for developing ideas once captured:

What can be SUBSTITUTED? (Who else? What else? Other ingredient? Other process? Other power? Other place? Other approach? Can you change the rules?)
What can be COMBINED? (How about a blend, an alloy, an assortment, an ensemble? Combine units? Combine purposes with something else? Combine appeals? Combine ideas?)
What can I ADAPT from something else to the idea? (What else is like this? What other idea does this suggest? Does the past offer a parallel? What could I copy? Whom could I emulate?)
What can I MAGNIFY? (What can be added? More time? Stronger? Higher? Longer? Extra value? Extra features? Duplicate? Multiply? Exaggerate?)
What can I MODIFY or change? (What can be altered? New twist? Change meaning, color, motion, sound, odor, form, shape? What other changes can be made?)
Can I PUT the idea TO OTHER USES? (New ways to use as? Other uses if modified? Can you make it do more things? Other extensions? Other spin-off? Other markets?)
What can be ELIMINATED? (What to subtract? Smaller? Condensed? Miniature? Lower? Shorter? Lighter? Omit? Streamline? Split up? Understate?)
What can be REARRANGED the parts? (What other arrangement might be better? Interchange components? Other pattern? Other layout? Other sequence? Transpose cause and effect? Change pace? Change schedule?)
Can it be REVERSED? (Transpose positive and negative? How about opposites? Turn it upside down? Reverse roles? Consider it backwards? What if you did the unexpected?)

Class Oriented Development

During his keynote Dave Thomas took more than a few well-deserved pot-shots at the received wisdom in the software community. He ticked through some of the usual suspects: type safety, methodologies, backward compatibility. Simply stated, his thesis was: if you don’t a have direct and clear understanding of why you are doing something: stop doing it. One point that resonated with me was his characterization of Java and many other object-oriented languages as really class-oriented. We spend our days developing sophisticated class relationships and hierarchies. Is-a. Has-a. While these languages provide anemic support for actually orienting your development around objects; the heart and soul of your application at runtime. Few true hierararchical taxonomies exist in the real world. Classes were invented by stamp-collectors. Why are we trying to put the world into neat little boxes?

This was of course a shameless plug for features like mix-ins that exist in many modern programming languages . Either way this is an idea worth kicking around the block.

Learn By Doing

In his Refactoring your Wetware talk Andy Hunt drew on the example of surgerical residents who follow the pattern of observe, assist and perform. For surgeons there is no substitute for doing. The same is probably true for developers.

Put down that book. Close Firefox. mkdir testapp

Vim7 Feature Roundup

Tue, 13 Jun 2006 20:33:00 -0400

I noticed Emacs 22 getting some press lately. So figured it was time to spread a little vim7 love .

Here is a grab bag of features I noticed.

Omni Complete

Completion now comes to vim. and vim does it’s best to complete the symbol. It’s not perfect. The implementations look like they are mostly regexp’ing around. No knowledge type systems. Based on the contents of $VIM/autoload it looks like there’s out of the box support for: XML, HTML, CSS, C, JavaScript, PHP, Python, Ruby, and SQL. Sorry Java!

Quoted Text Select

New quoted text selection shortcuts while in select mode. With cursor in a string i” will select all the text within double quotes. If you want the quotes a” will do the trick.

:vimgrep

Vim now comes out of the box with a grep implementation that works well with existing errorlist functionality.Recursively look for all the FIXME tags in the ruby source tree.:vimgrep /FIXME/j C:/ruby/src/**.rb

View the hits and step through them

:cope

Tabs

Enable this feature and you’ll feel like you walked into a room with 3 switches that control the same light. It looks nice, but when cycling through buffers with :bnext and you can see any buffer in any tab. It’s there if you want it, but I suspect this will largely go unused by most gvim people.To open a new tab:

:tabnew

Spell Check

It’s 2006! I want a jetpack not spell checker!

Nothing too surprising here. Ships with English support. You can grab OpenOffice dictionaries and use the :mkspell to convert it into something vim can cope with. Notice red squiggles under the spelling errors and blue under the punctuation errors.

:setlocal spell spelllang=Fr_fr

Undo Branches

This is an excellent feature with a confusing name. Think of it as restore to any point in time during an edit session. It will save your bacon when you make some edits, undo a few levels, make some more edits and realize you wanted something back from before the undo. New commands support restoring to a text state based on time:

g+ / g- This is like u / CTRL-R except it will walk across all changes over time. Good candidates for mapping.
:earlier 5m Restore to state 5 minutes ago
:later 1h Restore to state 1 hour later
:undolist Show leaves in undo / redo tree

:MkVimball

Package and distribute plugins and vim macros into vba files with this new command. When opening vim will recognize the vba files and prompt to extract. Uninstall would be a nice addition.

:GetLatestVimScripts

A very lightweight package manager is now included. Make sure you have a version of wget in your path. Then create a file $VIM/getlatest/GetLatestScripts.dat that looks like:

ScriptID SourceID Filename
————————–

A call to :GetLatestVimScripts will then attempt to update all plugins aware of this new functionality.

It will update the file with version and date information. Note that vim ships with more plugins than this, but GLVS, netrw and vimball are the only ones aware of this new feature.

ScriptID SourceID Filename
————————–
642 5228 :AutoInstall: GetLatestVimScripts.vim
1075 5814 :AutoInstall: netrw.vim
1502 5828 :AutoInstall: vimball.vim

Script Enhancements

Vim’s macro language continues to evolve. Things like this are now possible:

:let num = { 1: ‘one’, 2: ‘two’, 3: ‘three’ }
:echo num[2]

One step closer to emacs convergence?

Music to code by!

Wed, 22 Mar 2006 13:02:00 -0500

The already great somafm just launched space station soma.

Tune in, turn on, space out. Spaced-out ambient and mid-tempo electronica.

Aardvark'd: 12 Weeks With Geeks

Tue, 06 Dec 2005 08:38:00 -0500

I finished Joel’s Aardvark’d documentary last night. It captured the experiences of his 4 interns for 12 weeks. Pointing a camera at 4 guys typing for 3 months and editing that into something entertaining is non-trivial. I struggled to find the thesis, but there was plenty to like.

Some observations:

What it is:

A little bit of advertising. Without being cynical, let’s be realistic. I paid $20 for this thing. Now I’m blogging about it. Mission accomplished.

A look through the key-hole at this medium’s potential. Cameras and computers are cheap. There’s more of this type of thing to come and they will build it’s strengths.

A tease for folks interested in the Y Combinator. Paul Graham’s work in Boston is briefly contrasted. Someone send a guy and a camera up there!

What it might be:

A HOWTO for companies starting internships. There more than a few lessons to be learned: find them housing, buy them a TV, let them get some skin in the game the first day by putting together their workspace, give them the same hardware as the rest of the team, give them a spec, etc…

A depiction of the banality that comes along with software development. Long nights where the only thing interesting that happens is a cockroach crawling across the floor. Endless debates in the forums. The office manager / secretary provides us the view of some one looking in from the outside.

What it wasn’t:

A deep look into the journey of the intern; intern as Ubermensch; emerges from long dark 12 week tunnel having re-evaluated prior ideals.

An insightful portrayal of the starts and stops of software development; the highs and lows; peaks and valleys. While some demos crashed and some bugs were found, the contrast between the intoxicating sensation of shipping and sober realities along the way went largely unaddressed.

Book Review: Maven - A Developer's Notebook

Thu, 21 Jul 2005 07:33:00 -0400

Maven: A Developer’s Notebook is a recently published book by Vincent Massol and Timothy O’Brien in O’Reilly’s Developer Notebook series. It fills a void in the market for a quality book on Maven. I had a chance to spend some time with Vincent while in Paris earlier in the summer; we talked about the book, his company: Pivolis, the French language and every manner of other topics. I’ve just now taken the time read through the book and should have done so earlier. It’s excellent. Vincent’s passion and focus comes across loud and clear.

According to O’Reilly’s site the Developer Notebook Series “is for early adopters of leading-edge technologies who want to get up to speed quickly on what they can do now with emerging technologies.” This book delivers on that promise, weighing in at under 200 pages, but providing the reader with more than enough to get them up and running quickly. It goes beyond the basics, covering customization via maven.xml, multiproject configuration, site customization, and plugin creation. A very practical set of labs are woven through the book as well; making the book suitable for use in a training class.

It is published at an interesting time with Maven2 in alpha. The book targets Maven 1.x, but Vincent and Tim don’t ignore Maven2. They make frequent references to features Maven2 will offer and help prepare the reader for migration when Maven2 is production ready.

The book comes along with a counterpart web site that offers up info on the book and additional content of interest to the reader: www.mavenbook.org

Some areas for improvement:

Help the reader understand if Maven isn’t for them. When is it overkill? When would something else be a better choice? What are the alternatives?
Additional coverage of XDoc; a poorly documented aspect of Maven that is generally learned by looking at other examples (read: cut and paste).

Three and half out of four stars. Great job…

DAX Checked into Subversion

Tue, 05 Jul 2005 06:53:15 -0400

After a few months being off the air, the lights are back on. Relocation from the West Coast of the US to the East Coast, a month in France, selling a house, buying a house, and a new job have kept me a bit more occupied than I expected.

I’ve finally taken the time to check in DAX to the Subversion repository Lars setup for me. Point your svn clients over to: http://www.goshaky.com/svn/dax/

Some tiny changes:

Jar file naming is Maven compatible
@Path is now @Match
build.xml tidied up a bit

A few suggestions have been sent along; I may get to some of these…

Use annotations for the POJO -> XML mapping of simple Java types; mitigating the DOM4J dependancy.
Support XPath2; perhaps via Saxon

Introducing DAX: Declarative API for XML

Thu, 03 Mar 2005 02:28:00 -0500

Inspired by XSLTO, I’ve put together a bit of code to allow XSLT like tranformations from within Java. DAX glues together Java 5 annotations, Jaxen XPath and DOM4J to make possible the declarative style of processing shown below.

Getting Started

Install JDK 5 and a recent flavor of Ant
Download DAX here
Extract and do an “ant test”
“ant javadocs” for API guide
Look at the dax.examples package for sample code

More postings to follow. Stay tuned…


public class BindingTransform extends Transformer {

	public List items = new ArrayList();
	private RSSItem currentItem;

	public BindingTransform() {
		// tell engine about anticipated namespace
		setNamespace("dc", "http://purl.org/dc/elements/1.1/");
	}

	public void init() {
		items.clear();
	}

	public void complete() {
		for (RSSItem i : items) {
			System.out.println(i.getTitle());
		}
	}

	@Path("//item")
	public void item(Node node) {
		currentItem = new RSSItem();
		items.add(currentItem);
		applyTemplates(node);
	}

	@Path("item/title")
		public void title(Node node) {
		currentItem.setTitle(node.getStringValue());
	}

	@Path("item/link")
		public void link(Node node) {
		currentItem.setLink(node.getStringValue());
	}

	@Path("item/description")
		public void description(Node node) {
		currentItem.setDescription(node.getStringValue());
	}

	@Path("item/dc:creator")
		public void creator(Node node) {
		currentItem.setCreator(node.getStringValue());
	}
}

What Great Java Developers Ought To Know ( stone soup version )

Wed, 23 Feb 2005 01:13:34 -0500

What would a Java version of this post look like?

What Great .NET Developers Ought To Know (More .NET Interview Questions)

XPath Performance Followup

Thu, 20 Jan 2005 02:40:05 -0500

Shirasu has extended my XPath benchmarking work to include testing across object models.

His observations:

Traversal of Jaxen is a bit slower than others.

Performance of JXPath does not depend on object models generally. JXPath uses the original traversal mechanism (i.e. NodePointers), and this mesurement suggests that it does not use object model explicitly.

Jaxen and Saxon in DOM is a bit slower than ones in JDOM and XOM.

The following-sibling traversal of Saxon in DOM is a bit slow, but good in JDOM and XOM

Details in English

Details in Japanese

Tips for Computer Book Authors

Fri, 14 Jan 2005 21:09:16 -0500

Over the last 18 months or so I’ve had the great experience of reviewing a handful of books in various stages of development. In doing so I noted several patterns that emerged and captured them here. Enjoy!

Make sample code stand-alone test cases

Whether you are developing Python or PowerBuilder, chances are there is a unit-test framework available. Leverage these frameworks in creating your samples. It forces discipline upon the author to keep examples small and self-contained. More significantly, it provides code that will compile and be immediately useful to readers.

Never underestimate the power of copy and paste.

Lucene in Action

Testing Frameworks

Develop reader personas

Writers face the challenge of crystallizing target readers. What is their level of education? What is their experience with the subject being presented? More importantly, writers face the less obvious question: Who are they not writing for? Poor attention to these questions manifests itself in the form of inconsistencies like: assuming the reader understands the quicksort algorithm but in the previous chapter explaining how to download and unzip a file.

Borrow a technique from the design world to mitigate this challenge: Develop full-blown personas of your readers. Include photos. Include a work-history. Is this the type of person who wants to go home at 5PM or understand the subject from the inside out? What is their reading style? Make a few copies. Hang them on the wall where you write. Send them to your editors.

Mercilessly challenge yourself: Is this something the readers already know? Is this something readers need additional foundation on?

Perfecting Personas

Define your value proposition

Odds are, your book is not the only one on the subject. In the planning stages of the book define your value proposition in clear terms. Send it to your editors. What does your book bring to the party that no other does? Does it consider common pitfalls? Does it describe internal implementation details? Is it aimed specifically at beginners or advanced readers? Is it a comprehensive treatise on a subject?

Struggling to come up with your value proposition? Maybe the project isn’t such a great idea.

Consider various styles of reading

A small minority of your readers will make a front-to-back consideration of your manuscript. More readers look for answers to a specific problem. Still more read in an exploratory style, seeking an overall survey or looking for topics that interest them. Ask yourself: How do I serve these various types of readers? Devices such as “Best Practices” or “Pitfalls” beak-out boxes help capture the attention of explorers. A topical Q&A summary can serve the answer seekers.

Think about the guy at Borders flipping through your book on a rainy afternoon. Remember the frantic developer spilling cold coffee on your book as he works on solving a problem with a production system.

JUnit In Action
Head First Design Patterns

“Don’t be too pleased with yourself.”

This one is borrowed from the style guide provided to writers of the Economist magazine. The idea’s relevance to programmers is emphasized in Larry Wall’s consideration of great virtues a programmer. In it he defines “Hubris” as:

“Excessive pride, the sort of thing Zeus zaps you for. Also the quality that makes you write (and maintain) programs that other people won’t want to say bad things about. Hence, the third great virtue of a programmer. See also laziness and impatience. (p.607)”

Self-confidence is a good thing; leave it unchecked and you will endear yourself to few readers. For whatever reasons, the industry breeds excessive self-assurance. Challenge yourself: Can use of the first person singular be reduced? Better yet, eliminated entirely? Do you see phrases like: “A naive programmer would…” or “Inexperienced users might…”?

Respect your reader; they could probably teach you a thing or two.

3 Traits
Economist Style Guide

Leave your fingerprints on the manuscript

This is the corollary to “Don’t be too pleased with yourself”. Without being self-serving let your unique voice be heard. You represent a unique intersection of experience, education and ideas. Let this individuality come through. Have you been consulting for years and seen techniques succeed or fail. Tell readers about it! Have a relevant war story? Share it! Inject some personality; relentlessly combat a dry documentary style.

If you are not consciously battling tedium, you have ceded to it.

Enterprise Java w/o EJB

Use a single problem domain for sample code

Don’t be seduced by metasyntactic variables. Pick a realistic problem domain and stick to it. Don’t switch from stock quotes in the first chapter to class scheduling in the second. Let your reader reap the benefits of investing time in understanding the problem domain. Note: This is not to say examples should be cumulative and dependent on prior chapters.

Allow them to focus on the message and not the means to convey it.

Hibernate: A Developer’s Notebook
Metasyntactic Variables

Say more by saying less

Keep your language simple and eliminate digressions. Simple words are the most powerful. “It was the best times, it was the worst of times…”. “In the beginning…”. “We didn’t land on Plymouth Rock…” Unsparingly revise long and wordy sentences. Watch some DVD commentaries and listen to directors wax-nostalgic about scenes they deleted in the interest of producing a better film. Learn from them. Don’t make your book the next Ishtar. Unsympathetically delete sentences, paragraphs and chapters.

You’ve got work to do if you can’t remember the last time you simplified a sentence or deleted a portion of your manuscript.

Cryptography Decrypted

Cite other works

Citations reinforce the hard work that went into the manuscript. More importantly they show readers where more can be learned on a specific topic. Subscribe to O’Reilly’s Safari service. Spend some time at a good book store. Work with your publisher to get copies of books they publish on related topics.

Unix System Administration Book
Code Complete

Communicate compromises

Every technology represents a series of compromises. Communicate these to the reader. Don’t make them learn the hard way. This isn’t a hammer for every nail. This isn’t a solution to every problem. Challenge yourself when outlining a topic: What are the relative merits the implementation? What were the alternatives? When should this technology not be employed? What would my readers benefit most for knowing?

Prove to your reader you’re not a zealot.

Effective XML ( particularly items 24,25,41 )

java.util.UUID mini-FAQ

Tue, 04 Jan 2005 02:15:00 -0500

J2SE 1.5 gives us a new class to generate identifiers. Here are some details I found after writing a bit of code and doing some reading.

What is a UUID?

According to Wikipedia:

“…an identifier standard used in software construction, standardized by the Open Software Foundation (OSF) as part of the Distributed Computing Environment (DCE). The intent of UUIDs is to enable distributed systems to uniquely identify information without significant central coordination. Thus, anyone can create a UUID and use it to identify something with reasonable confidence that that identifier will never be unintentionally used by anyone for anything else. Information labelled with UUIDs can therefore be later combined into a single database without need to resolve name conflicts. The most widespread use of this standard is in Microsoft’s Globally Unique Identifiers (GUIDs) which implement this standard.

A UUID is essentially a 16-byte number and in its canonical form a UUID may look like this:

550E8400-E29B-11D4-A716-446655440000

How do I generate a new UUID?

Calling the randomUUID() factory method will give you a fresh instance. Calling toString() will provide a spec-compliant canonical string representation. See the testRandom method in the source below.

I thought these were called GUIDs. What is a UUID?

Per the draft spec referenced below these are synonymous:

…UUIDs (Universally Unique IDentifier), also known as GUIDs (Globally
Unique IDentifier).

I’m a Windows guy. Is this the same thing as CoCreateGuid?

According to MSDN the current version of CoCreateGuid also generates a ‘version 4′ random number based UUID.

The CoCreateGuid function calls the RPC function UuidCreate, which creates a GUID, a globally unique 128-bit integer. Use the CoCreateGuid function when you need an absolutely unique number that you will use as a persistent identifier in a distributed environment.To a very high degree of certainty, this function returns a unique value no other invocation, on the same or any other system (networked or not), should return the same value.

For security reasons, it is often desirable to keep ethernet/token ring addresses on networks from becoming available outside a company or organization. In Windows XP/2000, the UuidCreate function generates a UUID that cannot be traced to the ethernet/token ring address of the computer on which it was generated. It also cannot be associated with other UUIDs created on the same computer. If you do not need this level of security, your application can use the UuidCreateSequential function, which behaves exactly as the UuidCreate function does on all other versions of the operating system.

In Windows NT 4.0, Windows 95, DCOM release, and Windows 98, UuidCreate returns RPC_S_UUID_LOCAL_ONLY when the originating computer does not have an ethernet/token ring (IEEE 802.x) address. In this case, the generated UUID is a valid identifier, and is guaranteed to be unique among all UUIDs generated on the computer. However, the possibility exists that another computer without an ethernet/token ring address generated the identical UUID. Therefore you should never use this UUID to identify an object that is not strictly local to your computer. Computers with ethernet/token ring addresses generate UUIDs that are guaranteed to be globally unique.

I thought UUIDs used the MAC address to guarantee uniqueness

The specification defines multiple types of UUIDs. Version 1 UUIDs include a MAC address. Version 4 UUIDs are generated from a large random number and do not include the MAC address. The implementation of java.util.UUID creates version 4 UUIDs. See the testVersion method in the source below.

So java.util.UUID can only generate version 4 UUIDs. Can it parse and handle the other types?

Yes. Using the fromString method you can instantiate and interrogate a UUID of version 1,2,3 or 4. See the testFromString method in the source below; note it’s a version 1 UUID.

The draft spec talks about multiple variant types of UUIDs. Which variant types are supported by java.util.UUID?

As indicated by the javadoc for getVariant(), java.util.UUID only supports variant type 2.

I like the idea of using the MAC address. What makes version 4 UUIDs better?

Using a version 4 UUID could save you 20 months in a United States Federal Prison. Evidently the writer of the Melissa worm was tracked down in part by the MAC address in a UUID.

So these version 4 UUIDs are basically random numbers. Won’t my UUID collide with someone elses?

There are 122 significant bits in a type 4 UUID. 2¹²² is a *very* large number. Assuming a random distribution of these bits, the probability of collission is *very* low. How is the “randomness” determined?

Under the hood java.util.UUID is creating an instance of SecureRandom and using that to generate new UUIDs. If you are using the default Sun provider and default java.security file, you are using a SHA1PRNG ( Pseudo Random Number Generator based on Secure Hash Algorigthm 1 ) seeded from the operating system.

java.security

#
# Select the source of seed data for SecureRandom. By default an
# attempt is made to use the entropy gathering device specified by
# the securerandom.source property. If an exception occurs when
# accessing the URL then the traditional system/thread activity
# algorithm is used.
#
# On Solaris and Linux systems, if file:/dev/urandom is specified and it
# exists, a special SecureRandom implementation is activated by default.
# This "NativePRNG" reads random bytes directly from /dev/urandom.
#
# On Windows systems, the URLs file:/dev/random and file:/dev/urandom
# enables use of the Microsoft CryptoAPI seed functionality.
#
securerandom.source=file:/dev/urandom

On WinTel platforms this may map down to hardware

I want to preserve backward JDK compatibilty. I’ll just use java.rmi.server.UID. Cool?

Probably a bad idea. Per the docs this gives you 2¹⁶ significant digits. It is also makes no provision for the system clock being set backward.

What other options do I have?

commons-id is a general purpose identifier generator capable of generating UUIDs.

Where can I read more about UUIDs?

Current Draft Specification

Sample Code

import java.util.UUID;
  import junit.framework.TestCase;
public class UUIDTest extends TestCase {  	
	public void testRandom() { 		
		UUID a = UUID.randomUUID();
		UUID b = UUID.randomUUID();
		assertFalse(a.equals(b));
	}  	

	public void testVariant() { 		
		UUID a = UUID.randomUUID();
		assertEquals(a.variant(), 2);
	}  	

	public void testVersion() { 		
		UUID a1 = UUID.randomUUID();
		assertEquals(a1.version(), 4);
		// here is a version 1 UUID plucked from my own HKLMSoftwareClasses 		
		// for history of UUIDs ending in 444553540000 		
		// http://blogs.msdn.com/oldnewthing/archive/2004/02/11/71307.aspx#71356 		
		UUID a2 = UUID.fromString("d27cdb6e-ae6d-11cf-96b8-444553540000");
		assertEquals(a2.variant(), 2);
		assertEquals(a2.version(), 1);
	}  	

	public void testFromString() { 		
		String s = "d27cdb6e-ae6d-11cf-96b8-444553540000";
		UUID a = UUID.fromString(s);
		assertEquals(a.toString(), s);
	}  	

	public void testVersionFromCommonsTestCase() {  		
		// these UUIDs are from commons-id 		
		UUID v1 = UUID.fromString("3051a8d7-aea7-1801-e0bf-bc539dd60cf3");
		UUID v2 = UUID.fromString("3051a8d7-aea7-2801-e0bf-bc539dd60cf3");
		UUID v3 = UUID.fromString("3051a8d7-aea7-3801-e0bf-bc539dd60cf3");
		UUID v4 = UUID.fromString("3051a8d7-aea7-4801-e0bf-bc539dd60cf3");

		//UUID v5 = UUID.fromString("3051a8d7-aea7-3801-e0bf-bc539dd60cf3");
		assertEquals(1, v1.version());
		assertEquals(2, v2.version());
		assertEquals(3, v3.version());
		assertEquals(4, v4.version());

		// java.util.UUID doesn't support version 5 UUIDs while commons-id does 		
		//assertEquals(5, v5.version());
	} 
}