AsciiArmor

Building Go Applications for ARM Single Board Computers

Tue, 28 Oct 2014 20:58:00 -0700

Last week Chris Snell and I co-presented at the Seattle Go Meetup. Here are the slides to both presentations and a quick photo.

Developing Applications for Beagle Bone Black, Raspberry Pi and SoC Single Board ARM Computers with Go from ryancox

Solid Run generously sent us a dual core HummingBoard which we gave away to one of the attendees.

Jenkins: Now with more Gopher

Thu, 02 Oct 2014 17:03:00 -0700

At the last Seattle Go Programmers Meetup there was some interest expressed in work I had done using Jenkins to handle continuous integration for a couple of projects. I’ve captured details below along with a Dockerfile that lets you get up and running quickly with a worked example that builds the Hugo project.

Setting up the demo within Docker

A docker file and job config is checked into GitHub that shows these ideas in action.

On a 64 bit Linux system running a recent version of docker perform the following steps.

Workspace Setup

Jenkins creates a workspace directory for each job that contains all source and generated artifacts. This makes a convenient GOPATH root; ensuring that all dependencies are known and accounted for.

Setup

Install the EnvInject Jenkins plugins
Under Build Environment, select ‘Inject environment variables to the build process’ and enter 'GOPATH=$WORKSPACE’ under Properties Contents
All subsequent 'go get’ or other go tools calls will operate within the context of the Jenkins workspace directory for this job.

Git configuration

A useful configuration option when building projects from a Git repository is adding a Git reference that specifies the commit to be built by the job. The value can be defaulted to 'refs/heads/master’ which will build from master during automatic or ad-hoc execution if not overridden. However, if a one-off job run is desired for a particular tag, branch or commit; this can easily be specified as a parameter. This also makes it very easy to go back and run prior versions of a project, in order to see historical trends for test pass rates, benchmarks and warnings counts.

Setup

Install the Git Client Jenkins plugin
Configure a job parameter of COMMIT that defaults to 'refs/heads/master’
In the job’s Git Client setup, set the 'Branch Specifier’ to $COMMIT

Running Tests

While there is no native support for ingesting 'go test’ output, Jenkins does have excellent handling of JUnit output. With the go2xunit utility, output from 'go test’ and go check can be transformed into a format that the JUnit Jenkins plugin can process. This makes possible capture and visualization of test output.

Setup

Add a job step that installs go2xunit: 'go get bitbucket.org/tebeka/go2xunit’
Add a job step that runs tests with the ’-v’ option and capture the output to a file: 'go test -v github.com/spf13/hugo/… > test.output’
Add a job step that converts the go test output to JUnit xml: 'cat test.output | “./bin/go2xunit” -output tests.xml’
Add a Post Build Action to Publish JUnit test results and specify 'tests.xml’

Visualizing Benchmarks

Go has great support for adding benchmarks. I’ve written a tiny utility gobench2plot that extracts benchmark output and transforms it into an XML format that can be plotted. This format can then be the existing Plot Jenkins plugin.

Setup

Install the Plot Jenkins plugin
Add a job step that installs gobench2plot: 'go get github.com/ryancox/gobench2plot’
Add a job step that runs tests and benchmarks: 'go test -bench=Bench* -test.benchmem -v github.com/spf13/hugo/… > ${WORKSPACE}/test.output’
Add a job step that transforms benchmark values in test output to xml: 'cat test.output | gobench2plot > benchmarks.xml’
Add a post build job step that does a line style plot of the values; specifying benchmarks.xml and an appropriate XPath query: ’/Benchmarks/AllocsPerOp/*’

Static Analysis

The popular Jenkin plugin 'warnings’ parses and captures output of static analysis tools like lint and findbugs. There was no support for Golint and Go vet. So I added this capability and submitted a pull request to the project. This PR has been merged into master.

Note: At the time of writing, a new release has not been created for 'warnings’ and pushed to the Jenkins plugin server. So until then, you will need to build from source or use the binary in the example project by manually uploading the HPI file into Jenkins via the UI or just copying it to your plugins directory.

Setup

Install the Warnings Jenkins plugin
Add build step that installs glint or go vet: 'go get github.com/golang/lint/golint’
Add build step that runs glint or go vet and saves output: ’./bin/golint github.com/spf13/hugo/… > lint.txt’
Add a 'Scan for Compiler Warnings’ post build action that specifies the output file and Go Lint or Go Vet Parser.

Testing Multiple Go Versions

Package owners may want to test support and performance across several different versions of the Go. This can be accomplished using the Jenkins multi-configuration job type. This job type allow for various 'axis’ of execution to be defined. A new Jenkins build is created for each axis combination. A single axis named GOVERSION, for example, can be created that contains values such as 'go1.1.2 go1.2 go1.3 go tip’. In the shell execute steps, $GOVERSION would be used in place of the normal 'go’ command. This scheme requires that the system upon which Jenkins is executing has multiple Go. This can be accomplished by installing the binaries in a location that is accessible system wide ( e.g. /usr/local/go-1.1.2 and /usr/local/go-1.3 ), then creating symlinks in /usr/local/bin/go-1.1.2 and /usr/local/go-1.3 that point back to the binaries in these locations ). It can be useful to keep a current build from the tip of master installed and accessible as 'gotip’. This allows for testing of packages against the most recent changes to Go. TODO: Do I need the matrix plugin for this?

Setup

Create build job as 'Build multi-configuration project’ rather than a 'Freestyle’ job type
Under Configuration Matrix, define a User defined Axis of 'GOVERSION’ containing values 'go1.1.2 go1.2 go1.3’
Replace calls to 'go’ command with $GOVERSION
Configure host system with multiple versions of Go such that 'go1.1.2, go1.2 and go1.3 map to the respective versions of the tools

Note: To keep the example simple, this is not included in the accompanying Dockerfile

Special thanks to Dylan Meissner for his review of this post

motionless: taking the pain out of Google Static Maps

Wed, 04 Aug 2010 04:46:00 -0700

I have found myself needing to generate several map visualizations lately. Creating complex URLs for Google Static Maps can be time consuming and error prone. To make this process more civilized, I’ve put together a simple python library. It supports markers, paths as well as simple centered maps. All the details can be found on github. In brief, install with pip:

pip install motionless

Simple map centered on the Eiffel Tour:

from motionless import CenterMap 
cmap = CenterMap(lat=48.858278,lon=2.294489,maptype='satellite') 
print cmap.generate_url()

Slightly more complex example generated from a GPX file on my Garmin device illustrating how to plot paths and add markers. Code is in the examples directory

Tips for Computer Book Authors

Fri, 14 Jan 2005 18:09:16 -0800

Over the last 18 months or so I’ve had the great experience of reviewing a handful of books in various stages of development. In doing so I noted several patterns that emerged and captured them here. Enjoy!

Make sample code stand-alone test cases

Whether you are developing Python or PowerBuilder, chances are there is a unit-test framework available. Leverage these frameworks in creating your samples. It forces discipline upon the author to keep examples small and self-contained. More significantly, it provides code that will compile and be immediately useful to readers.

Never underestimate the power of copy and paste.

Lucene in Action

Testing Frameworks

Develop reader personas

Writers face the challenge of crystallizing target readers. What is their level of education? What is their experience with the subject being presented? More importantly, writers face the less obvious question: Who are they not writing for? Poor attention to these questions manifests itself in the form of inconsistencies like: assuming the reader understands the quicksort algorithm but in the previous chapter explaining how to download and unzip a file.

Borrow a technique from the design world to mitigate this challenge: Develop full-blown personas of your readers. Include photos. Include a work-history. Is this the type of person who wants to go home at 5PM or understand the subject from the inside out? What is their reading style? Make a few copies. Hang them on the wall where you write. Send them to your editors.

Mercilessly challenge yourself: Is this something the readers already know? Is this something readers need additional foundation on?

Perfecting Personas

Define your value proposition

Odds are, your book is not the only one on the subject. In the planning stages of the book define your value proposition in clear terms. Send it to your editors. What does your book bring to the party that no other does? Does it consider common pitfalls? Does it describe internal implementation details? Is it aimed specifically at beginners or advanced readers? Is it a comprehensive treatise on a subject?

Struggling to come up with your value proposition? Maybe the project isn’t such a great idea.

Consider various styles of reading

A small minority of your readers will make a front-to-back consideration of your manuscript. More readers look for answers to a specific problem. Still more read in an exploratory style, seeking an overall survey or looking for topics that interest them. Ask yourself: How do I serve these various types of readers? Devices such as “Best Practices” or “Pitfalls” beak-out boxes help capture the attention of explorers. A topical Q&A summary can serve the answer seekers.

Think about the guy at Borders flipping through your book on a rainy afternoon. Remember the frantic developer spilling cold coffee on your book as he works on solving a problem with a production system.

JUnit In Action
Head First Design Patterns

“Don’t be too pleased with yourself.”

This one is borrowed from the style guide provided to writers of the Economist magazine. The idea’s relevance to programmers is emphasized in Larry Wall’s consideration of great virtues a programmer. In it he defines “Hubris” as:

“Excessive pride, the sort of thing Zeus zaps you for. Also the quality that makes you write (and maintain) programs that other people won’t want to say bad things about. Hence, the third great virtue of a programmer. See also laziness and impatience. (p.607)”

Self-confidence is a good thing; leave it unchecked and you will endear yourself to few readers. For whatever reasons, the industry breeds excessive self-assurance. Challenge yourself: Can use of the first person singular be reduced? Better yet, eliminated entirely? Do you see phrases like: “A naive programmer would…” or “Inexperienced users might…”?

Respect your reader; they could probably teach you a thing or two.

3 Traits
Economist Style Guide

Leave your fingerprints on the manuscript

This is the corollary to “Don’t be too pleased with yourself”. Without being self-serving let your unique voice be heard. You represent a unique intersection of experience, education and ideas. Let this individuality come through. Have you been consulting for years and seen techniques succeed or fail. Tell readers about it! Have a relevant war story? Share it! Inject some personality; relentlessly combat a dry documentary style.

If you are not consciously battling tedium, you have ceded to it.

Enterprise Java w/o EJB

Use a single problem domain for sample code

Don’t be seduced by metasyntactic variables. Pick a realistic problem domain and stick to it. Don’t switch from stock quotes in the first chapter to class scheduling in the second. Let your reader reap the benefits of investing time in understanding the problem domain. Note: This is not to say examples should be cumulative and dependent on prior chapters.

Allow them to focus on the message and not the means to convey it.

Hibernate: A Developer’s Notebook
Metasyntactic Variables

Say more by saying less

Keep your language simple and eliminate digressions. Simple words are the most powerful. “It was the best times, it was the worst of times…”. “In the beginning…”. “We didn’t land on Plymouth Rock…” Unsparingly revise long and wordy sentences. Watch some DVD commentaries and listen to directors wax-nostalgic about scenes they deleted in the interest of producing a better film. Learn from them. Don’t make your book the next Ishtar. Unsympathetically delete sentences, paragraphs and chapters.

You’ve got work to do if you can’t remember the last time you simplified a sentence or deleted a portion of your manuscript.

Cryptography Decrypted

Cite other works

Citations reinforce the hard work that went into the manuscript. More importantly they show readers where more can be learned on a specific topic. Subscribe to O'Reilly’s Safari service. Spend some time at a good book store. Work with your publisher to get copies of books they publish on related topics.

Unix System Administration Book
Code Complete

Communicate compromises

Every technology represents a series of compromises. Communicate these to the reader. Don’t make them learn the hard way. This isn’t a hammer for every nail. This isn’t a solution to every problem. Challenge yourself when outlining a topic: What are the relative merits the implementation? What were the alternatives? When should this technology not be employed? What would my readers benefit most for knowing?

Prove to your reader you’re not a zealot.

Effective XML ( particularly items 24,25,41 )

java.util.UUID mini-FAQ

Mon, 03 Jan 2005 23:15:00 -0800

J2SE 1.5 gives us a new class to generate identifiers. Here are some details I found after writing a bit of code and doing some reading.

What is a UUID?

According to Wikipedia:

“…an identifier standard used in software construction, standardized by the Open Software Foundation (OSF) as part of the Distributed Computing Environment (DCE). The intent of UUIDs is to enable distributed systems to uniquely identify information without significant central coordination. Thus, anyone can create a UUID and use it to identify something with reasonable confidence that that identifier will never be unintentionally used by anyone for anything else. Information labelled with UUIDs can therefore be later combined into a single database without need to resolve name conflicts. The most widespread use of this standard is in Microsoft’s Globally Unique Identifiers (GUIDs) which implement this standard.

A UUID is essentially a 16-byte number and in its canonical form a UUID may look like this:

550E8400-E29B-11D4-A716-446655440000

How do I generate a new UUID?

Calling the randomUUID() factory method will give you a fresh instance. Calling toString() will provide a spec-compliant canonical string representation. See the testRandom method in the source below.

I thought these were called GUIDs. What is a UUID?

Per the draft spec referenced below these are synonymous:

…UUIDs (Universally Unique IDentifier), also known as GUIDs (Globally
Unique IDentifier).

I’m a Windows guy. Is this the same thing as CoCreateGuid?

According to MSDN the current version of CoCreateGuid also generates a ‘version 4′ random number based UUID.

The CoCreateGuid function calls the RPC function UuidCreate, which creates a GUID, a globally unique 128-bit integer. Use the CoCreateGuid function when you need an absolutely unique number that you will use as a persistent identifier in a distributed environment.To a very high degree of certainty, this function returns a unique value no other invocation, on the same or any other system (networked or not), should return the same value.

For security reasons, it is often desirable to keep ethernet/token ring addresses on networks from becoming available outside a company or organization. In Windows XP/2000, the UuidCreate function generates a UUID that cannot be traced to the ethernet/token ring address of the computer on which it was generated. It also cannot be associated with other UUIDs created on the same computer. If you do not need this level of security, your application can use the UuidCreateSequential function, which behaves exactly as the UuidCreate function does on all other versions of the operating system.

In Windows NT 4.0, Windows 95, DCOM release, and Windows 98, UuidCreate returns RPC_S_UUID_LOCAL_ONLY when the originating computer does not have an ethernet/token ring (IEEE 802.x) address. In this case, the generated UUID is a valid identifier, and is guaranteed to be unique among all UUIDs generated on the computer. However, the possibility exists that another computer without an ethernet/token ring address generated the identical UUID. Therefore you should never use this UUID to identify an object that is not strictly local to your computer. Computers with ethernet/token ring addresses generate UUIDs that are guaranteed to be globally unique.

I thought UUIDs used the MAC address to guarantee uniqueness

The specification defines multiple types of UUIDs. Version 1 UUIDs include a MAC address. Version 4 UUIDs are generated from a large random number and do not include the MAC address. The implementation of java.util.UUID creates version 4 UUIDs. See the testVersion method in the source below.

So java.util.UUID can only generate version 4 UUIDs. Can it parse and handle the other types?

Yes. Using the fromString method you can instantiate and interrogate a UUID of version 1,2,3 or 4. See the testFromString method in the source below; note it’s a version 1 UUID.

The draft spec talks about multiple variant types of UUIDs. Which variant types are supported by java.util.UUID?

As indicated by the javadoc for getVariant(), java.util.UUID only supports variant type 2.

I like the idea of using the MAC address. What makes version 4 UUIDs better?

Using a version 4 UUID could save you 20 months in a United States Federal Prison. Evidently the writer of the Melissa worm was tracked down in part by the MAC address in a UUID.

So these version 4 UUIDs are basically random numbers. Won’t my UUID collide with someone elses?

There are 122 significant bits in a type 4 UUID. 2¹²² is a *very* large number. Assuming a random distribution of these bits, the probability of collission is *very* low. How is the “randomness” determined?

Under the hood java.util.UUID is creating an instance of SecureRandom and using that to generate new UUIDs. If you are using the default Sun provider and default java.security file, you are using a SHA1PRNG ( Pseudo Random Number Generator based on Secure Hash Algorigthm 1 ) seeded from the operating system.

java.security

#
# Select the source of seed data for SecureRandom. By default an
# attempt is made to use the entropy gathering device specified by
# the securerandom.source property. If an exception occurs when
# accessing the URL then the traditional system/thread activity
# algorithm is used.
#
# On Solaris and Linux systems, if file:/dev/urandom is specified and it
# exists, a special SecureRandom implementation is activated by default.
# This "NativePRNG" reads random bytes directly from /dev/urandom.
#
# On Windows systems, the URLs file:/dev/random and file:/dev/urandom
# enables use of the Microsoft CryptoAPI seed functionality.
#
securerandom.source=file:/dev/urandom

On WinTel platforms this may map down to hardware

I want to preserve backward JDK compatibilty. I’ll just use java.rmi.server.UID. Cool?

Probably a bad idea. Per the docs this gives you 2¹⁶ significant digits. It is also makes no provision for the system clock being set backward.

What other options do I have?

commons-id is a general purpose identifier generator capable of generating UUIDs.

Where can I read more about UUIDs?

Current Draft Specification

Sample Code

import java.util.UUID;
  import junit.framework.TestCase;
public class UUIDTest extends TestCase {  	
	public void testRandom() { 		
		UUID a = UUID.randomUUID();
		UUID b = UUID.randomUUID();
		assertFalse(a.equals(b));
	}  	

	public void testVariant() { 		
		UUID a = UUID.randomUUID();
		assertEquals(a.variant(), 2);
	}  	

	public void testVersion() { 		
		UUID a1 = UUID.randomUUID();
		assertEquals(a1.version(), 4);
		// here is a version 1 UUID plucked from my own HKLMSoftwareClasses 		
		// for history of UUIDs ending in 444553540000 		
		// http://blogs.msdn.com/oldnewthing/archive/2004/02/11/71307.aspx#71356 		
		UUID a2 = UUID.fromString("d27cdb6e-ae6d-11cf-96b8-444553540000");
		assertEquals(a2.variant(), 2);
		assertEquals(a2.version(), 1);
	}  	

	public void testFromString() { 		
		String s = "d27cdb6e-ae6d-11cf-96b8-444553540000";
		UUID a = UUID.fromString(s);
		assertEquals(a.toString(), s);
	}  	

	public void testVersionFromCommonsTestCase() {  		
		// these UUIDs are from commons-id 		
		UUID v1 = UUID.fromString("3051a8d7-aea7-1801-e0bf-bc539dd60cf3");
		UUID v2 = UUID.fromString("3051a8d7-aea7-2801-e0bf-bc539dd60cf3");
		UUID v3 = UUID.fromString("3051a8d7-aea7-3801-e0bf-bc539dd60cf3");
		UUID v4 = UUID.fromString("3051a8d7-aea7-4801-e0bf-bc539dd60cf3");

		//UUID v5 = UUID.fromString("3051a8d7-aea7-3801-e0bf-bc539dd60cf3");
		assertEquals(1, v1.version());
		assertEquals(2, v2.version());
		assertEquals(3, v3.version());
		assertEquals(4, v4.version());

		// java.util.UUID doesn't support version 5 UUIDs while commons-id does 		
		//assertEquals(5, v5.version());
	} 
}