If you’ll recall, before my last post, my intention was to cover the Go RPC functionality.  I ran into a problem in that the gob package was used internally and I did not yet understand it.  Well, as it turns out, the reflect package is also used. Values are inspected and translated into corresponding gob data. So this time, we’ll look at reflection within Go.  This is an interesting topic as it not only prepares us for an upcoming RPC post, but it also really solidifies understanding of the Go type system.

Also, realize that I’m not an expert on Go. If you happen to be one with your Go-Fujitsu and you spot something incorrect (or even perhaps something not idiomatic), please leave a comment. That will help make me an expert!

The Inspected Code

In this example, we’ll dial back down to one source module, it’s just easier that way.  We’ll be using the little gogogo command that we created in the first Go post.  The first thing we’ll do is create a small collection of types and methods such that we can see what each component looks like when introspected. Here’s the source that we’ll actually be exploring at run time.

package main

import "reflect"
import "log"

type MailMessage struct {
  subject string
  from string
  to string
  body string
  high_priority bool
}

/* Create a new mail message. */
func NewMailMessage(subject string, from string, to string) (*MailMessage) {
  return &MailMessage{subject, from, to, "", false}
}

/* Set the message body. */
func (m *MailMessage)SetBody(body string) {
  m.body = body
}

/* Set message priority. */
func (m *MailMessage)SetPriority(high bool) {
  m.high_priority = high
}

/* Send the message */
func (m MailMessage) SendMessage() (bool) {
  log.Printf("Message to %s has been sent\n", m.to)
  return true
}

/* Typedef 'int' to a new MessagesSent type. */
type MessagesSent int

Note that at the top of this listing we also declare our package name and include the necessary import statements.  If you’re following along and wish to compile, then you’ll wish comment out the line importing reflect as we’re not using it yet. As an aside, I’m not sure I like that feature of Go just yet.

A few notes on this example here.

1. We define a new type, which is based on the structure. The new type contains a series of elements.
2. We define three methods and one function.  Two of the methods take a pointer to a MailMessage structure, whereas the third simply takes a MailMessage structure itself.
3. We’re not doing anything. Nothing. This is a highly contrived example meant to highlight reflection. We’ll tackle the SMTP package down the road a little bit, you know, once we finally get that RPC stuff out of the way.

Great. Now that we have enough code that does nothing whatsoever, we’ll write some code that does something (with nothing). First, a bit of a detour.

The Type Switch

First, we’ll look at what Go deems the “Type Switch.”  This gives us a mechanism, via a special syntax switch statement, to chose a code branch based on the underlying type of the object.   Consider the following listing.

package main

import "log"

func main() {
  x := 1

  switch x_type := x.(type) {
    case int:
      log.Println("It Was An Integer")
     }
}

This looks simple enough. Given x, we only want to print something if it is indeed an integer type. That special dot-type syntax you see above is what qualifies this as a type switch.  However, take a look at what happens when we try to compile this code.

$ 6g type_switch.go
type_switch.go:8: cannot type switch on non-interface value x (type int)
$

That’s slightly confusing.  It turns out that, in this case, x is being used as a object with a concrete type of integer. That is, it’s not being referenced via an interface.  If we update the code, we can get the desired result.

package main

import "log"

func TypeSwitcher(v interface{}) {
   switch v_type := v.(type) {
    case int:
      log.Println("It Was An Integer")
   }
}

func main() {
  TypeSwitcher(1)
}

Now, before we run this, note the interface{} type. That’s an empty interface. Empty interfaces can stand in for everything. Let’s build this again and give it a run, it should now print out what we’re expecting.

$ gogogo type_switch.go
2011/02/23 22:54:02 It Was An Integer

So, now you understand what a type switch is and how decisions based on type can be made in Go. However, what if we’re not checking against an interface? Or, rather, what if we want to know a little bit more about an object? Perhaps, say, how many fields a structure has? That’s where reflection comes in.

The Reflecting Code

So, like we’ve done in a few other examples, we’ll just dump the entire listing here and then walk through it afterwards.  There are quite a few new elements here. Hang in there, we’ll explain them all.

/* Pull info we care about from reflect.Type */
func GetTypeInfo(t reflect.Type) (string, reflect.Kind, int) {
  return t.Name(), t.Kind(), t.NumMethod()
}

/* Print information about a reflect.Value. */
func ReflectOnValue(v reflect.Value) {
  /* Nothing to do on a nil. */
  if v == nil {
    log.Println("nil value, nothing to do")
    return
  }

  /* Pull the information we require about the Type. */
  name, kind, methods := GetTypeInfo(v.Type())

  /* Now type switch and extract the Value Info. */
  switch v := v.(type) {

    case *reflect.StructValue:
      log.Printf("Discovered %s/%s that has %d fields and %d methods.\n",
        kind, name, v.NumField(), methods)

    case *reflect.PtrValue:
      log.Printf("Discovered a pointer with %d methods, dereferencing...\n", methods)
      ReflectOnValue(v.Elem())

    default:
      log.Printf("Found %s/%s with %d methods.\n", kind, name, methods)
  }
}

/* Make anything an interface. */
func PrintValueData(i interface{}) {
  ReflectOnValue(reflect.NewValue(i))
}

Alright! This code is responsible for digging into the details of our objects and reporting what it finds.  Let’s walk through it (almost) line by line and examine exactly what’s going on.

• The GetTypeInfo function takes a single argument, a reflect.Type, and returns a string, a reflect.Kind, and an integer value.  This little function returns the name of the type in question, the underlying kind, and the number of methods associated with this type in receiver context.
• The ReflectOnValue function takes a reflect.Value object v and returns nothing.
• First, if v is nil, we simple print that out and return. There’s no reason to go any further.
• Next, we call GetTypeInfo and pull information about this value’s corresponding type. That’s important. In short, this means that each value has a type that contains information such as which methods are associated with it.
• Next up, another type switch. This time, however, we’re switching on the reflect value and not the original value. This allows us to branch according to what the value is.
• The first option is a *reflect.StructValue. So, if v’s type is pointer-to-struct-value, then we’ll print out the name of the struct, the fact that it is a struct, the number of fields associated with it, and the number of methods it has.  There are calls which allow us to individually access each field by name and number, and to access individual methods as well. We skim over that in this example, however.
• The next option is very interesting – *reflect.PtrValue. Or, in English, a pointer to a pointer value. This is not the same as a pointer to a pointer! Since pointer types can have methods associated with them, we print the number here. Next, we dereference the pointer via “pointer.Elem()” and recursively call ourself.   This implies that pointers aren’t true pointers, rather, a flavor of smart pointer.
• The last thing we do in this function is print any “standard” types, followed by the number of methods they have associated with them.
• Finally, we have PrintValueData. This little helper function simply translates any given value into a reflect.Value, to allow for processing.

The Driver

Our goal here is to see how different types and values are viewed by the Go system.  We create a series of variables in our main method and send them to our PrintValueData function.

func main() {
  /* Mail Message Objects */
  m_ptr := NewMailMessage("subject", "from", "to")
  m_std := MailMessage{"subject", "from", "to", "body", false}

  var counter MessagesSent

  /* Print out Information about each. */
  PrintValueData(0)
  PrintValueData(nil)
  PrintValueData(NewMailMessage)
  PrintValueData(m_std)
  PrintValueData(&m_ptr)
  PrintValueData(counter)
}

Running the Application

Now we have a complete application.  Let’s see what happens when we run it.

$ gogogo reflect.go
2011/02/24 00:30:45 Found int/int with 0 methods.
2011/02/24 00:30:45 nil value, nothing to do
2011/02/24 00:30:45 Found func/ with 0 methods.
2011/02/24 00:30:45 Discovered struct/MailMessage that has 5 fields and 1 methods.
2011/02/24 00:30:45 Discovered a pointer with 0 methods, dereferencing...
2011/02/24 00:30:45 Discovered a pointer with 3 methods, dereferencing...
2011/02/24 00:30:45 Discovered struct/MailMessage that has 5 fields and 1 methods.
2011/02/24 00:30:45 Found int/MessagesSent with 0 methods.
$

Wow, neat. This output is almost what we expected.  Let’s again go through this line-by-line and look at why we’ve got what we have. Don’t worry if method information seems a little odd to you, we’ll double back to that afterwards.

1. Handled by our default case. This is simply an integer with zero methods.
2. Handled by our nil test.
3. Handled by our default handler. Go identifies it as a function (with no type name) that has zero of its own methods.
4. Handled by our structure case. We pass in m_std in our main function, which is a pass-by-value function. Go tells us that this structure has five fields and one method associated with it.
5. Go finds a pointer with zero methods attached to it, and dereferences.
6. Go finds a pointer with one three methods attached to it, and dereferences.
7. Go uncovers what we pointed to by &m_ptr (the address of a pointer to a structure of kind MailMessage). It correctly prints the same information as noted on line four.
8. Go prints that we have a type MessagesSent with a underlying kind of int, with zero attached methods.

Ok, what’s wrong with this picture? Anyone paying attention? We defined two methods that take a pointer to a MailMessage and one method that takes a MailMessage by value. However, when we dereferenced down to a MailMessage structure, we only accounted for one method. At the same time, we have a pointer (m_ptr after dereferencing &m_ptr), that has three methods! Why?

So, in Go, a pointer to a type’s set of methods is the union of all methods both on the pointer type as well as the pointed-to type. So, *MailMessage has methods SetBody, SetPriority, and SendMessage associated with it. Of those methods, only SendMessages takes a MailMessage by value, thus it only has one method associated with it.

Final Thoughts

So, at this point, we’ve covered a bit about reflection. The Go documentation on this is slightly sparse.  I pieced this entry together using both the Go reflect documentation, and the source code to the Go json module. So, if I’m off base on anything, please leave a comment and correct me.

Next, it is possible to dynamically dispatch methods using the Call method of a reflect.FuncValue object. This method takes a slice of reflect.Value and returns a slice of reflect.Value. Rigging this up would be an interesting experiment.

Lastly, it is not possible in Go to create new types at runtime.

As I got into the RPC code, I realized that underlying data serialization was handled by the gob module. The gob module appears to be analogous Python’s pickle approach.  Given that, it seems that the gob foundation ought to exist before I spend any amount of time on understanding the RPC mechanism.  So, that’s what this post is about. We’ll put together a module that serializes structure data. Of course, we’ll also provide reader functionality in order to close the loop.

Since we’re getting a little fancier (two files this time), the little gogogo test script won’t work here. Instead, we’ll have to do it the old fashioned way.  I promise we’ll just add a Makefile if we add a third file. That’s probably worth a post itself.

The gob Package

The documentation for the package can be found on the Go Language site. Based on those pages, it’s a rather flexible system.  Instead of terminating or triggering an error condition in many cases, Go will make an effort to follow pointers and do other creative things such as leave out elements if they’re not present in the type we’re attempting to reconstruct.  This is probably something to take note of as it could be slightly surprising if it is not a behavior you’re ready for.

The documentation on the encoding format is quite complete and it covers things such as integer encoding, structure formatting, and the handling of signed values.

Manipulating Gob Data

Our example application isn’t terribly complex.  It is composed of two files. The first file, goopy.go, contains all of the code necessary for handling the serialization.  The second file, main.go, acts as the driver and provides an interface to the user.  Here’s  quick run down of the application flow.

1. Startup and flag processing. Default to read.
2. If reading, load the gob file from disk and display it to the screen. We take care to handle missing files gracefully.
3. If writing, we encode the data in question and stream it to disk.
4. Application terminates, ensuring all files are closed.

The Stuff at the Top of the goopy.go File

This isn’t overly interesting.  We just include package dependencies and define a structure type that we’ll use for serialization.

package goopy

import "gob"
import "os"

type PersonInfo struct {
  Name string
  Age int
}

Serialization

The following method handles the serialization of data to a local file. The function accepts a pointer to a PersonInfo as its receiver and a path for the target data file.  We’ll return something implementing the os.Error interface if we run into a problem. Otherwise, the return value is nil.

func (p *PersonInfo)SerializePersonInfoToGob(to_file string) (os.Error) {
  /* Open file and check for error state */
  file_handle, err := os.Open(to_file, os.O_WRONLY|os.O_CREAT, 0600)
  if err != nil {
    return err
  }

  /* Automatically close when we finish in this function, consider
   * with open(to_string) as file_handle. */
  defer file_handle.Close()

  /* Serialize data out. */
  gob.NewEncoder(file_handle).Encode(p)
  return nil
}

Let’s step through this. It’s not overly confusing, but there are a couple of concepts covered here.

1. The first line opens the target file. Notice the similarities between the flags here and the flags available to the standard open(2) system call. This function returns two values. An *os.File, and a possible error condition.
2. We check the error condition. If it’s non-nil, we just return it.
3. Next, since the file was opened, we call ‘defer file_handle.Close().’ What’s this do? When a method exits, for any reason, anything that has been marked with the defer keyword will execute. This ensures that our file is closed.  If you’re familiar with Python, think “leaving a with statement.”
4. Next, we create an anonymous encoder and call the Encode method. We pass file_handle. Note that NewEncoder function expects something implementing the io.Writer interface. That is, anything with a Write method. Calling Encode flushes the serialized data down.

That’s it. We then return nil to signify a successful encoding.

Deserialization

Deserialization is a lot like the serialization process.  Here’s the code that does that for us.

func NewPersonInfoFromGob(from_file string) (*PersonInfo, os.Error) {
  file_handle, err := os.Open(from_file, os.O_RDONLY, 0600)
  if err != nil {
    return nil, err
  }
  defer file_handle.Close()
  var person PersonInfo

  decoder := gob.NewDecoder(file_handle)
  err = decoder.Decode(&person)
  return &person, err
}

Everything should look quite familiar. Note that we now follow the “value, error” idiom here as well. This is obvious as the returned elements are a pointer to a PersonInfo and something implementing os.Error. The other thing to notice is that we pass a pointer to our PersonInfo variable to the Decode method. This ensures pass-by-reference. Our structure will be populated by the deserialization routine.

Creating New Objects

We add one little helper function to create new PersonInfo objects.

func NewPersonInfo(name string, age int) (*PersonInfo){
  return &PersonInfo{Name: name, Age: age}
}

The main.go File

Now, we put together our driver code that handles user options.  Let’s look at this as one big listing and step through the interesting points.

package main

import "log"
import "flag"
import "./goopy"

/* File we read and write from. This is required in any case. */
var gobfile *string = flag.String(
  "gobfile", "data.gob", "A Go Pickle of a Different Flavor.")

/* Read mode? We default to this in order to be non-destructive.
 * we'll pull Gob data out and just dump it to the screen.
 */
var writing *bool = flag.Bool(
  "writing", false, "If specified, we write the command line data.")

/* Required only in writing scenario */
var age *int = flag.Int("age", -1, "Age for person record.")
var name *string = flag.String("name", "", "Name for person record.")

func main() {
  defer func() {
    if err := recover(); err != nil {
       log.Println("Fatal Error Encountered: ", err)
    }
  }()

  flag.Parse()

  if *writing {
    person_info := goopy.NewPersonInfo(*name, *age)
    person_info.SerializePersonInfoToGob(*gobfile)
    log.Println("Serialization Complete")

  } else {

    person_info,err := goopy.NewPersonInfoFromGob(*gobfile)
    if err != nil {
      panic(err)
    }

    log.Printf("Read Complete (Dump): %+v", person_info)
 }
}

Most of this should look familiar if you’ve followed my other posts to date. There are a few newer concepts (to me, as well!) here, so again, let’s walk through.

1. Required imports and package name. Notice the “./goopy” syntax used. This is because our goopy module is located in the current directory and not in a centralized library location.
2. Next up, we setup four command line flags. The file to perform IO on, whether we want to write, and then the values for our PersonInfo structure.
3. Now, look at the first couple of lines of our main function. Different, no? Here, we defer an anonymous function that calls recover(). If you look back at our deserialization routine, you’ll notice that we’ve called panic.  Panic causes the stack to unwind and the application to exit.  If, along the way, a recover call is executed, the error passed to panic is returned and control resumes at that function return.  Note that since the stack is unwinding, the only valid place (I believe) to stick a recover call is in a deferred function.  In this case, we just use it to print our error condition and exit.
4. We then parse the flags as we’ve done before.
5. If we’re writing, we call goopy.NewPersonInfo and then proceed to serialize that information.
6. Otherwise, we default to reading our data out of file and displaying it to the screen.

That’s fundamentally it in terms of our application. Notice that we’re using the log.Printf and log.Println functions here. That’s nice as it causes data printed to standard output to be prefixed with the date and time.

Compiling and Running

While this isn’t as straightforward as our previous tests were, it’s not difficult. First we’ll compile our application, link it, and display the command line options.

mcjeff@martian:~/my_go$ 6g goopy.go
mcjeff@martian:~/my_go$ 6g main.go
mcjeff@martian:~/my_go$ 6l -o gobber main.6
mcjeff@martian:~/my_go$ ./gobber --help
flag provided but not defined: -help
Usage of ./gobber:
  -writing=false: If specified, we write the command line data.
  -name="": Name for person record.
  -gobfile="data.gob": A Go Pickle of a Different Flavor.
  -age=-1: Age for person record.

First, if we run our application without a valid file, we’ll see our error handling in action.

mcjeff@martian:~/my_go$ ./gobber -gobfile=/no/such/file
2011/02/17 15:43:19 Fatal Error Encountered:  open /no/such/file: no such file or directory

Now, let’s run it again with the proper command line arguments needed to write new gob data out.

mcjeff@martian:~/my_go$ ./gobber -writing=true -name=jeff.gob -age=31 -gobfile=jeff.gob
2011/02/17 15:45:12 Serialization Complete
mcjeff@martian:~/my_go$ ls -l jeff.gob
-rw------- 1 mcjeff mcjeff 58 Feb 17 15:45 jeff.gob

Beautiful! It ran as it should, we created the file, and populated it with our command line data.  Finally, let’s run our new utility one more time in order to read the data from disk and ensure it prints the proper contents.

mcjeff@martian:~/my_go$ ./gobber -gobfile=jeff.gob
2011/02/17 15:46:51 Read Complete (Dump): &{Name:jeff.gob Age:31}

That’s all there is to it!

• inotify_init – Creates an instance and returns a file descriptor referring to that instance.
• inotify_add_watch – Updates the watch list associated with an inotify instance.  Based on its arguments, it configures the instance to let us know when certain events happen.
• inotify_rm_watch – The inverse of inotify_add_watch.
• read – Wait, read? Yeah. The application read()’s on the inotify file descriptor, which blocks until events occur. File descriptors can be monitored with the standard set of primitives — poll, select, et al.
• close – Calling close on an inotify file descriptor destroys the instance.

All in all, it’s a fairly straightforward API.  Watches are configured with bitmasks and events are reported by feeding event structures to the read file descriptor.  For more information, see the Wikipedia page.

Where is it used? Well,  consider desktop indexing applications. For a lower level reference, the udevd system monitors udev rules via inotify and automatically reloads them when changes are detected.

Our Example Application

We’re not going to write anything as fancy as udevd, or as useful as a desktop indexing daemon.  Instead, we’ll put together a simple Go application that uses inotify integration and a collection of channels to calculate the size of files written to a directory. Nothing fancy.

What we Stick at the Top of the File

First thing’s first. We simply name our package and import others that we depend on.

package main

import ("flag"
        "fmt"
        "os"
        "os/inotify"
)

var watch_target *string = flag.String("watch", ".", "The directory to keep an eye on")

The only other thing going on here worthy of mention is the flag.String line. Since we want our tremendously useful application to be configurable, we’re using the flag module to set up a string flag.  This line sets up command line argument processing. The watch_target variable will contain the argument value as set by ‘–watch’, or the default, once control is passed to main(). The Go runtime will also automatically setup the “–help” flag as well, which will print the help strings for all arguments defined. Neat.

Our Changed File Structure

When we fire up our application, we want to read change events, pass them to a goroutine, and then simply forget about it.  When we pass the changes off, we need to tell our system which file changed as well as what to do with our result output.  We define the following structure.

type ChangedFile struct {
  output_channel chan string
  filename string
}

This simply contains two elements. An output channel that we’ll stream our results to and the filename of the changed entity.

Reporting Output

Next, we define a function that handles printing output to the screen. We run this in it’s own goroutine to keep concurrent routines from stepping on each other.

func notificationReporter(input chan string) {
  for {
    queue_data := <-input
    if len(queue_data) == 0 {
      return
    }
    fmt.Println("Change Received: ", queue_data)
  }
}

This is fairly straightforward. We pass in the channel in which we’ll read our input from. Next, we enter into an infinite loop. We just print the data to standard out.  However, if we get a zero-length string, we terminate the routine.  Though this isn’t really going to happen in our little example as we’re not rigging up sentinel values. We just quit on a Control+C.

The Change Processor

Next, we put together the method that handles processing of each individual change. It’s a method because it has a specific receiver, without it, we’d call it a function.  Here we stat the file changed, build a log string, and send that value to the channel attribute of the receiver.

func (c *ChangedFile)ProcessChange() {
  statbuf, _ := os.Stat(c.filename)
  c.output_channel <- fmt.Sprintf("%s is %d bytes", c.filename, statbuf.Size)
}

As you may have guessed by now, c.output_channel is the other end of the channel that we read from in the above notificationReporter function.  We’ll wire it all up in our main driver method.

The Main Method

Let’s just take a look at this one and then step through it afterwards.

func main() {
  flag.Parse()
  var report_channel = make(chan string, 5)
  go notificationReporter(report_channel)
  watcher, _ := inotify.NewWatcher()
  watcher.AddWatch(*watch_target, inotify.IN_CLOSE_WRITE)
  for {
    select {
      case event := <-watcher.Event:
        change := &ChangedFile{report_channel, event.Name}
        go change.ProcessChange()
      case error := <-watcher.Error:
        panic(error)
    }
  }
}

First off, we create a channel named report_channel, with a buffer of 5. This allows us to queue up to 5 results for writing before the enqueuing would block on a full “pipe.” Next, we fire off the notificationReporter function defined above. By using the go keyword, we allow it to run concurrently with our main line of control.

Next up, our inotify.Watcher is created. We then “load” the watcher with an event.  This takes two arguments – the directory we want to watch and the events we care about.  In this example, we just care about file close events for files that were written.

The for loop is slightly interesting.  Here you’ll see a Go select block. The select system is much like a switch statement, however, the individual cases are channels or IO endpoints.  If the tested object is ready to perform IO, then the following block will execute.  A default block could also be created, which would fire if none of the specific cases match. Finally, the select group will block until it is explicitly exited.

Within this loop, we create a new ChangedFile and configure it with our reporting channel and the name of the file that has been modified. We then execute change.ProcessChange by using the go keyword again, which causes it to run concurrently with the main line of control and the reporting function. We now have three concurrent paths executing at the same time.

Lastly, we just panic on error instead of building proper error handling in.

The Output

Alright, we’ll first run our code using the little build script we stuck in our ~/.profile file in the first Go blog entry.

[jeff@martian ~]$ gogogo test.go

There’s not a whole lot going on there. If everything works correctly, the code will compile and execute. From there, it will just block waiting for directory change events.  In another window, we’ll write a few files.

[jeff@martian ~]$ touch a b c d e; echo "bytes" > f

Alright, now if we switch back to the other window, we’ll see something interesting.

Change Received:  ./a is 0 bytes
Change Received:  ./b is 0 bytes
Change Received:  ./c is 0 bytes
Change Received:  ./d is 0 bytes
Change Received:  ./e is 0 bytes
Change Received:  ./f is 6 bytes

Well, that’s it. As each file is written and closed, our events are handled as outlined above.  When the fun is over, simply terminate the Go application with a Control+C.  As I learn a bit more about Go, I’ll get to closing down channels properly and cover that in a later post. Additionally, we didn’t close down our inotify handle correctly, either.  We rely on the operating system to handle all of that for us when the process terminates.

Being the type of guy that doesn’t like to learn anything at the surface level, I decided to look into how one would integrate an existing C library with a Go application. My curiosity was two fold. First, I hoped to learn a bit more about what lives under the Go-hood.  Secondly, being a new language, my assumption is that there are a lot of native libraries that haven’t yet been wrapped.

I spent a while trying to hunt down a simple, step-by-step guide that would lay out all of the steps needed to do this.  Python has excellent documentation when it comes to handing extending and embedding. I was hoping that I’d come across such a document written from a Go perspective, but I wasn’t able to find one.  Using examples and tool documentation, however, I was able to get some test code put together. Below you’ll find a rundown of what I did and how I did it.

My goal was to link in Python and execute Python code from within a Go executable. I wouldn’t recommend ever doing this for any reason, ever. Actually, you may want to just stop reading now.

Using the cgo Compiler

When building Go code that needs to interface with an existing C library, you’ll compile using the cgo compiler instead of the usual ${num}g.  So, the approach I took was to isolate all of that code into a specific library and then access those functions as I would any other package.

The “C” Package

Generally speaking, C provides a flat, non-hierarchical namespace. Protection is limited to source file scope and is dictated by the programmer’s use of the static keyword. Simply put, everything is visible everywhere unless declared static. This is why C API definitions usually start with an identifying prefix. For Python developers, consider the Py_ prefix. It provides a way to create independent namespaces (though, with no protection between them).
When interfacing with C libraries, Go wraps everything C under the “C” package.  For example, printf as defined in stdio.h becomes C.printf. This provides a nice demarcation between the Go world and the everything else world.

Including Headers

So, now we know what we’ll use to build the code and how we’ll interface with it.  The next step is to ensure Go knows where it can find the necessary headers.  To do this, we embed the information in comments immediately preceding the import “C” line.

// #include <Python.h>
import "C"

This ensures that the definitions in Python.h are visible to cgo. Everything found here will be made available under the C namespace and is accessible to code within this source module.  Now, as an aside, I noticed that C preprocessor macros were not expanded. So, because PyRun_SimpleString is simply a macro that calls PyRun_SimpleStringFlags, I had to use the latter.

The Makefile

We’ll be using the following makefile.  This comes from the example code under $GOROOT/misc/cgo/gmp.

# Copyright 2009 The Go Authors.  All rights reserved.
# Use of this source code is governed by a BSD-style
# license that can be found in the LICENSE file.

include ${GOROOT}/src/Make.inc

TARG=pygo

# Can have plain GOFILES too, but this example doesn't.

CGOFILES=\
  pygo.go

CGO_LDFLAGS=-lpython2.6
CGO_CFLAGS = -I/usr/include/python2.6

# To add flags necessary for locating the library or its include files,
# set CGO_CFLAGS or CGO_LDFLAGS.  For example, to use an
# alternate installation of the library:
# CGO_CFLAGS=-I/home/rsc/gmp32/include
# CGO_LDFLAGS+=-L/home/rsc/gmp32/lib
# Note the += on the second line.

CLEANFILES+=pygo

include ${GOROOT}/src/Make.pkg

# Simple test programs
pygo:
  $(GC) pygo.go
  $(LD) -o $@ pygo.$O

There’s a not very much going on here. If you compare this file to the example, we’re simply changing a few internal variables.

• TARG is the name of the library we’ll be building.
• CGO_LDFLAGS & CGO_CFLAGS are generally the same as LDFLAGS & CFLAGS. They’ll be passed on during the compilation and linking phases.
• CLEANFILES is pretty self-explainatory.

Finally, we add the pygo rule at the bottom and remove the example rules for gmp.

The pygo Package

Now that we have our Makefile setup and ready to go, we need to put together our actual Go package. Below you’ll find the listing as I used it. Note that in a few places I purposefully used the long way as a learning tool.

package pygo

/* "C" isn't a real package. Rather, it's a virtual one created when building
 * via cgo. Cgo, as opposed go 6g, allows us to access C code/libraries from
 * within a Go application. Everything in the C-world is then jammed under the
 * 'C.' namespace.
 */

// #include <Python.h>
// #include <pythonrun.h>
import "C"

/* Some basic file IO helpers live here. */
import "io/ioutil"

/* Create a PythonCode object, which is a struct, containing
 * just one field. There's no reason this couldn't have
 * just been a type PythonCode string. When I started,
 * my intention was to be a bit more complicated and store
 * return values and interpreter state and all that, but
 * there's really no point.
 */
type PythonCode struct {
  code string
}

/* In this case, a pointer to a PythonCode object is the receiver,
 * our function is called ExecPy, and there is no return value.
 * we call our internally defined init/exec/term functions.
 *
 * Two things to note here:
 * 1. i/e/t are lower case, which means that they are the equivilent
 *    of a C top level 'static void F().' Not visible outside of
 *    the package (Encapsulation FTW!).
 *
 * 2. This is a METHOD because it has a defined receiver.
 */
func (p *PythonCode) ExecPy(result chan int) {
  initPython("PYTHON_EXEC")
  py_response := execPython(p.code)
  termPython()
  result <- py_response
}

/* This is our init method. There is no __init__ or __del__
 * equiv.  This builds a new object and returns a pointer to it.
 * Note that here, it's 100% okay that we're returning what
 * appears to be a pointer to a local variable. I know,
 * makes me feel a bit uncomfortable, but the docs clear
 * that up.
 */
func NewPythonCode(src *string) *PythonCode {
  contents, _ := ioutil.ReadFile(*src)
  return &PythonCode{code: string(contents)}
}

/* Here we tickle the Python C API. We set the program name,
 * initialize state, and ensure we init correctly.
 */
func initPython(interp_name string) {
  C.Py_SetProgramName(C.CString(interp_name))
  C.Py_Initialize()
  if C.Py_IsInitialized() == 0 {
    panic("Could not init interp.")
  }
}

/* This executes the code. This probably could also be a method on
 * our structure-based type above, but I wanted to highlight
 * the differences between a Go method and a Go function.
 */
func execPython(python_code string) int {
  return int(C.PyRun_SimpleStringFlags(C.CString(python_code), nil))

}

/* Fall Py_Finalize and free up interp. data structures. Again,
 * this probably would work better using an interpreter Go
 * object and then a method to pass in code to exec, but this
 * covers more ground
 */
func termPython() {
  C.Py_Finalize()
}

There’s quite a bit going on in there, so let’s step through it in a bit more detail.

1. The first few lines are straight forward. We name our package pygo, include the correct C headers, and import io/ioutil.
2. Next, we create a PythonCode type, which is struct. This object simply contains a string which will be actual Python code.  As the comment states, we could have done this simpler, but I think the struct approach is probably more representative of idiomatic Go (correct me, someone?).
3. Next, we define ExecPy. Since it has a capital ‘E’, it is exported from the package and available elsewhere.  This method does three things, all by calling other functions internal to the package.  Notice that it takes a channel of int objects as its sole parameter. We pass the return value of execPython back up the channel instead of simply returning it.  In the real world, a return value would be simpler, however, this illustrates the usage of channels.
4. After that, we have NewPythonCode. As Go doesn’t provide initializers/constructors, we simply do the build in an appropriately named factory function like this.  Here, you see the calls directly into the Python library. In this case, C.Py_SetProgramName, C.Py_Initialize, and C.Py_IsInitialized. In actuality, Go is simply calling Py_Initialize, Py_SetProgramName, and Py_IsInitialized. The other thing to note here is the use of C.CString(). This converts a Go string object into a C char *.
5. Next up is execPython. Here, we actually return the int value from C.PyRun_SimpleStringFlags. The int call/cast is necessary as a C int is not the same as a go int. Again, you’ll see the call to C.CString converting from string to char *.
6. Finally, we simply call Py_Finalize from within termPython.

Pretty simple if you ask me.  I’m becoming a fan of the “C.$identifer” approach as it allows code authors to avoid having to write stub modules in C. (SWIG Status, anyone?).

The Driver

The driver is pure Go. There’s no C code whatsoever. We simply import our new module and call it as we would any other Go library.

/* Everything goes into a package. */
package main

/* Could import individually, but this is cleaner. Sort of like wrapping Python
 * imports in order to group them on multiple lines.
 */
import ("pygo"
  "flag"
  "fmt"
  "stdio"
)

/* Flag Parsing. This is silly easy. */
var pythonSource = flag.String("pysource", "test.py", "Python Source File")

/* Main entry point. Just like C/C++ */
func main() {
  flag.Parse()

  /* Print a goofy little header so we can visualize the Go exec
   * vs. the Python exec.
   */
  fmt.Printf("Exec Python: %s\n", *pythonSource)
  fmt.Printf("-----------------------------\n\n")
  stdio.Stdout.Flush()

  /* Create a channel, which is like a queue. We'll block on
   * this for our Pythoning to finish.
   */
  ch := make(chan int)

  /* Create, set code, Exec Py. But.. do it in a different thread
   * of control! All we did was fire it after the keyword "go."
   * Sweet.
   */
  go pygo.NewPythonCode(pythonSource).ExecPy(ch)

  /* This says "fill python_result with whatever comes out
         * of the channel." This is a blocking call.
   */
  python_result := <-ch
  fmt.Printf("\n\n-----------------------------\n")

  /* Another way of printing... */
  fmt.Println("Channel Returned Code: ", python_result)
}

That’s all there is to it. Again, let’s walk through this source listing.

1. We name our package and import our dependencies. Note that our new package appears here and it is listed like any other package.
2. Next we setup a –pysource flag, so we can specify a Python file on the command line. This lets us run arbitrary Python code from our Go executable.
3. We print a header and flush standard output. This is to ensure Go finishes printing (buffered?) before Python starts running.
4. Next, we create a channel of integers. The channel is used to send the result of our Python execution back to our driver code. Note the use of the make function versus the new function here.
5. The next line passes the location of the Python source file to a new PythonCode object (that we created via NewPythonCode). See how this call is prefixed with the “go” keyword? This allows the Go runtime to run both the driver code as well as the Python system concurrently.
6. Finally, we wait on our channel and print the exit code.  If we didn’t wait, the driver would complete execution before our Python library ran. In short, much like terminating the main thread before a worker thread has a chance to complete.

The Test Code

The following snippet of Python code is what I used to test. The metaclassery is not needed, but I was curious as to whether it would cause any issues (I didn’t think it would!).

import sys

class GoofyMeta(type):
  def __new__(*args, **kw):
    print "In Go->Python Metaclass"
    return type.__new__(*args, **kw)

class HelloFromGo(object):
  __metaclass__ = GoofyMeta
  def __str__(self):
    return "Hi, you are running Go."

if __name__ == '__main__':
  h = HelloFromGo()
  print h

  print "See, not kidding: "
  print sys.version

  print "Here's my global namespace: "
  print dir()

So, as a control, let’s run the Python code from the command line first, to ensure it runs as we think it should.

mcjeff@martian:~/pygo$ python test.py
In Go->Python Metaclass
Hi, you are running Go.
See, not kidding:
2.6.5 (r265:79063, Apr 16 2010, 13:57:41)
[GCC 4.4.3]
Here's my global namespace:
['GoofyMeta', 'HelloFromGo', '__builtins__', '__doc__', '__file__', '__name__', '__package__', 'h', 'sys']
mcjeff@martian:~/pygo$

Now that we’re certain everything works, we can move forward with compilation and testing.

Compilation and Running

For this example, I’m going to use the same little shell function I used in my first Go post. Everything else is new.  Let’s take a look at what’s required to build this extension.

mcjeff@martian:~/pygo$ export GOROOT=~/go
mcjeff@martian:~/pygo$ make
CGOPKGPATH= cgo -- -I/usr/include/python2.6 pygo.go
6g -o _go_.6  pygo.cgo1.go _cgo_gotypes.go
6c -FVw -I"/home/mcjeff/go/pkg/linux_amd64" _cgo_defun.c
gcc -m64 -g -fPIC -O2 -o _cgo_main.o -c -I/usr/include/python2.6  _cgo_main.c
gcc -m64 -g -fPIC -O2 -o pygo.cgo2.o -c -I/usr/include/python2.6  pygo.cgo2.c
gcc -m64 -g -fPIC -O2 -o _cgo_export.o -c -I/usr/include/python2.6  _cgo_export.c
gcc -m64 -g -fPIC -O2 -o _cgo1_.o _cgo_main.o pygo.cgo2.o _cgo_export.o -lpython2.6
cgo -dynimport _cgo1_.o >__cgo_import.c && mv -f __cgo_import.c _cgo_import.c
6c -FVw _cgo_import.c
rm -f _obj/pygo.a
gopack grc _obj/pygo.a _go_.6  _cgo_defun.6 _cgo_import.6 pygo.cgo2.o _cgo_export.o
mcjeff@martian:~/pygo$ make install
cp _obj/pygo.a "/home/mcjeff/go/pkg/linux_amd64/pygo.a"
mcjeff@martian:~/pygo$

That’s it. The files included in our Makefile handle the setup and compilation for us. There are a series of intermediary files created that we really don’t have to concern ourselves with.  If you’re curious, they’re left around by the compiler. A simple ‘ls’ will display them.

Finally, we can run our Go binary and execute our small Python example.

mcjeff@martian:~/pygo$ source ~/.profile
mcjeff@martian:~/pygo$ gogogo main.go
Exec Python: test.py
-----------------------------

In Go->Python Metaclass
Hi, you are running Go.
See, not kidding:
2.6.5 (r265:79063, Apr 16 2010, 14:15:55)
[GCC 4.4.3]
Here's my global namespace:
['GoofyMeta', 'HelloFromGo', '__builtins__', '__doc__', '__name__', '__package__', 'h', 'sys']

-----------------------------
Channel Returned Code:  0
mcjeff@martian:~/pygo$ file main
main: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), not stripped

Well, there you have it. A quick run down on how to access C libraries from a Go application.  I find it rather elegant, honestly. Now, as for the embedded Python?  I wouldn’t use this anywhere, ever, for any reason. I’ve no idea what’s going on with regards to internal thread states and structures. This is nice little example, but probably not such a hot idea!

• It’s quite expressive. This was one of the main things that got me into Python. I liked the fact that I could squeeze a lot of functionality out of just a few lines of well thought out code. Go provides that same functionality and it gives near-C performance (though, of course, there is overhead).
• It’s a small language. This is why I’ve always preferred C to C++. C is a very, very small language. There isn’t much to it at all. Once one gets the idioms down, C programming is not overly difficult. Some of the OS level work might be, but the language itself is very small.
• It’s quite easy to extend.  I actually wrapped Python in Go this afternoon after only about an hour of effort. The cgo compiler provides very straightforward means of integrating and linking native C code.
• It provides a very intuitive and easy-to-use threading/parallelism model with the use of goroutines.

What’s there not to like? Exactly. While the standard library has yet to read “batteries included” status, it is getting there. Now, you’ll find modules for SMTP, networking, HTTP,  and IO, among other things.  There’s even a growing collection of third party modules available on the Package Dashboard.

After reading through the available documentation, I decided to try my hand at a few examples. I couldn’t very much decide where to start, so I opened the standard library reference and started with ‘A.’  In this case, that means the archive packages – tar and zip.

Reading Tar Files with Go

First off, if you don’t have Go installed as of yet, head over to the getting started guide. Installation is source driven, but it’s truly very simple.  The compilation is slightly bulky as you’ll need to compile, link, execute.  That’s a cycle I haven’t used in a while.  In order to speed it up, I’ve defined the following shell function in my .profile.

gogogo() {
  source=$1
  base=$(basename $source .go)
  6g $source
  6l -o $base $base.6
  $(pwd)/$base
}

Of course, this little rule only works when building one file at a time.

Now, let’s setup the environment.  This would work with much larger tar files, but for example purposes, we’re going to limit ourselves to some very small ones.  The test environment was setup as follows:

$ echo "one" > 1
$ echo "two" > 2
$ echo "three" > 3
$ tar -cvf test.tar ./[123]
a ./1
a ./2
a ./3
$

Notice that we’ve simply created three text files, named 1, 2, and 3. Each file contains its english representation.  Let’s spice it up a bit and change the contents of “1″ to один, which is simply the Russian word for “one.”

$ echo 'один' > 1
$ cat 1
один
$

Alright. Now, let’s have a look at the Go code needed to read from the tar file we created.

package main

import "archive/tar"
import "fmt"
import "os"

func main() {
  f,_ := os.Open("test.tar", os.O_RDONLY, 0600)
  reader := tar.NewReader(f)
  file_count := 1
  for hdr,err := reader.Next(); err == nil; hdr, err = reader.Next() {
    b := make([]byte, hdr.Size)
    fmt.Printf("%d: %s (%d bytes)\n", file_count, hdr.Name, hdr.Size)
    reader.Read(b)
    fmt.Println("-----> ", string(b))
    file_count++
  }
}

That’s all there is to it. Before we go into detail and explain exactly what’s going on here, let’s built it (using our shortcut) and run in order to verify the output.

$ gogogo tar.go
1: ./1 (9 bytes)
----->  один

2: ./2 (4 bytes)
----->  two

3: ./3 (6 bytes)
----->  three

As an aside, if you run an ls from the current directory, you’ll see the intermediate files needed by the go compiler. We’re just skipping that step with our profile function. Let’s take a closer look at the code, and then double back and review the results.

The very first line of the file declares that we’re defining package main. Each Go source file requires a package name. Larger applications are simply collections of linked packages. The idiomatic name for the main package is, not suprisingly, main.

Next we import three more packages. These are all used within the file. Note that if we were to import a package that we’re not using, the application would not compile. That ensure we always have clean code without a polluted namespace.

Now, we define a main function. This is the application entry point, much like you’ll see in a C, C++, or Objective-C program.

1. We open a file by using the os.Open function. In Go, identifiers are only exported from packages if they begin with a capitol letter (think static vs. non-static C functions).  The os.Open function returns a file object.
2. Next, we create a tar reader. The NewReader function expects an object implementing the Reader interface. Note that with Go, there’s no need to declare what you’re implementing. If you add the required methods, you automagically implement the interface. I like this approach. Duck typing light, if you will.
3. We then iterate through the contents of the tar file by calling reader.Next. In Go, functions and methods can return more than one value. It’s common to see the actual value and a possible error condition passed back. As long as the error condition is not nil (Go’s None/NULL), we keep reading. The reader.Next method returns a header structure as well as an error value.
4. Now, we create a slice of bytes. The make syntax creates a slice and an underlying array. For more information, see the Go documentation on arrays and slices.
5. We print some status information, read the full contents of each element in the tar file, and print the results. We increment our file number counter so we can display how many files we’ve read.

Just for reference purposes, the Tar header is defined as the following in archive/tar:

type Header struct {
    Name     string
    Mode     int64
    Uid      int
    Gid      int
    Size     int64
    Mtime    int64
    Typeflag byte
    Linkname string
    Uname    string
    Gname    string
    Devmajor int64
    Devminor int64
    Atime    int64
    Ctime    int64
}

Pretty straightforward, no?  Now, double back and look at the results from earlier. They should all make sense. Well, maybe except for our little Russian file! Note that this record says that it is 9 bytes in size, while the remaining files state that they are equal to the number of files plus the trailing newline.  The answer is pretty simple. Each Cyrillic letter takes up two bytes when UTF-8 encoded. Go transparently handles that complexity for us. So, we’re looking at four two-byte letters, followed by a standard newline.

$ file 1
1: UTF-8 Unicode text
$ file 2
2: ASCII text
$

So, all in all, I’m starting to like this language. Take a minute and dive into it. Now, of course, the challenge becomes finding something worthwhile to write in Go. I’ve learned a small collection of languages over the past year, only to forget most of them due to lack of use. As an interesting aside, I had been planning on refreshing my C/C++. I think I may defer that a bit and really spend some time on this language.

Update: I’ve been asked how one woud manage a gzip compressed tar file.  Well, remember interfaces? The reader we pass into the tar.NewReader function simply has to implement the Reader interface! So, we can update the code above to open a gzip file and pass that reader in, like so:

package main

import ("fmt"
        "archive/tar"
        "compress/gzip"
        "os"
       )

func main() {
  fhandle, _ := os.Open("test.tar.gz", os.O_RDONLY, 0600)
  zhandle, _ := gzip.NewReader(fhandle)
  thandle := tar.NewReader(zhandle)
  hdr, _ := thandle.Next()
  fmt.Println(hdr.Name)
}

See? All we’ve done here is chain our NewReader calls, as each returned object implements the Reader interface.  Running the code provides the following output.

mcjeff@macbook:~$ ls test.tar.gz
test.tar.gz
mcjeff@macbook:~$ gogogo gunzip.go
./1

There. Hope that clears it up!

I’ve recently finished writing my first book, Python 2.6 Text Processing Beginner’s Guide. What an interesting, yet demanding experience. Did I learn a lot? You bet. Would I do it again? Maybe. It’s a large time commitment; one I’m not really interested in again anytime soon. After all, I’ve got monsters to slay and levels to attain. One can’t invest in Dwarven Shaman while being productive!

The Process

A long, long, long time ago, I was contacted by the Packt staff and asked if I would be willing to write a book about Python text processing. Being overly excited at the prospect of doing so, I agreed.  They did a pretty good job of ironing out the process. Here’s an outline of what one has to go through when writing a book for Packt.

1. The outline. The first element I had to come up with was a 4-5 page summary of what I intended to cover. Being as this was a beginner’s guide to text processing, I elected categories such as XML, PyParsing, Nucular, and some introductory material to the Python IO system (after all, you need to read it to parse it). Working chapter titles and “example examples” were included here.
2. Initial drafts. After acceptance of my outline, the contract was finalized and it was time to crank out my first drafts.  As there were 12 chapters in the outline, I wrote 12 initial drafts. Each one took about three weeks and topped out at about 30 pages. Editor feedback trickled in as I worked through.
3. Chapter rewrites.  After finishing the first drafts, I received the chapters back. This time, they were quite marked up both by the editing staff at Packt and the technical reviewers that Packt had selected.  This is the part I enjoyed the least. The goal? Clarification, fix some wording problems, generate screen shots (I had elected fixed-width fonts initially), and address feedback. Each chapter took about a week.
4. Final Reviews. After submitting my rewrites, the chapters were then transferred to PDF format and were sent my way for final review. Each took a couple of days.  I made quite a few corrections to these. At the time of this blog post, I’m still hoping they’ve all made it in the way I wanted them!
5. Publication.  The book was officially published on December 15th. The contract is dated March 11th, 2010. That’s a lot of  busy evenings in front of the iMac.

Along the way, I had to write things such as author bios, chapter summaries, introductions, sales points, and so on. I rather enjoyed the experience.   Though, I’ll be the first to admit that I was ready to be done around October.

The Product

I’m happy with the material covered.  I think it’s a good selection for a beginner’s guide. Not a lot of theory, but enough to clarify points when needed. My only regret? I would have loved to have gotten another round of drafts in, especially after the final review stage. But, I’m a bit of a perfectionist!

Chapter Summaries

Here’s a rundown of the table of contents and what was covered. This ought to paint a pretty clear picture. I made an effort to keep Python 3 in mind and even included a quick Python 2 to Python 3 upgrade example in the final appendix.

One of the additional goals I had was to cover some best practices while detailing methods of handling different types of textual data. For example, if we’re going to cover PyParsing in any detail, let’s go ahead and install it in a virtualenv environment.  Or, if we’re going to create some classes, let’s follow Python’s PEP8 and hopefully introduce a reader to some existing standards. The downside there? Fitting code on a printed page is quite difficult, especially when it’s more than a block or two deep. I’m not so happy with the number of spurious parenthesis or raging backslashes I had to use in order to display nicely on paper.

1. Getting Started covered introductory material such as running virtual environments and elements of text encoding (ASCII basics).
2. Working with the IO System brings readers up to speed on how to interact with external sources of data. File IO, network information, and the beauty of “file like.”
3. Python String Services covers what can be done using basic string methods.
4. Text Processing Using the Standard Library introduces a few common standard library modules for crunching data such as CSV and JSON.
5. Regular Expressions covers the Swiss Army knife.
6. Structured Markup details XML via SAX, DOM, and a bit of XPath. We also visit HTML parsing via standard library routines.
7. Creating Templates switches from parsing to producing. Using Mako to build and inherit text templates.
8. Understanding Encodings and i18n is an introduction to character encodings, Unicode, and Unicode data encodings (UTF). I’ve always felt that these concepts are confused more than they ought to be.
9. Advanced Output Formats goes over a bit of Reportlab, xlwt, and ODFPy.
10. Advanced Parsing and Grammars gives a bit of background on BNF and parsing techniques and then hands the heavy work off to PyParsing, with an example on the Python NLTK.
11. Searching and Indexing introduces Nucular and details how it can be helpful when searching large quantities of text data. Examples show how a linear search is much less efficient than an indexed approach.

All in all, I’m happy with those topics. I’ve also tried to include a lot of community information in the appendix. Conferences, users groups, PEPs, mailing lists, and so on.

While We’re Talking About Books…

I had the pleasure of playing technical reviewer for Cody Precord’s wxPython 2.8 Application Development Cookbook.

This was the first experience I’ve had technically reviewing a script like this. Out of the whole process, I learned a lot about wxPython! Historically, I’ve found UI development work less than entertaining. Why? It’s always seemed slightly monotonous to me.  More than likely that’s due to an unpleasant experience with AWT during a previous lifetime.

Reviewing this book gave me the chance to really get in and understand the inner workings of wxPython. You know what? It’s quite easy to build a respectable looking UI! There are a collection of tips and tricks, that I wasn’t aware of, that make it an enjoyable experience.  Give it a read. Check it out. I’m quite a fan of the “cookbook” style publications.

And Finally…

Packt is running a discount on all Python books this December, more information is available on their site. No more pushing or advertising on my part. I promise.

I’d love to answer questions about the whole process. While at PyCon 2010, I attended a workshop manned by a collection of established Python authors. I really learned a lot there; enough to look into getting started myself. If I can return the favor or pass on a bit of knowledge, please let me know!

This was a very well written book. Quite honestly, I enjoyed reading it. Really. I’m not just saying that because I didn’t have to pay for my copy. In fact, this is a book I probably would have went out and bought.  This is is a wonderful addition to the shelf of any systems administrator getting into Python development. It’s also a nice “Patterns in Python” reference for the more experienced developer. But, that said, something kind of bothered me while working my way through it. I really couldn’t put my finger on it until just now. More on that later.

Chapter Reviews

Dusty did a great job of really covering a bit of everything in about 350 pages. Let’s step through the chapters and go through the plusses and minuses, according to me.

Object Oriented Design

I really liked the fact that this was the lead in. As the title suggestions, this book is an object oriented view of the language. This chapter presents the composition vs. inheritance argument up front, touches on multiple inheritance, and even provides an introduction to UML. Now, that said, I rarely use UML. I use it infrequently enough that I have to look up syntax rules when it comes up.  I’m not so certain this is a necessary component of the book, but it’s certainly interesting and easy to understand.

Objects in Python

From an experienced Python developer’s point of view, I didn’t see as much value here. That doesn’t make this a bad chapter, though. We cover things such as classes, modules, and doc strings. All of these are very, very important Python attributes and need to be understood.  I’m just not entirely certain how much the experienced folks are going to take out of this chapter.

When Objects are Alike

Various flavors of inheritance, MRO calculation, super, argument list formatting. This chapter falls into that same trap for the experienced developer as the second chapter did.

Expecting the Unexpected

Exceptions. Raising, catching, handling, recovering from, and using as a flow control technique. Hey! I like that! I use them in my own code to signal things such as terminal conditions (StopIteration, anyone?), and not-so-exceptional exceptional cases.  I like the fact that a text like this points that out.

When to Use Object Oriented Programming

See, I liked this chapter. We’ve all seen the standard example. This is a bike. A bike has gears. Gears can shift. While this does cover the attributes & behaviors approach, it also lays an earlier foundation for things such as composition.

Python Data Structures

Build in data structures. I very, very rarely extend a built in object. I don’t think I’ve ever had a reason to do so. Each time I’ve tried, it’s felt like an academic exercise to see if I can correctly implement all of the __this__ and __that__ methods. Dusty goes into when it may or may not be appropriate to do so. I agree with what he has to say, I just prefer to not do that. Ever, really.

We also cover things like namedtuple. I love those little guys.

Object Oriented Shortcuts

Dunder methods and the built-ins that love them. Well, that and the comprehension collection, generators, default arguments, unpacking, functions as objects, callable objects, and a partridge in a Python tree. This chapter covers quite a bit, however it’s mostly at a superficial level. I would have liked more detail here.

Python Design Patterns I & II

Ok, so I’m cheating on the review here a bit. These are actually two individual chapters, but they cover the same concept.  Traditional design patterns and how they relate to Python. For you OO types out there, iterator, decorator, observer, strategy, state, template, adapter, facade, lazy init, flyweight, abstract factory, and composition. Though I really think patterns are a wonderful tool of the trade, I also think it’s necessary to stress that you’re making a mistake if you look at a problem as a pattern puzzle (those are my words).  All in all, though, this book does an absolutely wonderful job of running through some of the classic design patterns and applying a Python twist to them. I enjoyed these two chapters the most.

Files and Strings

A good introduction. I would have liked to have seen more on the Python IO subsystem (i.e. dir(io) and explain it). More than enough to get you up and running if you’re new to the language, though. This is a pretty broad chapter. Files, to strings, to basic Unicode, to JSON, to pickle.

Testing Object Oriented Programs

Obligatory Testing Chapter(tm).

Common Python 3 Libraries

This chapter provides a survey of some commonly used Python 3 3rd party libraries and how they’re used. A very nice thing to have when dealing with a major, and incompatible, update to the language itself.

Conclusion

So, way back in the beginning of this post I mentioned that something bothered me about this book, but I couldn’t place it. I finally figured it out when I hit that last chapter. This book is great for someone wanting to learn Python from (close to) the beginning. In my opinion, I’d say that the Python tutorial (it’s on the Python.org site) should be a prerequisite. Then, go through this to learn a bit about Object Oriented Design using the language.

But, the trouble is most people that want to get into Python will still need to use the 2.x series. This is a great resources, but I’m not 100% certain the target audience is at it’s full potential yet. Why? Because the third party tool chain is still tiny as compared to 2.x. I guess, to sum the thought up, it’s an intermediate level book aimed at an audience that stands to benefit mostly from learning an earlier release.

Final thought. It’s worth it. While it’s a Python 3 book, the material is still 90% valid when shifted to Python 2. It’s a great way to mix some OO theory with implementation. On the bright side, you’ll pick up some Python 3 in the process. Which, of course, means you can go port some of that 3rd party tool chain.

On an unrelated note, Packt is now offering a subscription service to their online catalog — http://packtlib.packtpub.com/.

Open Source Awards Home

They’ve really turned out to be a great company to work with, so take a minute to check this out.

At the day job, I still work with Python 2.x. It works wonderfully and I love it and all that, but I’d really like the opportunity to learn more of the details in Python 3. Sure, I’ve read all of the release notes, checked it all out, and ran a few examples. I just haven’t actually sat down and learned it.  I’m looking forward to this.

They’ve put a sample chapter up if you care to check it out – Chapter 7: Object Oriented Shortcuts. I’ll make sure I post a full review here once it’s arrived and I’ve had the opportunity to read it.

Thus far, I’m really, really enjoying Google. It’s a breath of fresh air.  It’s possible to read up on all of the perks and whatnot on the ‘net, so I won’t go into that.  Above all, there’s one thing I’ve really come to like:

This afternoon, when I was driving home, I realized that everyone in all of the cars around me use a service that I help to produce.

That’s a pretty awesome feeling. I’m looking forward to spending quite a while here. I’ve learned quite a bit over the past seven days. Everything is done with the utmost quality and effort. It truly is a world class organization. Additionally, the people have been spectacular. Ready to help and share information whenever I need a bit of guidance.