Electric Duncan

OpenStack at DreamHost

2011-12-06T19:21:00.001-07:00

So I guess this is old news now, but DreamHost is really into OpenStack :-)

(In fact, during recess, DreamHost asked if I would pass a note to OpenStack. I didn't look inside the note, but we can all guess what it said...)

I was hired specifically to work on cloud stuff here at DreamHost, and we've got a new team that's super-excited about this -- they're starting to gear up for increased contributions and community engagement, gettin' themselves some cloud. We've now got our own Launchpad team, we're working on a handful of blueprints, chatting it up on mail lists and IRC meetings -- you get the picture :-) Exciting times.

For official blog posts and other news items that highlight DreamHost's interest and involvement in OpenStack, check these out:

I'll be writing more about our OpenStack work later, but wanted to get a quick cloud-shout-out done before too much time passed...

P.S. We're hiring Python rock-stars!

P.P.S. Did I mention that DH is an AWESOME place to work? We got an award for that, two years in a row :-)

Occupy's Declaration of Independence

2011-11-21T13:37:00.001-07:00

Illustration by Peter Whitley

There's one thing I would really, really love to see under the tree this year -- under everyone's tree: Occupy the World.

For the first time in history, it seems that there might be enough momentum, enough communication, enough strength of individual convictions, and enough mass support to be able to have a world-wide, non-violent, revolution.

Taking the US as an example in this beautiful hope: imagine 2 or 3 million people showing up on the lawns of the US law-making machinery in Washington, D.C., issuing their declaration of independence and simply stating a fact: "Things are now going to change, we will not leave until we have the government that we want."

Far from mob rule, the 99 is intelligent, lucid, and a collection of THE overwhelming majority... regardless of old political parties. An enormous amount of discussion, insightful inspection, exploration of alternatives has been researched, written about, and promoted over the past 5 to 10 years, co-culminating in what we see around us today as the Occupy movement. I have the utmost faith in these thinkers (by which I mean everyone from Gar Alperovitz to my second cousins working in Detroit, MI automotive plants) and their (our!) ability to produce a new constitution that provides for the 99 fairly. Such a new system would stand in stark contrast to that of today's system: one that caters to policies driven by enormous, "legal" bribes or banking systems that continuously steal from the customers and shareholders, running off with the loot.

The 99 is saying it, and has been saying it for a while: "It's time for a change."

They've taken things further by showing an undeniable presence at the scenes of crimes (financial and governmental institutions). I say let's take the last logical step, and fix the problem: let's put a new system of government in place. Let's have radical, peaceful change. Let's do it with poise, grace, and while keeping the benefit of the entire planet foremost in our minds. Let's do it world-wide, and not stop until the job is done. Let's have a revolution.

Let's have the revolution.

Physical Beings with Digital Lives

2011-06-23T17:44:00.004-06:00

There's a lot that one could say about that title. In fact, it could be the title of a high-volume collaborative blog... That aside, here's the context for this post: books. Books and Reality. And data.

This post got so long that I now need to add a list of sections here, just to make it more accessible. My apologies :-/

Mini Table of Contents

Books
Books in the Sky
Yeah, I Know: Go Social
Human Data History
Reality Merges
Conclusion

Books

I have tons of books. Actual, physical books. Walls of them. Some I use all the time (reference). Some I read once a year (good books that support multiple reads). Others I've only read once, perhaps as far back as high school (when I started collecting). My bookshelves are like a random associative memory array: reading each title or the act of pulling one from the shelf brings back a flood of memories, relived experiences, sometimes actual sense perceptions. It's a powerfully visceral activity.

But that's just my books. When I'm at friends' homes or offices, I cannot keep my eyes off their bookshelves. It's an irresistible compulsion. I linger and browse, often past any semblance of socially acceptable time limits. My conversational replies experience a rapid exponential die-off -- in duration, gaps, and semantic value -- culminating in grunts and finally silence. (My favorite offices to visit so far? friend mathematicians/maths professors!)

Books in the Sky

Oddly, I love books in digital format. I never really got hung up on the bit about not having the paper entity in my hands (though I have turned a digital book reader over, expecting the next page... though that was deep in the plot of a Greg Egan novel!). Traveling as much as I do, I'm in heaven with ebooks. I feel like Superman, carrying around a library with me everywhere I go.

But when I passed a bookshelf the other day on my way out the door and fell under the spell of a book-memory flashback, I realized what was going away as I transitioned to virtual books. And the painful question arose: How am I going to nurture future layers of book-mulch and text-humus with this new æthereal, cloud-bound library I'm building?

How can I share with others, my stacks of books? How will I browse friends' books in their offices, asking about author X and title Y? How will we borrow from each other? What can be done to add this and related missing richness back into our lives once we adopt the virtual versions?

Light-emitting walls that can display titles from your Amazon account? Virtual over-lays visible with wearable/immersive computing accessories? Whatever we end up with, a gimmick isn't going to cut it. It will need to reflect the same depth of history that stacks of physical books have come to represent to us and the collective human psyche since we first started gathers works of the written word.

Yeah, I Know: Go Social

I'm hung up on books here, I admit it. But the same goes equally well for much of what we experience in online social media as well. Everyone's trying to make a buck on people chatting, playing games, reading, etc. Business as usual.

But the problem is that everyone coming up with their own little solution, one piece at a time. Google, Facebook, LinkedIn, Last.fm, etc. "We socialize X." Wow. Good for you. Now, for every activity or group I'm interested in, I've got some tiny little corner of the internet that I need to pay attention to.

Right.

Maybe I'm just an online social idiot, but this isn't working for me. Too many places and pieces. The physical analogy would be me spending all day on the road, hitting all the social hot spots in the Colorado Front Range. Ain't gonna happen. Ever.

Human Data History

The social data scene is a big stick up my butt. I really don't like it there and I'd love to get rid of it. It's poorly engineered, primitive state makes me grumpy. I don't own a thousand hammers [1]; I don't want a thousand of anything that all do basically the same thing [2]. Most of us probably don't own hundreds of houses, either (for ourselves, that is). We keep most of our stuff in the same location or two.

Speaking of houses, let's talk about settlement. How did we choose where to set up camp, towns, etc.? Trade routes, availability of resources (direct physical presence or presence by virtue of trade routes). Are we doing that now on the internet? Are we looking at the analog to fertile valleys, productive rivers, and protected harbors? Whose priorities do we have in mind? As we set up virtual presences, are we in locations that benefit businesses? Or ourselves?

If we choose the latter, the businesses will come because the people are there. If we do it the other way around, we'll be looking for new virtual homes if the businesses close shop or change the rules too much.

Coming back to data (but on the same anthropological note), historically we've had distinct divisions of our data:

the secure location of our huts/houses/castles
what we presented about ourselves in adornment/fashion (public data)
what we could carry with us in bags/crates/vehicles

Because of their prevalence in our history, any attempt at realizing personal data in a virtual environment would do well to reflect on these. We're naturally already predisposed to such approaches; such divisions are things that anyone can intuitively grasp.

The problem is that we're all used to a single platform: our mutually agreed-upon reality. There's no such thing online yet. And if there were, who would own/run it? Monopolies are eventually overthrown. We hate them. So how do we get around this?

Reality Merges

This very naturally led to thoughts on digital lives in general. And this is more than just a question of usability or human-computer interaction. Rather, this is a question that borders on the metaphysical: how do we solve the problem of syncing divergent realities? Reality-reality interaction.

The problem shouldn't be minimalized by analogy: this isn't a "simple" matter of ensuring that the data in my address book on Google is the same as what's on my iPhone. My self-perception, many reminders of self-reflection, etc., take place as a result of various interactions I have with my surroundings: both real[3] and virtual. No problem. Except that the things that remind me are also things that others can see and interact with as well. Often, they will have associations that spark a neural cascade for them too.

I've had many conversations take place around objects in a shared environment where the name of the object was never mentioned, it was implicitly understood. When there's no shared object (or concept), we have to name the object, define it, share some basic associations, make sure that we're talking about the same thing, etc. Thats all prelude. Only with that done, can we have genuine communication take place around the given concept.

Now rinse and repeat for everything you want to talk about that revolves around or is at least related to something that exists virtually for you and isn't part of your shared, physical environment.

I can't imagine many useful general solutions to this. In fact, I can only imagine one (given our biological wiring): use what we know (in our bones) and overlay or augment our visual reality with another.

With augmented shared realities, there's no platform. You just need hardware that runs it and senses that can perceive it. Just like reality. At that point, we can start sharing what we want, allowing access to data about ourselves and what we like by dumping it into a shared perceptual space, regardless of the original data source. Merged.

Obviously, we're not there yet. We're going to need crazy improvements in mobile technology, storage, computer vision, etc. But once the technology catches up, I think we'll see some powerful needs being filled. And we might start coming out of the internet dark ages...

Conclusion

None of this is new; Pick up any number of books by Charlie Stross[4], and there's all sorts of fun to be had by exploring his ideas. But the point of this post wasn't to be new. While we're all busy enjoying the latest fad in social media, I think it's important we think about where the progressive succession of fads is taking us. At each point, there's a natural next step (more accurately, set of possible next steps). Let's look more than just one in front of us and let's not forget what our biology has made us. We may not be able to engineer truly wise decisions about our future, or even make our lives better/more efficient. But it would be nice if we could at least not make things worse :-)

Notes

[1] I think I have three, actually.

[2] This is one of the reasons I'm a big fan of http://ping.fm.

[3] Here I mean "real" in the "conventional" (shared) reality sense of Mahdyamikas. Ultimate reality... well, that's a topic for an entirely different sort of post...

[4] Check out his Amazon page: http://www.amazon.com/Charles-Stross/e/B001H6IW0Q/. Accelerando is probably his most praised book (and likely my favorite), but the ideas touched on in this blog post are explored in other works of his, most notably Glasshouse and Halting State.

txStatsD Preview

2011-06-21T11:34:00.002-06:00

Sidnei da Silva (of Plone fame) has recently created a Launchpad project for an async StatsD implementation. He's got code in place for review by any Twisted kingpins who'd like to give it a glance.

statsD was originally created in 2008 as a Perl implementation at Flickr for their statistics counting, timing, and graphing needs. Engineers at Etsy ported this work to Node.js (which Sidnei based his version on). A few months ago a regular Python implementation was created (also based on Node.js).

More than another (excellent) addition to the tx family, txStatsD will provide folks with the luxury of collecting stats using a Python server without having to write any blocking code :-) Sidnei also implemented a graphite protocol and client factory for passing the messages along.

Enjoy, and let him know what you think!

The Future of Personal Data: A Followup

2011-06-08T10:29:00.006-06:00

The Next Step

A few years ago, I wrote a post about the Future of Personal Data as a result of all the ultra large-scale systems reading and exploration I was doing. Google was foremost in my mind when writing that, but Apple has since come into the spotlight here as well.

Recently, Matt Zimmerman has decided to leave Canonical and join forces with Singly, an exciting startup company focused on secure user data storage and the socialization of (and development around) that data. In this blog post, the following core values were given about the software underlying Singly:

I own my personal data
I want my data to be useful to me
I make the decisions to protect or share my

I would like to see the following added:

I make the decisions on how my data is used
If my data is sold, I should get a return for this

This is not petulance speaking :-) This comes from a historical perspective on social and economic fairness. Witness the changes in individual rights and personal finance since the industrial revolution...

Fair Market Value

There are probably many solvent entities out there who would claim users are already getting a return for the use of their data: free email and office docs from Google, free cloud services from Apple, etc. But I would imagine that there are massive margins being made on user data, and services (as valuable as they may be) are a paltry return for such a gold mine. I cannot help but be reminded of the selling price of Manhattan Island or Alaska (in 2011 values, the Lenape Indians got around $29.61/square mile; the Russians got about $150.77/square mile).

Money makes a good point, but personal data (and this post) isn't about the almighty coin. This is about clearly defining who owns what and ensuring that those who don't want to be taken advantage of, aren't. This is about identifying exploitation, and building something better and longer-lasting in its place.

Who's Going to Pay for What?

In David Pakman's "Disruption" blog post about Singly, he makes the following comment:

"I cannot see consumers getting into the business of selling their data to marketers so they can see personalized advertising. Instead, I believe marketers will be encouraged to offer value to us in exchange for access to our data."

I have to agree with him... but I can only offer a qualified agreement. True, I find it hard to believe that users will be selling their data directly to marketers. From the user-side, the pain of inconvenience would not likely be worth the payoff; at the marketer end, individual data is useless, and munging an in-house-built collection would incur a lot of overhead not part of their core business.

However, users' data stored in lockers, updated regularly, pre-processed, has enormous value in the market. Right now, Apple and Google are making eye-crossing amounts of money from data just like this. Again, I would imagine that this data is only really valuable in large quantities and for interesting, identifiable demographics

If users provide their data, but in exchange only get a "nicer app" or a "useful utility" I'm going to cry "foul!" (unless someone can show me the actual numbers involved and unequivocally prove that fair exchange is occurring).

You Say Disruption, I Say Revolution

Instead, if a service such as Singly, were to offer a co-op style dividend payment system to all of its users, that would seem to be much more fair. Not only that, it would be the beginning of a market revolution. This is not to say that co-ops are some perfect economic model, but rather that the data we, as users, generate is of immense value. The more that Singly has, the greater potential for revenue. The more buzz that builds around Singly users generating revenue from their data, the more users they get. With mass-adoption, a new sub-economy is born.

Perhaps a better model than co-op is that of a mutual fund investment firm. Each Singly user has a portfolio of data. Depending upon each user's preferences, any or all of that data could be used by Singly to generate revenue. Some groups of users will generate more than others, and users in these groups would get greater returns.

Whichever analogy you prefer, with a little exploration it seems fairly clear that opportunities for a large payout are present. For instance, I like to imagine a world where entities like Apple and Google can't harvest user data, but must go through brokers whom users have given their permission to sell their data for the most profit. I also imagine there's a lot of lawmaking that would have to take place... and even more lobbying.

Even with that, Google and Apple would still make money hand over fist (or they'd become brokers themselves) -- enough to continue providing free services. Yet at the same time, users would be in control of their data; they'd be financing (or financially augmenting) their data-consumption lives with said data.

With the right press coverage, Singly could find themselves not only swamped with a massively growing user base, but at the very center of a new economy. With the right level of negotiation and coordination, businesses could buy into this new paradigm without losing their shirts in the disruption.

A Plea

In summary, I applaud the goals and vision of Singly. I, for one, would deeply appreciate writing applications against their data locker (to any Facebook or any of its dubious applications), where a user's rights are clear and protected. That being said, Sinlgy would have my eternal allegiance if they also took up the cause of rights for user data in the market place; if they helped transform the current nascent data economy into a world economy capable of achieving as-yet unimagined financial heights.

And if not Singly, my loyalty would be given to whomever did do this.

Packt: A Publishing House for the Future

2011-05-23T08:40:00.006-06:00

Since I first heard of them several years ago, I've viewed Packt as the underdog in the world of technical book publishing. In the past year or so, Packt seems to have gained greater and greater influence: their catalog continues to grow, they are attracting talented and knowledgeable engineers as authors, and their titles are things that I'm actually interested in.

Two examples of this are the books Expert Python Programming and Zenoss Core Network and System Monitoring. I received a copy of the former and blogged about my take on it. For the Zenoss book, last year I agreed to be a technical reviewer and am currently preparing a blog post on my pre- and post-publishing experiences.

In both cases, I agreed to work with Packt based solely on the technical merits of their works. However, my experience as a technical reviewer with them was so positive (I have had consistently excellent experiences with their staff over extended periods of time and on long-running conversations) that I have not only agreed to review more titles, but have read up on Packt themselves a bit. Here are some highlights from their wikipedia article:

They published their first book in 2004 (the same year Ubuntu started!).
Packt offers PDF versions of all of their books for download.
When a book written on an open source project is sold, Packt pays a royalty directly to that project.
As of March 2008, Packt's contributions to open source projects surpassed US $100,000 (I would love an updated stat on this, if anyone has a newer figure).
They went DRM-free in March 2009.
Packt supports and publishes books on smaller projects and subjects that standard publishing companies cannot make profitable.
Their stream-lined business model aims to give authors high royalty rates and the opportunity to write on topics that standard publishers tend to avoid.
Bonus: they also run the Open Source Content Management System Award.

These guys have some keys things going for them:

They've got what appears to be a lean approach to business.
They know how to effectively crowd-source, keeping their overhead low.
They are rewarding both the authors as well as the open source projects.
Their titles continue to grow in diversity and depth.
The have an outstanding staff.

Oh, and I really like the user account management in their website! When I log in, I see a list of owned books, source code links for them, clear/clean UI, very easy to navigate. I can't emphasize this enough to vendors, service providers, etc.: if you want a loyal user base:

make a good product that lasts a long time;
make simple and great tools that enhance the experience of those products, that truly improve the experience of your users.

All in all, Packt really appear to be leaders in publishing innovation, taking lessons learned from the frontier of open source software and applying that to the older industry of publication production. I would encourage folks to evaluate Packt for themselves: if you like what you see, support them in readership and authorship :-)

I, for one, will continue to review titles that appeal to me personally and that I think others would enjoy as well. I have two books in the queue and three pending blog posts for the following titles:

And who knows, if I feel like writing a technical book at some point, you may see me in the Packt catalog, too ;-)

Call for Testing: nVidia Cards in Ubuntu Natty

2011-03-16T07:33:00.001-06:00

I was supposed to write this blog post early yesterday morning, since the first results from community testing are going to be examined today... but we're still going to need on-going data. So not all is lost :-)

We're working very hard with chip vendors to make sure folks are getting the best experience possible running Unity in the forth-coming release of Ubuntu (Natty). Jean-Baptiste Lallement has posted to the QA mail list, asking for committed volunteers to test the nVidia drivers in Unity on a weekly basis. You can read his email here:

https://lists.ubuntu.com/archives/ubuntu-qa/2011-March/001484.html

Jean-Baptiste links to detailed instructions in his email. The QA Tracker for this effort is here (but please be sure to read his email and follow the instructions that he links to):

http://xorg.qa.ubuntu.com/qatracker/test/5154

We look forward to hearing from you!

Photo Credits: jackyalcine (slightly modified)

Uncle Canonical Wants U!

2011-01-21T07:08:00.004-07:00

you and uTouch, that is. And more!

Do you love Ubuntu? Are you an HCI freak? Are you an X hacker, kernel driver maintainer, Linux input enthusiast, Qt or GTK application developer, C/C++ hacker?

Do you enjoy being part of online technical communities? Are you actively involved in open source projects? Do you enjoy advocating for software?

Do you love test-driven development and thirst for high-quality open source software running on devices with touch interfaces?

Does the idea of working from home in your own time-zone, on cutting-edge, strategic projects at Canonical have you sitting on the edge of your seat?

If so, then please send me your resumé/CV! I'm looking for a few brilliant engineers as we grow our Product Strategy teams (DX, Design, and others).

Canonical and Codethink at Bostom GNOME Summit

2010-11-07T09:54:00.008-07:00

Today is the second day of the Boston GNOME Summit, and the second day of Canonical providing morning sustenance for the hackers here. Codethink and Canonical coordinated these efforts, with Codethink sponsoring food later today. It warms my heart that we can do this sort of thing.

Yesterday Cody Russell and I held a session about getting a gesture API into GTK 3.x. There were a great many questions about the uTouch framework, how we're handling multi-touch in the absence of MT support in X (coming in XInput 2.1), and what sort of dependencies would be needed (none! if GEIS is present on the system, gesture support will be added at build-time). At the end of the session, there was a consensus for Cody to present his plans to the GTK developers list and then to start getting branches reviewed for merge. We're hoping to make it for GTK 3.2.

In this vein, Cody and I have been hacking on libgrope for GTK compatibility, and this is serving as the sandbox for the GTK 3 gesture API development. My efforts have been focused on creating the GTK 2 Python C extension for grope. Given that the last time I coded C was in 1989 (and then a bit later in the mid-90s, when I had to hack a slackware driver to get ethernet working), this has been quite an effort. However, after a night and morning of hacking, I've got a handle on C extensions and am using the example code I wrote as the basis for pygrope. I've even managed to rope Barry Warsaw into reviewing the C extension code for us, to be sure we're not doing anything too crazy :-)

The Python C extension will be of immediate use to us in our test harness for gestures and exercising the stack. We will be creating a GTK app for recording user gestures for later playback and inclusion in test suites.

Probabilistic Input for uTouch?

2010-10-19T12:10:00.003-06:00

Well, it seems to be the season for a flurry of multi-touch posts. Guess that's helped along by the release of uTouch in Maverick and next week's Ubuntu Developer Summit multi-touch buffet :-)

One of the talks that John, Mark and I enjoyed greatly at UIST was Julia Schwartz's presentation on "A Framework for Probabilistic Input". Julia's an HCI PhD student at Carnegie-Mellon and works in their exciting dev lab. The UIST paper was written by Julia, Professor Hudson, Professor Mankoff, and Andrew Wilson (the latter of Microsoft Research).

One of the reasons I was personally so deeply appreciative of the paper was that when we were building the uTouch framework, Henrik Rydberg and Chase Douglas brought up issues around input uncertainty, and started proposing ways that might be employed to counter this. Getting plugged into folks who are working on this actively and thoroughly is phenomenal.

Since UIST 2010, the uTouch team has been in contact with Julia, sharing ideas and asking questions. Today we had a call with her and Professor Scott Hudson; as such, we have now started exploring possible avenues of development for probabilistic input support in uTouch. Scott and Professor Jen Mankoff will be attending part of UDS this year, and we've set up a session where they can share their research with us and engage in discussions, Q&A, etc., about it, and explore ways in which we can start moving forward on this in Ubuntu (and Linux in general).

The UIST talk slides and related paper are available here:

https://wiki.ubuntu.com/Multitouch#Probabilistic%20Input

A Visit with System76

2010-10-18T14:04:00.004-06:00

Earlier today, I had the pleasure of visiting Carl Richell and part of the System76 crew in their Denver offices today. Pictured to the left is their very sweet Starling NetBook. I became an instant fan, due to the sleek good looks, a fantastic keyboard, and a flawless installation of Ubuntu. Carl also showed me the Starling EduBook, which I completely fell in love with. There's a charter school in Colorado that bought a whole mess of these guys, and the kids at school adore them... and how could they not? A rugged, easy-to-use Linux netbook with Edubuntu installed on it? Total win.

Before getting a device, personnel, and facilities tour, Carl and I talked shop: uTouch, the future of multi-touch in Linux, Unity, Ubuntu on a plethora of devices. You know, the usual good stuff. I asked him what he'd like to see in Ubuntu that he feels would be necessary to totally rock out the tablet experience. His list of top picks echoes the sentiments of many people with whom I have had similar conversations. In no particular order, the list goes something like this:

fully-working auto-rotate
a great on-screen keyboard
browser, music, video, photo, and document apps -- all with a user interface designed for touch
the ability to deliver and play games
a sweet note taking app that integrates with email
TV remote control support

Much of this is already scheduled for discussions in UDS sessions next week :-)

Carl's excited about UDS and the continued conversations that will take place there. As am I. But I also can't stop thinking about that Starling NetBook... I'm going to be replacing my ailing AspireOne with System76's gorgeous offering...

Multitouch and Qt

2010-10-16T14:17:00.011-06:00

It's been 11 days since the Qt announcement of new gesture support, and I wanted to blog about it right away... but alas, now will have to do. The folks at Qt have been working on multi-touch support for a while now. They blogged about gestures, multi-touch, Mac support, Windows support, and then at UDS in Brussels (May 2010), they shared their 4.8 plans for multi-touch with the Ubuntu community.

Until recently, there has been no MT stack for Linux. The great news is that the folks at Qt are very interested in getting Qt to work with uTouch. Stephen Bregma has been working on the GEIS API that toolkits will have the option of taking advantage of, and we were delighted to hear from Zeno and Denis that the Qt API they have envisioned and planned is very similar to GEIS, and should make for an excellent match. They are going to be talking with the community about this at UDS two weeks from now, in fact :-)

In support of Qt's commitment to multi-touch and gestures, I wanted to encourage folks to take a look at their post about Qt Quick, Qml, and Gestures. The source code is now available for viewing, and we've started to dive into it ourselves, and it seems that others are as well (I know some GTK guys who are double-checking their own plans by looking at what Qt has done so far).

Also, if you haven't taken Qt Creator for a drive yet, do so; it's an impressive GUI editor/IDE for Qt, and I've really enjoyed it so far. You can enable GUI visual design for Qml projects by going to the "Help" menu and selecting "About Plugins..." after which, you will need to check the box next to "QmlDesigner" (under "Qt Quick"). Also, a nice cherry on top: the Qt Quick design interface looks stunning with the Ambiance theme in Ubuntu -- the color scheme is a perfect match :-)

Q & A: The Ubuntu Developer Summits

2010-10-16T10:41:00.008-06:00

Last week I had a nice email exchange with a good friend of Ubuntu, Professor Michael Terry at the University of Waterloo. We met at UIST 2010, but he and Ivanka Majic have been conferring about design and usability in Ubuntu for a while. Delighted with UIST in general and specifically with conversations Mark, John, and I had with Michael and one of his students, we invited them to UDS in Orlando at the end of October.

Coming from an academic institution (and an HCI lab in particular), Micheal had some questions about UDS, how to prepare, and the ways in which it might be different from various academic conferences. I asked him if it would be alright to share our conversation with the wider open source, academic, and professional communities, many of whose members will be attending UDS in Orlando for the first time. He was only too glad our conversation might benefit others :-)

I've edited the content slightly for a blog format (and for improved clarity), but it remains in essence unchanged:

Q: Since we're all kind of new to UDS, could we pick your brain a bit? :-) As we plan for attending, we're trying to envision how to best use UDS.

A: UDS is very practical in spirit, so it's actually fairly straight-forward to make good use of it. The purpose of attendees is to:

Engage with developers, community members, users, etc., on well-defined topics for software in Ubuntu, the software that builds Ubuntu, Ubuntu itself, the community, or the prominent tools and technologies that are build on Ubuntu.
Assemble and execute. After each discussion, each topic should have enough information and feedback such that the direction forward in any given project is clear. Next steps can be taken immediately (whether that's actual development, coordination, etc.).

The enormous amount of discussion that takes place there is recorded (some sessions with video, all with shared notes), so very often folks formalize this information as a spec.

For a list of the wide range of categories covered at UDS, check out this page (still in progress): http://uds.ubuntu.com/tracks/

To give you more of a sense of things:

There are lots of rooms, each with two projected screens: an IRC channel for that physical room, and a network-shared communal notes screen.
Sessions are about 50 minutes long, with breaks every few sessions.
Large monitors are placed in the halls with the schedule for the day displayed, so that folks can more easily find the sessions/rooms they want to be in.
There are topical plenaries for two hours after lunch that take place in a auditorium so that all who want to attend, can.

In other words, it's very active :-) Lots of moving about. Lots of intense discussion in short sessions.

Q: What is an ideal outcome from your perspective (and from our perspective)?

A: The ideal outcome is consensus on the discussed topics, unblocking all involved, and moving forward with a shared vision.

Q: What kind of work or goals do we want to accomplish while we're there?

A: In general, you want to attend the sessions that are meaningful to you, share your views during the discussion, and hopefully convince those present of the necessity of what you envision :-) If your vision needs adjustment, then UDS is the perfect place to get it tweaked and reach a mutual agreement with community members about a form suitable for an action plan.

Q: You just saw us in our "native" environment -- an academic conference; how is qualitatively different from such conferences?

A: Yes, that sort of scenario is a prototypical academic environment: disseminating of memes for processing and adoption in a particular field. The essence of that exists in the open source world: meme's are very important to us, but often implicit in nature; they arise as generalizations of practical experiences with implementation. However, they are only the first step (or a supporting factor for other steps).

The primary focus of the attendees at UDS is pragmatic: "Let's do X." As such, UDS attendees partake in a meme-disseminating event, the object of which is to generate project plans, task lists, direction changes, etc., within each given project (be it software, governance, or others).

Q: Any ideas/guidance on how to think about and plan for UDS?

A: Hopefully what I wrote above should give you enough of an intuitive feel for the environment that you'll have a good idea of what you can do... but just in case, I'd focus on doing the obvious, like putting together notes/supporting material for the sessions that you are interested in. In addition, though, you want to spend some time thinking about how to engage session attendees such that they are motivated to attend the session(s) and speak up while attending.

Well, that's all there is to this Q & A. I hope folks new to UDS will find this conversation useful.

We look forward to seeing Micheal and the rest of you at UDS!

Ubuntu 10.10 Multi-touch Video

2010-10-13T07:45:00.006-06:00

As many are aware, we released our maverick Maverick a few days ago. Part of what comes with Ubuntu 10.10 is the new uTouch stack. We been having lots of really great talks in the public IRC channel (#ubuntu-touch, Freenode.net) and on the general multi-touch mail list (subscription info at the bottom of that page). More and more folks are trying it out, experimenting, and contributing.

As part of the "Maverick Movies" initiative, fellow employees Gerry Carr, John Lea, and Neil Patel worked on getting this little gem produced (embedded to the left; here's the full-sized version). Gerry has written an excellent blog post about it here.

This video shows off most of the touch capabilities that were integrated into Unity for Maverick. Neil Patel (fellow Big Lebowski fan) and John Lea will be working very closely together during Natty to expand this feature set to take more advantage of the gesture capabilities of uTouch. Simlarly, Henrik Rydberg will be working with them to enhance the uTouch gesture engine. And all of us will be working on the newest addition to the uTouch family that Mark mentioned in his blog post: a gesture language. More on that later :-)

For right now, let's check out this video! For the impatient, the target list of gestures in Unity is available in a gestures guidelines document, lovingly put together by John Lea and Mark. As one might expect, there's a lot there. If you skip to section 5, you'll be able to get a nice visual overview of most of the gestures with some descriptive text. For those who chose to watch the video first, here's what you saw:

single taps (i.e., a touch "mouse click")
a window grab (three-finger touch, let one up)
moving the window (three-finger touch, let one up, move it around)
drag up to maximize
drag up and right (or left) to fill the right (or left) half of the screen
un-maximize (use the drag gesture on the maximized window)
bring up the dash (4-tap)
bring up all windows for one app (three-pinch)
bring up all windows (three-pinch again)

Have fun playing with Unity on your touch devices! There's lots more to come...

Multi-touch at UDS-N in Orlando, October 25th-29th

2010-10-07T14:41:00.004-06:00

The up-coming UDS in October is looking to be quite fantastic. The community around the Ubuntu distro is, as always, deeply involved, passionate, and eager to break new ground... but lately there seems to be even more than the usual anticipation. We're seeing folks getting involved for more than Ubuntu, and this is an interesting change.

At UDS people tend to focus on Ubuntu and the Ubuntu community. As the "Ubuntu Developer Summit" this makes obvious sense :-) Maybe it's just my own perceptions, but it seems that when we engineers get deeply involved in something, less and less of the outside world makes it through the filters, potentially leading to situations of isolated development. Reflecting on how our interests are connected to others' can open the view back up, and I think lots more people are doing that these days.

Ubuntu is not just a community; it's also part of a community. Part of many communities, in fact -- very large and thriving ones. The obvious candidates come to mind: Linux, GNOME, KDE, GTK, Qt, the massive collection of upstream applications. But there are more and subtler ones.

All the work that Ivanka and the Design team have done over the past year and a half has brought open source software into a new place with regard to aesthetics and how to make our applications more appealing to people across the globe, folks who don't have the same engineering-based perspective on software that we have. This is hugely important and I personally feel that I owe the Design team a HUGE debt of gratitude for what they are doing for something I hold dear to my heart: open source software.

The other big change I've noticed (perhaps mostly because I'm neck-deep in it!) is the rallying that has happened around multi-touch. Hackers, academics, industrial researchers, casual enthusiasts and business folk alike are interacting on IRC and the multi-touch mail list to a greater and greater degree. Amazing cross-project cooperation is taking place, and setting really excellent precedences. It seems that the field of Human-Computer Interaction is starting to find and hold its own in the larger open source community. And we're reaching out more, too -- some of us from Ubuntu recently attended UIST 2010 in NYC and had fantastic discussions with the other attendees there. I expect that we'll be more and more present at events like these.

There are other examples of this sort of thing happening elsewhere in the open source world, within projects as well as companies that sponsor them. It's very encouraging to see such new forms of growth and evolution in our midst :-)

Now down to business! If you can make it to UDS, you'll be in for a treat. The Platform team has been hard at work refactoring the UDS experience. Tracks are no longer based on and named after teams inside Canonical who are primarily responsible for the tech. Instead, a series of brainstorms have resulted in a very natural and organic approach to community discussions: areas of interest. This was done in such a way as to encourage very strong cross-pollination of ideas and development strategies. Projects from all across the open source world, team members inside and outside of Canonical, and passionate individuals will have new opportunities to impact decisions about the software that not only goes into Ubuntu, but which gets incorporated into many other distros and projects as well.

The tracks are listed at this UDS page, but I will duplicate that here for the lazy :-)

Application Developers
Cloud Infrastructure
Performance
Multimedia
Hardware Compatibility
Application Selection and Defaults
Ubuntu the Project
Other

As a particular example, the sessions focusing on multi-touch technology in Linux and its toolkits, applications, etc., intersect with many of these (Application Developers, Performance, Multimedia, Hardware Compatibility, Ubuntu the Project, etc.). The same is true for other projects. Tracks that used to be dedicated to Desktop, Server, Foundations, etc., are now going to be exposed to a more diverse audience that focuses on individual's interests, brining community members even closer to the goodness they love :-)

If you're interested in tracking multi-touch sessions, you can watch our session planning evolve at the link below:

https://blueprints.launchpad.net/ubuntu?searchtext=hci-n

We've still got one more blueprint coming: A Gesture Language. I haven't had a chance to compile my notes about this yet, but when I do, there will likely be another blog post just discussing the exploration we want to do around the idea of such a language at UDS.

If you can't make it to UDS, remember that we broadcast live streaming audio and project IRC channels for each room up on a screen so that all present can read remote comments and respond to listeners' questions.

Hope to see you there!

The Growing Linux Multi-touch Community

2010-09-21T06:31:00.007-06:00

There has been a lot happening with the intersection of the Linux and HCI communities of late; it seems that the Linux community is getting more and more involved in not just multi-touch, but human computer interaction as well. It's not just for researchers anymore :-)

X.org

The X.org recently held XDS in Toulouse, France and by all accounts it was absolutely fabulous. I hear folks had some great discussions a day or two prior to the event and hope we get to hear about those :-) I'm watching Peter's blog for an update and hope that Chase will do the same when he gets back from Taiwan.

Point-of-Sale

A few weeks ago, I was delighted to hear from Gene Mosher (inventor of touchscreen point-of-sale) on the multi-touch mail list. We had a good conversation about his use case and I am really looking forward to checking out the work he's done.

Community Support

Folks have been popping into IRC (#ubuntu-touch on Freenode) on a more regular basis, asking everything from beginners' questions to exploring deeply involved technical issues. Fabián Rodríguez has been helping out with community support, as have the core developers. In combination, the team is able to get most folks up and running, sometimes even with non-supported hardware. (Oh hell, they're gonna hate me for saying that publicly! Here comes the flood...)

PyMT, Qt, and GTK

Of particular note, we've been having really amazing interactions with PyMT and Qt folks, and it's great to see those relationships growing. Mathieu Virbel of PyMT has done some fantastic work on adding Python support to mtdev and is currently running PyMT on those bindings. Christopher Denter (another PyMT-er) has shared some really great stuff about Movid, a project he co-founded and one that we're keeping a close eye on.

On the Qt side of the house, Denis Dzyubenko and Zeno Albisser have been making great strides in MT development. Originally working on the gestures API for QT, Denis has recently taken over for Bradley Hughes on the multi-touch support in Qt. Zeno is a project lead for improving the gesture API. Stephen Webb from the DX multi-touch team will be working with them closely in the coming cycle.

Cody Russell has started chatting with Carlos Garnacho, Ryan Lortie, and others in the GTK community about getting full support of MT in GTK (continuing on Carlos' excellent work). These conversations are expected to ramp up at GTK Hackfest in Spain, and continue at UDS in Orlando, FL. With most of Cody's Maverick work done now, he's been able to pick up some prototyping he did last month and run with it some more. I can't wait to see the results :-)

Ubuntu on the Phone?!

And for dessert, I have saved this amazing bit of info: we recently received an email on the multi-touch list from Lukas-David Gorris who is working on the HTC Linux project. It turns out, he is running Maverick on the HTC HD2 (Leo) and is testing the uTouch framework with this device. Henrik Rydberg is helping him debug issues in his spare time. We were completely stunned to not only to hear about Maverick running on an HTC, but attempts to use uTouch on it!

The HTC Linux project looks really awesome. I hope they get more community interest, as this is the sort of project that gives Linux its power base: development and support from the community that large companies rarely invest in themselves. HTC Linux is a major step towards the prevalence of Linux on phones.

Though not Ubuntu, this is also a perfect time to mention MeeGo: another effort being made with the Linux community, and a rare exception to the "rule" of big companies not putting their money on the line. MeeGo is a joint effort between Nokia and Intel to produce a Linux distro for phones (and other devices). MeeGo has a focused touch effort, as well (I'm subscribed to the mail list).

This is a huge boon to Linux, and like the HTC Linux project, very, very cool stuff.

HCI at Canonical

2010-09-02T04:00:00.003-06:00

uTouch

Back in March, I blogged about future possibilities (in a blue-sky sense) of multi-touch, mentioning the project management I was doing for MT hardware kernel driver support in Lucid (and then proceeding to dive into the deep end of speculation). It's now an Ubuntu cycle later, and holy crap... I'm having a hard time finding the words. I think the blog title says it all. But I'll try to elaborate :-)

Unless you've been living under a rock, you've probably noticed the big announcements we made a few weeks ago:

uTouch mail list announcement
Mark Shuttleworth's blog post
The Canonical uTouch announcement
Chase's blogging extravaganza (uTouch, MT architecture, Magic Trackpad, Trackpad update)

For the next few days, we were all over Google news. This was quite a shock, given that we'd been heads-down into the project for so long and hadn't really come up for air nor fully anticipated the impact (to others or ourselves). Needless to say, after the intense amount of work that the team had engaged in over the previous couple months, this was quite gratifying, if somewhat unexpected.

There has been a lot of discussion in blog posts, mail lists, IRC (#ubuntu-touch on freenode.net), Launchpad bugs and merge proposals, etc., so much so that touchscreens now pursue me feverishly when I sleep at night. I'm really not interested in writing more of the same :-)

As such, I want to mix things up a bit...

HCI Remixed

I've been reading an amazing anthology of essays on human-computer interaction. I still haven't finished the book (yeah, I've got about 10 in-progress titles on my nightstand), but am relishing every word in this particular collection. The book is HCI Remixed: Reflections on Works That Have Influenced the HCI Community.

While doing some research at the beginning of the Maverick development cycle, I came across HCI Remixed at the local library -- the title intrigued me and I couldn't resist. Weeks later, after having maxed out the number of times I could renew the book, I just purchased it -- I simply couldn't get enough of the book. Every essay I'd read up to that point was fantastic; each one provided volumes of information, experiences, insights, ideas for follow-up, etc. Whenever I finished one essay, I spent days and sometimes weeks reading up on references, pondering the past and future of human-computer interaction.

Due to the unusual nature of the book, describing it is surprisingly difficult. That being said, the MIT Press page gives you a great taste:

Over almost three decades, the field of human-computer interaction (HCI) has produced a rich and varied literature. Although the focus of attention today is naturally on new work, older contributions that played a role in shaping the trajectory and character of the field have much to tell us. The contributors to HCI Remixed were asked to reflect on a single work at least ten years old that influenced their approach to HCI. The result is this collection of fifty-one short, engaging, and idiosyncratic essays, reflections on a range of works in a variety of forms that chart the emergence of a new field.

If you're into HCI, learning from others, and discovering new sources of inspiration for your own work, this is simply a must-have book :-)

A Small Piece of History

By the time I checked the book out of the Golden public library, it was May and we had begun building the MT team. By July -- once it became clear how astounding the team's work was -- I realized that in 10 or 20 years I could very well be writing an article about Henrik, Chase, Stephen, Ikbel, and Rafi. Much like those in the book, I could be sharing the conversations I'd had with Stéphane Chatty, Mark Shuttleworth, Neil Patel, David Siegel, and John Lea. And that's only the crew which which I was collaborating or discussing directly. There are a lot of folks who've been working very hard on multi-touch infrastructure solutions and exploring ways of integrating these for several years (e.g., Peter Hutterer and Carlos Garnacho).

Though many foundations have been laid, as of yet (to the best of my knowledge), no Linux distribution has released a multi-touch stack that integrated gestures in a unified manner across everything from applications to window managers and beyond. This was something that Mark wanted us to provide to the open source world. In this spirit, the multitouch team hasn't just hacked things together to get a product out in time. A lot of generative, creative thought and care has gone into uTouch. A lot of original problem solving has taken place. Physics PhDs, kernel hackers, X.org hackers, driver creators, application integrators, toolkit gurus -- all of this knowledge was concentrated, applied, and used to distill a first approximation of what a gesture stack in Linux could look like, using the latest available technology and methodologies.

To be honest, we weren't really sure we could pull it off. There was a very good chance we could have failed at our task, quietly chalking up the loss as a lesson learned. Now that we've managed to shape these ideas into actual software, taken the threads of dreams and woven something real, we are thrilled to be engaging with others to see where all of us can take multi-touch and gestures from here.

Thanks to expert input from the wider open source community, we're already looking at ways in which we can improve upon the first version, ways of bringing new ideas and experiences to developers and users of multi-touch hardware running Linux. Things are only just warming up, and the greatest contributions have yet to be made. Every single person in the community has before them a world of possibilities for getting involved and creating the future human-computer interfaces for the free and open source world in the coming weeks and months. These are indeed exciting times.

Ubuntu Foundations and Maverick Meerkat 10.10

2010-05-27T17:39:00.004-06:00

Introduction

For those that don't know, The Ubuntu Foundations Team is responsible for delivering the core Ubuntu system, which is common to the whole Ubuntu family of products and services. For the past couple months, I had the pleasure and honour to work with the Foundations team, assisting in preparation for the Foundations Track at UDS and planning for the 10.10 cycle.

Below I will give a brief summary of the major topics covered at UDS which, in turn, generated scheduled work items. They are listed in alphabetical order.

Boot Work

Several boot-related areas were identified for work during Maverick. These include the following:
cd boot - by converting CD boot to use grub2 with its new graphical goodness, we will only need to maintain use of a single bootloader
continued performance improvements
grub2 framebuffer - the end goal being a near flicker-free graphical boot splash experience
UEFI - support booting on systems that use UEFI firmware

Related blueprints:
foundations-m-cd-boot
foundations-m-boot-performance
foundations-m-grub2-boot-framebuffer
foundations-m-uefi-support

btrfs

In Maverick, we will be adding support for btrfs. Our tasks include such work as making ureadahead work with btrfs, adding btrfs support to grub2, integration work, and features support.

Related blueprint:
foundations-m-btrfs-support

Cleanup

Just after an LTS release is a perfect time to clean house. We will be taking this opportunity to do so, with such work as dropping unused/unneeded packages from the base system, double-checking package dependencies, and investigating space-saving measures.

Related blueprints:
foundations-m-spring-cleaning
foundations-m-package-culling

Installer Redesign

The installer is getting a serious make-over. Foundations and the Design team are working very closely together, improving the workflow, minimizing user clicks, improving the look-and-feel, and providing utility with increased ease of use.

Related blueprint:
foundations-m-installer-redesign

Software Center

We want to get new applications into the Software Center, ideally providing developers with a means of generating revenue with the applications. For the former, we need to define some good social and technical processes to ensure ongoing quality and excellent producer/consumer experience. In conjunction with that, we need to work on getting a billing system in place.

Related blueprint:
foundations-m-software-center-roadmap

Upstart

Upstart is getting major work this cycle. New and improved features include the following:
Manual mode
Resource limits
Dependencies
Better support for UIs that want to use Upstart
Simple skeleton to make life easier for sysadmins
Provide an API for services and tasks so that folks don't have to think about the event-based model if they don't need to
Explore the conversion of conf files into jobs
Possibly extend the debug capabilities into an interactive mode
Improve job disabling

Foundations will also be working closely with the server team to get their init scripts converted to Upstart. Conversely, the Kernel team will be providing new features that will allow Foundations to fully develop the planned Upstart features.

Related blueprint:
foundations-m-finish-upstart

Miscellaneous

There is lots of other work we'll be doing, some of which are highlighted in the following:
i686 Default Compile
Stop building the ia64 and sparc community ports
Multiarch Support for gcc, binutils, dpkg, and apt
Foundations Python improvements
Upgrade and install testing
Related blueprints:
foundations-m-686-compile
foundations-m-drop-ia64sparc
foundations-m-multiarch-support
foundations-m-python-continuous-integration
foundations-m-python-versions
foundations-m-robust-python-packaging
foundations-m-upgrade-and-install-testing

References
Foundations Work Items for Maverick
Maverick Release Schedule
btrfs
Upstart

Some Thoughts on the Mobile Device Interface

2010-03-12T16:25:00.005-07:00

Smartphones: Yesterday's News

As some of you know, I started 2010 by working in a new position at Canonical: Ubuntu Project Manager. I've been having an absolute blast; working my butt off has never been more fun, challenging, or interesting. I'm finding that nearly every side-interest I've had in the past several years is coming to the forefront in my project management work.

There were all sorts of adjustments I needed to make before PM'ing again, and one of those was catching up on communication technology. I now live by email, calendars, IRC, Skype, and phone conversations. Gone are the days of going heads-down into some code for a week or two. I need to stay connected, 100% of the time. I needed to get a smartphone.

What I really wanted was a Nokia N900. Sadly, T-Mobile's not offering one, so I got an Android phone with a physical keyboard instead: a new G1. Yeah, out of date, but considering that I was still using a Razr, the G1 is cutting-edge ;-)

I gotta tell you, this little phone has changed my life. The craziest thing is not the apps, the Market, the features, etc.; it's the touchscreen that has made me a believer.

You Can Touch This

I can't believe how radically the touchscreen phone has changed my computing habits and preferences. When I sit down at my laptop or desktop to do something quickly, I don't want use the keyboard or mouse. I want to point, swipe, and tweak with my fingers. When I'm on the phone, my brain has my speaking and text-processing faculties tied up. To easily multitask while talking, I need to be able to use a different part of my brain: that part involved in motor control.

I like using the touchscreen so much that I will often use my G1 for tasks that are better suited for my laptop, merely because of the joy I get from using the interface.

Touch, tap, drag, push, swipe. I love it. Can't get enough of it.

The best thing of all? This is really silly: I love the virtual desktops and being able to navigate between them with a swipe. Whose idea was this? That designer or engineer needs to be promoted! I have never experienced a more intuitive way of switching virtual desktops. I didn't even know how much this was important to me until I used the G1. I want this for my laptop!

Having such a positive bias towards tactile technology, you can imagine my joy when I saw this Ubuntu blueprint. And then when I was asked to work with Bryce and Andy on the PM portion of multi-touch support, I was quite delighted. This will give application developers and device engineers what they need in order to start creating new exciting stuff for the Ubuntu world. You will be able to have an iPad-like experience on your Linux devices (that have the proper hardware).

The thing is, as much fun as tactile interfaces are, I want way more now. I've been given a taste... now I want the banquet.

An Interface for the Future

So the iPad has been getting lots of press. It's bigger than a phone, I like that: I could replace my pen-and-paper medium-sized Moleskine with one. The thing I like the best though? Yeah, you guessed it: the interface. Curling pages and apps that have been re-worked specifically for the new format/size. I love it when a device lets me use it in a way that is natural and intuitive, and provides visual (or other sensory) feedback on my use.

Apple has made, in my opinion, a good and interesting product. But certainly not a revolutionary one. It's a natural progression from what folks are already doing with smartphones and netbooks.

So let's talk about revolutionary :-)

Imagine you've got a crazy new shirt, one whose fibers convert your movements to electrical current and can power your devices. Perhaps it comes with a "battery net" for literally flexible power storage. Now imagine that some clever sod has equipped your shirt with sockets for micro-SD cards (or something similar). If you've read my "After the Cloud" posts, then you probably know where I'm going with this :-)

Now add some embedded micro-controllers, and bluetooth, a smart phone with cloud-controller software, and you've got a personal S3 with potentially terabytes of light-weight, wearable storage. And your phone controls the nodes, redundancy, failover, etc. Maybe your phone runs Ubuntu and you're pushing backups of your personal cloud onto your U1 account.

But how do we interface with all this great storage? Here comes the banquet I mentioned :-)

I want to be able to reach "into" my phone, grab an icon (application, file, contact, whatever) and put it where I want. But I mean really put it: I can pick up some data off my desktop, and throw it over my shoulder or at my feet. With appropriate sensing equipemnt on the power shirt's sleeves, my arms and hands are now the perfect "mice."

In fact, there's no more virtual desktop (that's old skool). One of the new primary functions of your mobile device is to peer "into" the halo of data that surrounds you now. You can either spin your virtual storage space around you like an inverted, 3D lazy susan, our you can physically move your phone around, like a diving mask peering into the water.

Closing Thoughts

The more data I have, the more I feel that I live in it. The problem is that our current tech forces us into tiny sardine cans and we have to consume our data with the equivalent of a single chop-stick. If I'm going to live in my data, I want to have the best possible experience of immersion that I can. I want an interface that can handle my future.

I could go on and on... I love this sort of thing. The important thing to know now is that the community is working on the building blocks for our technological future. The first steps are being made in open source software that will allow us to take giant, insanely cool steps in the not-to-distant future.

7 Years of Blogasmic Action

2010-02-19T11:24:00.005-07:00

Earlier today, I was looking through old blog posts, reminding myself of issues and topics that I want to continue reflecting on actively. When I got to the end of the posts, I looked at the earliest date and realized that my seven-year blogiversary was this past Wednesday. At my vocalized comment to this effect, Marjorie asked what got me started on blogging. Interestingly, it ties in with my professional life over the past twleve years...

As a physics undergrad and employee of the University of Maryland, I was hired away by a large internet startup for my systems and programming background. Already a fan of Office Space, I got to experience the reality of cubicle life as the not-so-funny, real complement of that classic movie. However, it wasn't my personal experience of cubihell that had the biggest impact upon my psyche; it was the corporate atrocities suffered by various customers and partners of dot-boom era, high-rolling companies.

Already a passionate open-source user and contributor, it become one of my personal goals to make sure a greater number of people (especially those making decisions in companies) were informed about the software freedoms that were within their reach. Within two years of this epiphany, I had my own company and was open-source consulting for everything from small businesses to large international corporations and the Federal government. Everything I did was open source; all of it saved my customers from spending millions of dollars unnecessarily and the enslavement of proprietary lock-in.

It was at this time that I started blogging. My primary interest was to be a voice in the wilderness of consultants, finding other like-minded programmers, project managers, sales engineers, etc., who believed in "open business", transparency, and accountability. As time progressed, my blogging habits adapted to fill other needs in various communities. However, all of my professional decisions have remained true to that initial desire to work on technology that was not only free, but actively prevented the bondage of its users.

For the past year and a half, I've been working at Canonical, the company that funds and actively supports Ubuntu and the Ubuntu Community. Every time I work with a different engineer or a new team, I am blown away by two things: the incredible talent embodied in each interaction, and the dedication that each person has to improving how free technology is delivered to the world.

My blog posts have definitely changed in character over the past seven years. But I'm delighted that, where I once felt alone with my ideals, I can count true giants in the field among my fellow community members, cooworkers, and mentors.

txAWS 0.0.1 Released!

2010-02-12T10:24:00.007-07:00

The first version of txAWS just made it out the door, thanks to prodding from Scott Moser, who is helping to get txAWS into Ubuntu Lucid Lynx. It's been uploaded to PyPI now too, so you can do the usual (easy_install txAWS), or you can download it from Launchpad.

For those interested in writing async code for the cloud, txAWS is the library for you :-) What's more, if you're interested in contributing to a Twisted-based project, this could be just the thing to get you started. The use of Twisted is pretty basic in txAWS (though we do adhere to various coding idioms pretty strongly, to enhance maintainability), and would be a nice introduction. What's more, there are lots of exciting features and work still to do, and you could really make a difference.

Recent Work on Various Open Source Projects

2009-10-26T08:16:00.008-06:00

Over the last few months, I've been doing lots of work on various open source projects. I've been so burried in them that I haven't blogged (or microblogged) much about them. So much has been happening, though, that I needed to take a break from the coding and communicate some of this :-)

txAWS

Over the last few months, Robert Collins, Thomas Herve, Jamshed Kakar and I have been putting lots of effort into improving cloud support in the async (Twisted) Python Amazon EC2 library. It's been a lot of fun to see that part of the library take shape and start getting production use from Canonical. We have implemented the following functionality groups and their associated API methods:

Instances
Key Pairs
Security Groups
Elastic IPs
Availability Zones
EBS

There is a ticket for the following two groups of functionality, and branches in progress for both:

Images (ticket #461136 on Launchpad)
Image Attributes (ticket #461139 on Launchpad)

Once those are merged, EC2 support in txAWS will be complete.

As a bonus, we've added support for arbitrary endpoints and with that in place, have successfully tested txaws.ec2 against Eucalyptus clouds :-)

txJSON-RPC

There's a new release of txJSON-RPC out now, downloadable from PyPI. Work on the next version has been a great deal of fun. What started out as a conversation on IRC with Terry Jones, ended up as spec-driven doctest work on trunk for implementing support for multiple versions of the JSON-RPC spec (pre-1.0, 1.0, and 2.0).

With these changes in spec support, txJSON-RPC has really started to mature, and that's been fantastic. Even more, the jsonrpclib module that's included in txJSON-RPC (and can be used with non-Twisted projects) is getting spec version support as well.

SOOM in txULS

As some may know, one of my computing passions is ultra large-scale systems. After a phone conversation with Jamshed Kakar and some nice exchanges on the Python ULS-SIG mail list with Alex Drahon, I started working on a set of coding experiments in self-organizing objects. The Google Doc informally outlines the various stages and goals.

For now, the code is living in a txULS series on Launchpad. The reason for its inclusion in txULS is that ultimate goal of the SOOM (self-organizing object meshes) code is to produce an async API for building Twisted services that provide behaviours as outlined in the Google Doc (linked above).

I would to emphasize the networking-library-agnostic nature of the ULS-SIG: Twisted comes up since I spend a lot of time with Twisted, but ever networking library is welcome. I'm personally interested in exploring (or watching other developers explore) various Stackless Python experiments in the ULS systems space.

txSpore

This project was a spontaneous effort resulting from an evening of code review when I first discovered the official Python API from EA/Maxis for the game Spore. It's been a blast and something that Chris Armstrong and I have been working on together. The API is currently feature-complete, but Chris has some excellent ideas about improving usage as well as some additional API augmentations that will make life easier for game developers.

Already more featureful and usable than the official Spore Python API, there are great things in store for this library. Chris has come up with several very cool demo ideas that take advantage of the new API and will push it to the limits. We're both pretty excited :-)

Isomyr

I love isometric games. I'm a freak for the classic look. One night about a month ago, Chris and I discovered Isotope, an isometric Python game engine by Simon Gillespie. It was last updated in 2005 at version 0.9, so I started working on a branch that could be released as 1.0. I never heard back from Simon after an inquiry for his permission to release as 1.0, so I forked the code to a new project: Isomyr. I released the code rewrite work I had done to that point (plus some changes such as replacing some old code with Numpy) as 0.1. At which point things just went nuts...

Isomyr now has support for multiple worlds, customizable (per world) in-game time and calendars, and basic interactive fiction development. The latest chunk of code (that hasn't been pushed up to Launchpad yet) is adding support for general planetary simulation (e.g., axial title, varying daylight hours, seasons, and weather). As you might imagine, this has been a great deal of fun to work on!

PyRRD

PyRRD has gotten some recent community love, with requests for a mail list, new developer-oritnted features, etc. Currently at version 0.0.7, the 0.1.0 release isn't to far away. Folks have been using trunk for a while, which added support for the RRDTool Python bindings back in March of this year. (PyRRD's focus has primarily been for users/developers who didn't have the RRDTool Python bindings installed). In the next couple weeks or months, I expect that we'll be adding a few more features, and then preparing the new release.

PyRTF

Another fixer-upper project, PyRTF (mirroed on Google code as pyrtf-ng) has been on hiatus for a while, due to my diminished need to manipulate and interact with RTF files. However, a new developer has joined the project and the code-cleanup and unit test development now continues. Thanks Christian Simms!

A while ago, Simon Cusack (the original author of PyRTF) and I had some great discussions about the future development of PyRTF and his interest in merging the recent changes into trunk on SourceForge. I deferred on that action, wanting to wait until the code cleanup, unit tests, and API changes had been completed. With Christian's help, we may get there now :-)

Wrap-up

It's been about a year since I've been so active in open source development, and it feels really good to be at it again :-) Being back in Colorado seems to have helped in subtle ways, but mostly it's been the increased interaction and interest from developers in the community that I can thank for my increased activity (and thus enjoyment). You guys are awesome. You're the reason for any code I produce.

PyCon 2010 Talks Neeed!

2009-09-17T11:12:00.002-06:00

Hey folks,

At my last count, we've only received 20 talks so far for PyCon 2010! There are only 14 days remaining for talk submissions... if you've had a great idea about a talk for PyCon, now's the time to make it happen!

Below I've pasted some links that folks might find helpful. The first one has pretty much everything you need to know about submitting a talk for PyCon.

txSpore: Twisted Spore

2009-09-13T23:05:00.002-06:00

I just had a delightful weekend of coding :-) I spent the past two days porting the Spore Python API to Twisted. You can now incorporate Spore data (from static XML as well as REST requests) into your non-blocking Python applications/games!

This was a pretty easy task, really. The API just makes HTTP requests with twisted.web.client.getPage. There was a little bit of work involved in creating object models for the XML, and some head-scratching for the error-catching deferToThread unit test I tried to write (it's still buggy... need to figure that one out). Everything else was pretty much cake.

In fact, it was so much fun to kick back and write some playful code that I might overhaul the sync Python code as well and incorporate both into txSpore.

Do be aware, however, that the code still has some big improvements coming. The first thing I want to hit is actually create a client object. Right now, the client module contains a series of functions (since state's not currently needed). However, I want to start doing some basic object caching in order to limit the number of requests made to spore.com and increase the response time. That's the big item for 0.0.2. Update: 0.0.2 is now released!

Next I'd like to create some more demo apps that show off the API usage better. Right now, there's one demo (a .tac file). All it does is ask for a user name, renders a user page, and then links to a user "Spore assets" page (that's the thumbnail image above).

One thing that might be fun to do is write a script that checks for the latest achievements and publishes them to various microblog/status sites with the Twisted PyngFM client :-)

There's a project page up on Launchpad for txSpore, and I've posted a notice and some updates to the Spore developer forums. It's also been published on PyPI.

Enjoy!

Windows Media to MP3 Conversion for Mac OS X and Linux

2009-09-07T16:47:00.004-06:00

For the past couple years, my girlfriend has been amazingly (astonishingly) patient about a whole slew of .wma files that we've got on the network drive... backups of her CD collection made when she was a Windows user. We managed to save them right before the computer died, but she hasn't been able to listen to them when she's booted into Ubuntu or Mac OS X.

Late last month, after getting back from two weeks abroad, Marjorie said that she'd really like to have access to her music collection again (the CDs are cumbersome and stored away in boxes for our impending move back to Colorado). With that said, I did some digging around, and found some immediately helpful links (two years ago, a few google searches had turned up results that indicated too much effort was involved).

I started out by trying a couple free Mac OS X GUI applications, but these ended up being quite horrible: either they did not offer the functionality I desired, they were buggy to the point of being unusable, or they rendered audio with unlistenable artifacts.

In the end, I had to use mplayer and lame in combination. After googling around and some trial and error, I discovered the combination of mplayer options that would successfully extract the audio data from .wma files and dump them as .wav files.

I started with a shell script, but quickly changed to Python, since there were several locations for the .wma files, and none of them on nice paths. I've used this script several times since then, when more .wma files were discovered, and have yet to encounter any issues in sound quality. Once nice-to-have would be to extract .wma metadata and save it in the new .mp3 files as id3 tags...

Anyway, here's the code:

#!/usr/bin/python
import os
import re
import subprocess
import sys


# script configuration
if sys.platform == "darwin":
    MPLAYER_PATH = os.path.join(
        "/Applications/Non-Standard/Audio and Video",
        "MPlayer OS X 2.app/Contents/Resources/mplayer.app/Contents/MacOS")
    # lame was manually installed into /usr/bin
    LAME_PATH="/usr/bin"
elif sys.platform == "linux2":
    MPLAYER_PATH = "/usr/bin"
    LAME_PATH="/usr/bin"
MPLAYER = os.path.join(MPLAYER_PATH, "mplayer")
LAME = os.path.join(LAME_PATH, "lame")
DUMP_FILE = "audiodump.wav"
BACKUP_DIR = "wma"
WORKING_DIR = "/tmp"


def make_audio_dump(filename):
    """
    Use mplayer to dump the audio contents of the .wma files as .wav files.
    """
    command = ("\"%s\" -nosound -vo null -vc dummy -af resample=44100 -aid 1 " 
               "-ao pcm:waveheader \"%s\"" % (MPLAYER, filename))
    subprocess.call(command, shell=True)


def convert_wav_to_mp3(input, output):
    """
    Use lame to convert the .wav files to .mp3 files. Remove the raw .wav file
    when done.
    """
    command = "\"%s\" -b 256 -h \"%s\" -o \"%s\"" % (LAME, input, output)
    subprocess.call(command, shell=True)
    os.unlink(input)


def convert_wma_to_mp3(wma_filename):
    """
    Given a .wma filename, get a filename for the new .mp3 file based on this,
    convert the original to a .wav and then that to an .mp3 file.
    """
    mp3_filename = re.sub("\.wma$", ".mp3", wma_filename)
    make_audio_dump(wma_filename)
    convert_wav_to_mp3(DUMP_FILE, mp3_filename)


def has_wma_files(filenames):
    """
    Given a list of filenames, check to see if any of them have the .wma file
    extension. If so, return a true value; otherwise, a false one.
    """
    for filename in filenames:
        if filename.endswith(".wma"):
            return True
    return False


def convert_wma_files(path):
    """
    Walk a given file system directory and all its child directories in order
    to find .wma files. If found, convert them to .mp3 files and backup the
    originals.
    """
    for dir, subdirs, filenames in os.walk(path):
        # we don't want to convert files that have already been converted
        if os.path.basename(dir) == BACKUP_DIR:
            continue
        # if there's nothing to do, move on
        if not has_wma_files(filenames):
            continue
        # define and create the backup dir, if it hasn't been already
        backup_dir = os.path.join(dir, BACKUP_DIR)
        if not os.path.exists(backup_dir):
            os.mkdir(backup_dir)
        for filename in sorted(filenames):
            # on mac os x samba shares, sometimes ._*.wma files are present;
            # skip these
            if filename.startswith("."):
                continue
            if filename.endswith(".wma"):
                print "Dumping audio for %s ..." % (filename)
                wma_filename = os.path.join(dir,  filename)
                wma_backup = os.path.join(backup_dir, filename)
                convert_wma_to_mp3(wma_filename)
                os.rename(wma_filename, wma_backup)


if __name__ == "__main__":
    path = sys.argv[1]
    os.chdir(WORKING_DIR)
    convert_wma_files(path)

Hope someone else finds this useful and their significant others don't have to wait 2 years for their music!