Step Three: Profit!

High-level Languages for the 6502

2012-04-14T14:31:00.003-07:00

What's up, hackers? I know you guys don't particularly care about this subject, but I am have become temporarily obsessed with it, so here for your perusal, my research on high-level languages for the 6502!

Common-Lisp assembler - This is lisp-syntax assembler. Except you can also use lisp structures such as conditionals and loops. You can also define new lisp functions. It compiles to normal assembly. So I think it's more like macros than an actual lisp runtime. (See also COMFY)

Python on a chip - A subset of python syntax and VM!

Requires roughly 55 KB program memory
Initializes in 4KB RAM; print "hello world" needs 5KB; 8KB is the minimum recommended RAM.
Supports integers, floats, tuples, lists, dicts, functions, modules, classes, generators, decorators and closures
Supports 25 of 29 keywords and 89 of 112 bytecodes from Python 2.6
Can run multiple stackless green threads (round-robin)
Has a mark-sweep garbage collector
Has a hosted interactive prompt for live coding
Licensed under the GNU GPL ver. 2

Lots of great stuff about 6502 languages, particularly Forth. - Great info on how to implement your own languages on the 6502. Forth is an obvious choice, but I haven't found any good Forth implementations as of yet. There seem to be a lot of Forth projects that may or may not be related which I need to evaluate.

New Blog, Step Three: Privacy!

2011-06-13T20:53:00.000-07:00

This summer I have an internship with the Tor Project through Google Summer of Code. I started a blog for that and to generally talk about privacy stuff which will never be a profitable endeavor. Ironically, I'm getting paid by Google to work on it, but for Google this is an entirely non-profit endeavor in the name of charity and the general betterment of the world.

Privacy and profit have something in common in that each is an elusive goal. Thanks to my friend Drake Wilson for suggesting the related name for the new site, Step Three: Privacy!

I'll still be posting on here when I have startup and coding related matters to discuss, although I would recommend coders check out the other blog as well as there's some neat stuff on there if you do networking stuff in python.

How to Help in Eqypt: A Historical Perspective and a Call to Action

2011-01-29T07:51:00.000-08:00

It's really amazing how the censorship bar keeps getting raised. When I co-founded Freenet over ten years ago, there were lots of assumptions shared about what online censorship was and how far people would be willing to go and also about what free speech was and what people wanted to communicate online. These assumptions have carried through to the design of today's censorship resistant systems. For instance, Tor still uses SSL we used to think that no one in their right mind would block SSL because then they'd be blocking HTTPS and critical systems such as any online commerce. The essential assumption was that there was a certain level to which censors would not go and we just needed to hide our traffic below that level. This was a good assumption for a long time, but now the game has changed.

Iran was the first wake-up call. They went farther than China was every willing to go by severely throttling SSL specifically. This was really a smart move because it didn't slow down the ability to read pages on the Internet, as most are unencrypted. It did slow down Tor and the ability to log in to any sites that use SSL for logins (hopefully all of them at this point). Since logins are required for most publishing services such as email, Twitter, Facebook, etc., this throttled both the ability to send information out. Of course online commerce was affected, but they were willing to accept that. The Iran attack became the new gold standard in online censorship. All systems need to adapt to this new reality. Since SSL is now a target, SSL is no longer a good wrapper for traffic. This is why I started Dust, to provide a more modern transport layer for bypassing current censorship methods. I really believe we can make something which is undetectable and thus cannot be throttled or blocked. I think information theory is on our side here and that this is a war we can win.

However, Egypt raised the bar yet again by simply unplugging the Internet. This is remarkable as it not only shows how much farther censors are willing to go now, but also how the nature of online freedom of speech has changed. One of the classic examples we used to use to discuss the purpose of Freeenet was that China blocked access to CNN. This seems like a comically naive goal now. People aren't trying to access news from major publishers. They're organizing protests via Twitter. This is totally decentralized content, it's peer-to-peer communication.

Unfortunately, this isn't a software fix. If the cables aren't plugged in, there's no clever ways we can encode the data to get it past the censors. This particular situation is a hardware problem. The infrastructure is centralized in such a way that it's easy for the government of a country to just switch it off and so this is what happened. Some respected individuals have called for the building of a new Internet without these problems.

I think this is a very noble endeavor, but I want to be straight with you about the problems with this idea. Essentially, we've tried this and it doesn't work. We've been trying this for ten years. Building an Internet out of Wifi mesh points is like wiring a city for electricity using USB cables. The 3G and 4G wireless Internet that we have now is connected by a high speed wired backbone and this is what makes it work well. There are many problems with a entirely mesh network, but the primary one is range. Once you start looking at coverage areas and doing the math you quickly discover that the number of mesh nodes required to cover any decent areas is astronomical, particularly because you need to connect out to the larger Internet either by crossing the border to a friendly nation or connecting to a satellite link.

There is something we can do, though. We can design a custom network for these situations which, while it doesn't connect to the general Internet, provides network connectivity to people on the ground with each other. Here's a brief overview of my design:

Femtocells are superior to Wifi access points here. The computing device of choice is going to be the camera and GPS-equipped phone, not the laptop. Phones are used to drifting from tower to tower. They have hand-off protocols for switching towers seamlessly. That's why you can talk on the phone while driving down the highway. A femtocell is essentially a "fake" cellular phone tower that intercepts your phone signals and routes them over your own network connection instead of the phone company backbone. You normally get these to improve reception in areas with poor or non-existent tower coverage. Places like, for instance, Egypt right now.

My proposal is to combine portable, battery-powered femtocells with a custom backend that, instead of routing your data packets over an ethernet connection, stores the data for exchange on a store-and-forward mesh network, much as in the FidoNet network referred to in the Rushkoff article. Then, instead of having fixed towers and moving phones we have moving towers.

This is all kind of technical, I suppose, so let me break it down for you in terms of a use case. You're in Egypt and you want to get news about what's going on, send out videos of important happenings to the world, and organize with your fellow citizens to take political action. You have a phone with a camera and text and MMS messaging. There are mobile cell phone towers roaming around the city. (This is something you'd need to organize and is a whole issue in itself, but some people specialize in this kind of theory. It's solvable.) When you come in range of an access point, you can send text and MMS messages. You also receive any that have been sent to you. The access point is actually another citizen with a backpack femtocell and battery. They could be walking, although I've also seen a similar plan executed using motorcycles. The tower stores your sent messages. The towers move in such a pattern that they come into range of each other. At this point they exchange stored messages. When a tower comes in range of a phone for which it has stored messages, it sends them to the phone and then deletes them. Sending messages to people in your phone contact list works the same as always. Getting information out of the country just requires sending a text or MMS message to someone that is known to have a satellite, dial-up, or other link outside. Once the information moves through the mesh to them, they can send it on.

I think this is the right way to do decentralized mesh networking in situations like what is happening now in Egypt. This is something we can build right now. I'm ready to start on this whenever you are. The first step is that we're going to need some femtocells. After that, it becomes a software problem again, like hacking a Wifi router to run OpenWRT.

If this is a topic you're interested in, I will be giving a talk about this on March 11 at the Dorkbot SXSW event: The Vision of the Future: 2021. Come by and say hi and we can figure out how to make this happen.

Retro Indie Game Development with HTML5 - The Series

2011-01-05T18:57:00.000-08:00

Like many web developers I've become interesting in the recent developments in HTML5. Web browsers can now do things they've never been able to do and it's an exciting time. It's also an exciting time for games right now. Two phenomena in game development have arisen which make game development fun again: retro games and indie games, although they are often found in conjunction. Retro games use the old school 8-bit graphics and sound we loved when we were kids. Some of these retro titles are from studios such as the new Megaman games or Cave Story for the Wii. There has also been a rising tide of indie games such as Minecraft and the games of the Humble Indie Bundle such as Braid. Some of these have retro graphics and some like Machinarium have pretty nice art. Not to save that low resolution art isn't nice, "pixel art" has become its own genre with its own talented artists. Some of these indie games have actually been quite successful with both Minecraft and the Humble Indie Bundle raising millions of dollars in sales.

I think the simultaneous rise of HTML5, the retro style, and the commercial success of the indie development methodology have created a great opportunity for the hacker turned entrepreneur. Additionally, all of the open source code, DIY tools, and Creative Commons licensed artwork provide a relatively low barrier to entry. Take, for instance, Realm of the Mad God. This is a really fun retro indie MMO. It was actually developed as part of a contest where artists first made Creative Commons licensed art assets and programmers then used these to make a game. The map generation code is also open source. You could go out and write a game like this today and, even better, you don't have to do it in Flash like they did. A game like this could be written in pure HTML/CSS/Javascript, which is good news for web developers that have already been working in this medium for a while.

My plan is to write a series of posts about all of the great things I've discovered about developing retro indie HTML5 games as I've been working on my own game. While this may seem like a very specific topic, it opens up the doors to a variety of topics with nice concrete examples. For instance, I've often wondered, HTML5 sounds cool I guess but what is it good for, actually? In developing my game, it became quite apparent that it would be pretty much impossible without some very specific HTML5 features, not the obvious things like the Canvas and Audio APIs, but specifically Web Workers have been indispensable.

So watch the blog for future posts in this series. I'm going to start with Akihabara, the HTML5 game library specifically designed for retro games. Also let me know if there's anything specific you're interested in it and if I something to share on the subject then I'll try to make a post about it.

The Original Introduction Problem in P2P Networks

2010-06-17T22:13:00.001-07:00

BitCoin was released this week, a very interesting P2P currency based on proof-of-work with a novel method to deal with double-spending via a P2P timestamp server. Cool stuff.

On the BitCoin forums, a discussion was going on regarding how new BitCoin nodes connect to IRC in order to find other BitCoin nodes. This method was somewhat controversial because it was drawing the ire of the IRC network admins because it looked like they were running a botnet. Additionally, if the IRC server goes down then new users can't join the BitCoin network. However, what are you going to do? When you first run a node, it doesn't know about any other nodes. It's a tough situation.

This is a common problem in P2P, known as Original Introduction, although bootstrapping is also a good word for it. The problem with bootstrapping is that you can't decentralize it. Whether it's IRC or HTTP or DNS, the client needs to be hardcoded with an address or list of addresses which is sufficiently fresh that at least one of the listed addresses is still active. After the first node is reached, you are no longer in Original Introduction mode and can use the full range of techniques for decentralization, such as gossip. Unless, of course, you get disconnected from the network and all of your known peers go away, in which case you're back to bootstrapping.

There are two properties that are at odds when you chose a bootstrapping method: robustness (scalability/reliability) and freshness. Robustness is increased at the expense of freshness by caching on multiple servers, as is usually done with HTTP peer lists. Freshness is maximized (at least up to the TCP timeout) at the expense of robustness by having everyone connected, as with IRC. Of course, the key is finding the right mix of robustness and freshness because you need both for the bootstrap to be successful.

Here are some of my current favorite methods for bootstrapping:

Append list of fresh peers to executable or installer dynamically on download. People usually get the application from its official website, so the website is already a point of failure for new users. You're already hardcoding an address in the application, the address that the application will use to bootstrap. So instead just add fresh peers at the moment of download. You need some fancy code in the executable to read the list off the end, but I've implemented this in an NSIS installer and it's not that hard. Most software developers are upset by the idea of this method.

Connect via XMPP to Google App Engine application. This gives the freshness of IRC, but with more robust scaling. App Engine is mostly for writing web apps, but it provides email and XMPP handling as well. It would be simple to write one application that could handle peer lists via either XMPP or HTTP with the same handler code. I'm currently using this in an application and it works well and is very reliable. I only wish there was a second App Engine to use as a fallback because it does have occasional downtime.

An alternative to requiring all nodes to include the complexity of a protocol like IRC or XMPP is to have a few special sentinel nodes which sit on the network and collect addresses of connected nodes via the usual decentralized methods available to an active node. These sentinel nodes periodically upload fresh addresses, say via HTTP POST to a number of websites. A new node can then download a fresh address list from any of the websites which is currently functioning and reachable. If you have 5 sentinels each uploading every 5 minutes (staggered), then you'll have updates roughly once a minute. This is on par with IRC in terms of freshness and is robust as you care to make it by varying the number of HTTP mirrors and the number of sentinels.

The Truth About Mobile Bandwidth Pricing

2010-06-07T21:52:00.000-07:00

AT&T just ended unlimited bandwidth for the iPhone and people seem to be confused about what this means. As a follow-up to my post on consumer bandwidth pricing, let me break down the mobile bandwidth pricing strategies for you.

It's not really a cap, it's a pricing strategy. Also, it's not about keeping a few extreme users from ruining the network for everyone. For congestion management you'd need peak usage pricing like electricity companies use, only for geographical areas instead of (or in addition to) time-based pricing. For instance, raise the price of bandwidth in Manhattan during daytime and at the Austin Convention Center's cell tower during SXSW. Cumulative usage-based pricing doesn't solve congestion. It's just a strategy to raise prices.

Here's the breakdown of how much you'll pay per month depending on your data usage on the various networks that support smartphones.

As you can see, AT&T starts low and then after the 2GB "cap" quickly cuts across all the prices of the carriers that offer unlimited bandwidth. If you actually use less than 2GB/month, it's still a pretty good deal, second only to Sprint. At 4GB/month, it's the most expensive.

Also notice that Tmobile is more expensive if you get a 2-year contract that if you have no contract. This is their terrible new pricing plan in which they no longer subsidize phones in order to lock you into a contract. Instead, they essentially finance your phone by having you pay less up front but then more per month. When your 2-year contract is up, you will have paid more than you saved on the initial phone purchase. So if you get a Tmobile phone, don't get a contract. Just buy the phone outright.

The Truth About Consumer Bandwidth Pricing

2009-04-21T13:49:00.000-07:00

There's been a lot of noise made recently about Time Warner instituting bandwidth caps. Everyone was angry at Time Warner, whereas Time Warner claims it's losing money because of a few people hogging all the bandwidth, that usage based pricing is more fair and also necessary to pay for building up their networks, and that all of this BitTorrent traffic and streaming video is killing their networks and needs to be capped.

I have an inside perspective on this matter because when I was the Director of Product Management at BitTorrent, we often spoke with ISPs. We knew that Comcast was throttling BitTorrent traffic far before it made it into the news and I flew down to Comcast headquarters in Philadelphia to discuss the situation. I was suprised when the told me that they had plenty of bandwidth and that BitTorrent wasn't anywhere close to crushing their network. Their problem was that they don't want to sell bandwidth, a comodity with a price racing to zero. They want to sell entertainment services, which have a higher profit margin. They are therefore threatened by online video as it competes with cable TV.

The consumer ISP strategy thus has a twofold purpose: raise the price of bandwidth, and at the same time make the Internet a less appealing way to watch video. Both of these purposes are accomplished by bandwidth caps. Additionally, the new pricing models make it complicated to determined how much you're going to be paying exactly for bandwidth, allowing the ISPs to increase prices covertly. If they were to just declare that prices were going up because they felt like it, people would be very angry indeed, and it might lead to government regulation of pricing.

In order to unravel the mystery of the new pricing models, I've made some graphs that show how much you will pay in dollars for a number of total gigabytes transferred in a month. I was very suprised by the results.

To start, here is a graph of a lot of different plans, such as various Time Warner plans, AT&T DSL, and the main 3G mobile carriers.

On the bottom is gigabytes and on the left is dollars. Yes, dollars. 300 GB would costs you $140,000 on AT&T 3G. You'll notice that only the 3G providers show up at all, everything else being squished into a single line on the bottom. This is because while Time Warner is charges overages of $1/GB, Sprint is charging $50/GB, Verison $280/GB, and AT&T a ridiculous $480/GB after you exceed the 5GB cap. Everyone is mad about the Time Warner caps, but it's really the 3G caps that are totally insane. Every iPhone user is on AT&T, so when Hulu for iPhone comes out it's going to be crazy.

So don't use more than 5G of 3G per month or else you're getting ripped off. Let's compare some ISPs just in the 1-5G range to see how they stack up.

Amazon S3 is included here at the bottom just to show how much more expensive consumer bandwidth is than hosting bandwidth. The bottom tier of Time Warner service is a clear winner here, following by the original capper Comcast. 3G services are in the middle, with premium tier cable and DSL services losing. In this bandwidth bracket, you don't really get much benefit from upgrading your service.

Now let's look at ISP choices excluding 3G.

The lowest Time Warner tier wins again if you lose little bandwidth, and then Comcast wins everything else up to 250G where they have put a hard cap.

Now let's look in depth at just the Time Warner tiers.

The graph is interesting because Time Warner imposes an overage fee cap of $75. This causes the lowest tier to come out best for both low and high numbers of gigabytes. The lowest tier charges $15/month for 1GB and $2/GB for each additional GB, up to $75 in overages, meaning that your total bill is capped at $90. You therefore get unlimited bandwidth for $90 with that plan. Whereas their highest tier plan is $75 for 100 GB and then $1/GB after that up to $75 in overage charges. You get unlimited bandwidth for $150 with this plan. So the lowest tier wins and the highest tier loses. The middle tiers only come into play for medium amounts of bandwidth.

So, let's look at medium amounts of bandwidth where the multiple tiers come into play.

This graphs shows a situation similar to the one pitched by Time Warner. There are multiple tiers and you get the best deal by choosing the right tier for the amount of bandwidth you use. However, note that the goal is not to avoid overages. The goal is to avoid having your overage charges cost more than the monthly charge of the next plan up. So while the lowest tier only includes 1GB/month, it's the best plan up to around 10GB/month. Similarly, the standard plan will be better than an upgrade up to 50GB/month. The highest tier is only good for people that use >80 GB/month. And Time Warner Business Class is, as shown on all of the graphs, always just a terrible deal.

It was just discovered that AT&T DSL is implementing bandwidth caps. They have a different model because they don't have a cap on overage fees. That sounds like it would probably be a worse deal than Time Warner. Let's take a look, first at just the different AT&T DSL tiers.

This is the more classical model that you'd expect with overages. Since there are no caps on overage fees, you get the best deal by choosing a plan matched to your usage. If you guess incorrectly, you overpay. The ordering of plans from cheapest to most expensive becomes inverted from low usage to high usage.

Now let's compare the various AT&T DSL plans to the various Time Warner cable plans.

There are a lot of lines on this graph, but you only need to look at the bottom. The lowest tier of Time Warner again wins for low bandwidth. After than, successive AT&T DSL plans win. Despite the fact that their pricing structure is worse, their actual prices are better than Time Warner as long as you're good at guessing how much bandwidth you're going to use. If you're bad at guessing, only the lowest two tiers of Time Warner could ever possibly be better than AT&T DSL and only for a small range of usage. So if you're bad at guessing your usage, your best bet is to get the highest tier of AT&T DSL.

Conclusions

I was suprised by the outcome of these charts. The Time Warner caps are not that big of a deal and the AT&T caps are even less of a big deal. What you really need to watch out for is the 3G caps. Those are just totally off the rails.

The best deal for consumer Internet is AT&T DSL, even with the caps and overage fees. If you know how much bandwidth you're going to use, buy the appropriate tier. If you don't know how much bandwidth you're going to use, you're safest buying the highest tier.

If you're going to go with Time Warner, the lower tiers are a better deal. Go with the lowest tier you can and only upgrade if your overage fees are costing you more than the next tier. Never buy the highest tier or business class, they are ripoffs.

3G is a terrible deal. If you use less than 5G a month, all the 3G providers are priced the same and are not a very good deal for Internet. Use the lowest tier of Time Warner instead. Under no circumstances use more than 5G of 3G in a month, you will get ripped off big time.

Also, Hulu for iPhone is going to be a train wreck.

Diakonos: A Programmer's Text Editor in Ruby

2009-02-27T12:06:00.000-08:00

A text editor (or for some an IDE) is the most important tool a programmer has, other than the programming language itself. Religious wars over editors are inevitable because people spend so much time with their editor. Some people flip-flop, but many people become both functionally and emotionally attached. No one wants to spend time learning new keybindings when they could be programming instead.

Personally, I use nano. This is not out of ignorance, mental damage, or a deep moral perversion as my friends that use emacs and vi insist. I want an editor which is small and quick to install. It must be available on all platforms and easy to install (if there's no Debian/Ubuntu package in the main repositories, forget it). I'm not going to mess around with configuring it. And I basically just don't like vi. So nano has been winning the war for my soul for many years. However, like all programmers, I dream of a better world. I wouldn't mind a slightly (or even somewhat) better editor, but I everything I've ever tried lacked the beautiful simplicity of nano. With more features comes more hassle.

Then I found Diakonos. It's a console-based text editor (which I like because I ssh into my server and edit things as much as I edit them locally), and it's written in Ruby. It has the modern features, such as multiple buffers, syntax highlighting, and syntax-aware indentation. It's scriptable, either through the Ruby interface or through external programs (in any language) which are fed the old buffer on stdin and output new buffer contents on stdout.

Like all editors under my consideration, it has packages in the main repositories of both Debian and Ubuntu. It also has Windows and OS X binaries (also a Ruby gem for you Ruby guys). It's as quick and easy to install as nano, and through it has lots more features, they are not obtrusive. The keybindings are the "standard" Windows-style ones (ctrl-x cut, ctrl-c copy, ctrl-v paste). You can of course configure it to emacs or whatever style you want, but I am personally happy to use a similar set of keys across my editor and web browser.

I am particularly excited about finally having an editor that's not written in C. This is a personal issue. Many people like C, but I just think it's time for us to move on as a society. I have a T-shirt that says "I would code in C for love, but not for money." While you may love C, autoconf, and make, I am personally very excited about an editor both written in and scriptable in Ruby. It seems like a step towards the future. It's also nice to have a fresh codebase which doesn't inherit several decades of design decisions.

My apologies for insulting your favorite text editors and programming languages, my Internet friends. I meant no harm. Just check out Diakonos for a bit and see what you think. It has a feel which is both fresh and yet somehow also classic. A "modern classic" if you will. And it's fun. In a way I can't really articulate, it's just enjoyable to use. Also, the author is a really nice guy and the IRC channel isn't full of obnoxious jerks (#mathetes on freenode), just good folks like you and me, hacking on code. I'll see you there!

Startup Camp Austin, Feb 28th

2009-02-20T10:55:00.000-08:00

Next Saturday, Feb 28th, from 1pm-6pm, is the second annual Startup Camp Austin!

Last year's Startup Camp Austin was pretty great. A lot has changed since then in the Austin Startup Scene. It's really quite booming. With events like SXSW Accelerator and the CapitalFactory application deadline coming up at the beginning of March, we decided that now was a good time to get together again and talk about the ongoing developments of interest to Austin startups.

There are still a few slots left, so if you'd like to do a presentation, pitch, or demo, or lead a roundtable discussion, sign up on the wiki and I'll save you a slot in the program. Also feel free to just add discussion topics and we can discuss whatever anyone feels like discussing.

Also, please RVSP on the Facebook event so that we know how much food to provide.

I have to say, I'm pretty excited about this camp. The first Startup Camp was kind of scary because we'd never put on an Unconference before and I had just moved back to Austin and started my own startup. I really wanted to help make Austin a great place for startups, but it was just one person's dream. Since then, things have become so exciting! There are lots of events for startups now, from SD2020 to SXSW Accelerator. There are several new experiments in funding going on, including a startup incubator and a startup organized as a coop. Coworking spaces and BarCamps have become hot items, sprouting up in Dallas, Houston, and San Antonio as well. So many of my friends have lost or quit their jobs due to the economic turmoil and instead of feeling down about it have decided that now is a great time to start a startup. It's really a very optimistic time for startup entrepreneurs as we see opportunities in every problem.

So if you're currently at a startup, are interested in starting one, or just curious about how things are going in the Austin startup community, come to the ACTLab next Saturday. It's located on the UT Campus in the Communications Building (CMB) on the 4th floor, in Studio 4B. The Communications Building is on the southeast corner of Dean Keaton and Guadalupe, across from Madam Mam's. There's a parking lot right across the street (south of Madam Mam's) which is usually $6 to park all day.

I hope to see you there!

Austin Gets Its Own Startup Incubator

2009-01-30T09:34:00.000-08:00

I love having a startup in Austin. I think it's a great place to do a startup right now. At the Tech Happy Hour last night, one early stage investor likened Austin to a gasoline soaked pile of rags just waiting for a spark. Indeed!

For a while I've felt that the missing element in the Austin startup scene is an early stage, small investment startup incubator in the spirit of Y Combinator and TechStars. Austin is a great place to bootstrap, and angel and VC funding are available, but for many young entrepreneurs the best way to get started is with a startup incubator. You get to meet people with startup experience, you get to pitch, and you get some press. It's one of the best ways to get started, especially if you're on the engineering side and you need to meet people with business experience.

Austin finally has such a venture, and it's called Capital Factory. You can read their press release to get the sales pitch, but let me just break down the numbers for you. If you're one of the three companies picked, you get $20,000 for 5%, giving you a $380,000 valuation, which is comparable or slightly better than YC and TechStars in terms of valuation. There are of course many intangibles to compare between the various incubators, but it basically comes down to where you want to start your company: the Bay Area, Boulder, or Austin. For myself, I choose Austin!

They're also still looking for a few good investor-mentors, so if you want to help the Austin startup scene and you've got some time and money to invest, check them out. I can't wait until pitch day to see what new startups are started!

P2P Money with App Engine, OAuth, and QR Codes

2009-01-23T10:48:00.000-08:00

In honor of National Service Day, I decided to take a day off from my regularly scheduled Ringlight hacking and work on some community service hacking. In Austin we have a complimentary currency called the Austin Time Exchange Network (ATEN). There's a lot to say about complimentary currency and its role in helping economies during a downturn. However, I want to delve mainly into the technical details of my hack, so if you're interested I recommend Bernard Lietaer. The basic idea is that you can pay people for their time in ATEN currency, denominated in hours, rather than dollars. This is quite good for situations where no one has dollars they want to spend, but they do have work they want to do and get done, such as the current economy. There's no shortage of needs or workers, only a shortage of money. So let's make our own money! Problem solved! You'll still use dollars to pay taxes, your mortgage, and Wal-Mart, but you can use ATEN hours to buy local goods and services from people in Austin that accept this currency.

The goal of this project, named Austin Time Machine (ATM) is to provide a means to withdraw electronic currency into a physical paper form (cash) and later deposit paper to an electronic account. This is particularly useful for the sorts of situations which are normally "cash only", for instance festivals where it's unreasonable to expect all of the booths to have computers and Internet. Since the paper currency is backed by a separate online currency (in this case OpenSourceCurrency.org), the ATM service doesn't need to manage things like account balance. It only needs to keep track of bill serial numbers and manage authentication to the "bank" so that it can transfer credits to and from user accounts.

So on to the technical details. The first interesting bit is that OpenSourceCurrency.org supports OAuth for authenticating users. Additionally, I implemented the whole service on App Engine, which is wonderful because I don't have to run it on my server or manage uptime. However, this meant that I had to port the python OAuth library to use the App Engine API. In particular, I had to replace all of the use of httplib with App Engine's urlfetch service. This code will be useful to anyone attempting to authenticate to external services from inside an App Engine application. This app also provides a handy example of how to write an OAuth client. It's a little bit more complicated than it needs to be, but it's not that bad if you use an OAuth library to generate the signatures and such. It's basically involves just POSTing some fields to a few URLs and providing callback URLs that the website will POST back to. You pass some tokens around this way and end up with a token which, when included in a call to whatever web service you're trying to access, will serve to authenticate you as acting on the behalf of the user.

The next component of the app is the storage of serial numbers when you withdraw bills and verification of serial numbers when you deposit. Nothing particularly exciting here. I created an App Engine Model for each bill and save and access them using the standard App Engine ORM API. This is worth checking out if you haven't used App Engine before though because it's a simple example of how it works, and it's very different than SQL. Basically you need to assign a unique (string) key to each object and this is how you access them. The mechanisms you might except from SQL such as the UNIQUE keyword are absent.

With all of the nitty gritty storage and OAuth stuff taken care of, the bulk of the application is very simple. OpenSourceCurrency.org is a Rails app and so exposes a simple REST and JSON (or XML) API to do transactions. There are a couple of gaps in the API (from the perspective of this particular app) which I work around in this code. The API only lets you transfer money from the current user to a specified destination user, and you need the userid of the destination user. For withdrawl it's easy, I transfer money from the authenticated user to my own account, since I happen to know my userid. For deposit, I perform a tricky manuever. I charge the user a 0.1 hour fee, transferring it from their account to mine just like in a withdrawl. The result of that call includes their userid in the JSON output. I then take that userid and have the ATM service log into my own account (specifying credential via HTTP Auth, not OAuth) and transfer from my account to the account of the user, specified by their userid. A bit complicated! However, I'm working with Tom Brown, creator of the OpenSourceCurrency.org API, to create a simpler API.

Finally, once you've made a withdrawl, the bill needs to be generated so you can print it. This is currently done with just a little bit of HTML. A PDF export would be nice for printing multiple bills on one page, but for the prototype HTML was of course the fastest. The QR code generation turned out to be extremely simple because the Google Chart API recently added QR code support. So the QR code is just a single HTML img tag with a URL which will automatically generate a QR code. Nice!

Feel free to play this all this stuff. Check out Tom's screencast on using the ATM, the live ATM site, and of course the source code (also available as a zip).

Scalable Clustering with Thrift and SQS

2008-12-09T14:39:00.000-08:00

Since the Ringlight beta launch, we're edging up towards 100 users. It's certainly not the load that the engineers at Twitter have to deal with, but I would like to impress upon you my Law of Scaling:

Every power of ten, something different breaks (or becomes unusably slow).

So even with modest growth from 10 to 100 users, it's probably time to fix something.

Once principle of scalable design is to decouple slow operations from the user interface.

For instance, subscribers of Ringlight Personal Edition have the added feature of one-click backup of all of their files. However, this operation can take a long time to complete. Even just generating the list of files to back up can be time consuming if you have a really large number of files. Therefore, it is advantageous to move all of this out of the website and into a background process. The web application just records that you have clicked the one-click backup option and then alerts the background process that it's time to figure out exactly what needs to be done about this. This sort of architecture will keep your web page loading snappy and your users happy even on a heavily overloaded website, as you're not wasting their time making them wait for the page to load.

There are a number of ways to communicate between your web application and the background process. Of particular interest from a scalability standpoint are message queueing services such as Starling and SQS. These allow for high scalability by allowing many producers (your web server instances) talk to many consumers (your background processes).

Starling is a server written in Ruby (for Python, see Peafowl) that you run yourself. SQS is a hosted service that you pay for based on usage (number of messages sent and bandwidth used). Both are reasonable choices and have pretty similar APIs. You connect to the service, and then push strings onto a particular queue (identified by a queue name, which is also a string). Other processes can fetch strings from the queue given its name. Pretty easy! They also both have client libraries in most major languages, so integration into your app shouldn't be very difficult.

Of course, they only support strings, so if you have fancy objects that you want to send then you'll need to serialize and deserialize them to and from strings. There are of course language-specific ways to do this (Java Object Serialization, Python Pickles, etc.), but I prefer Thrift because it's fast, efficient, and is the same in multiple languages. This is handy because you can implement different components in different languages, which is sometimes useful. For instance, my web server is in Java and my background process is in Python.

Thrift also provides some additional handy components besides serialization, in particular a transport layer that provides RPC semantics over arbitrary transport mechanisms. It comes by default with socket and HTTP transports.

What I have implemented and made available for you in case you might find it useful is an SQS transport for Thrift. It effectively provides cross-language multicast RPC in a few lines of code. The key piece of code is TSqsClient, which provides the SQS transport using the boto library for Python. This is the piece that you'll need to port if you want to support other languages. The rest of the code is just for example purposes and is derived from simple-thrift-queue, which is a nice example of how to build an application using Thrift. The available methods are defined in the thrift file. It's important that they are defined as async and void, as this is a one-way transport. The producer calls methods on the stub classes generated by the Thrift compiler. These method calls are queued up SQS. The consumer gets the method calls from SQS and calls the methods on the handler class. Additionally, there are a couple of utilities. One to fetch a single message from SQS, so you can test the producer, and one clear the queue if you send too many messages.

Once nice thing about using Thrift is that you can swap out the transport easily. You can replace my SQS transport with a Starling one, or ditch queues altogether and use sockets or HTTP. The advantage of using SQS is that the producers and consumers can all be on different machines or the same machine, it makes no difference. Used together, you have a very flexible and very scalable system with very few lines of code. Just update your thrift file and handler class to use your API and everything else is handled for you!

Ringlight Beta Launch Party

2008-11-03T12:20:00.000-08:00

After almost a year of hacking, testing, and web design, I'm ready to release a Ringlight beta to the public.

I'd like all of my friends and coworkers to join me at Conjunctured to celebrate this momentous occasion. The party is this Wednesday, November 5th, from 6-9pm.

Austin is a great place to do a startup and I couldn't have made it this far without the community. Let's work together to make things awesome!

The Rackspace Cloud Event

2008-10-22T11:29:00.000-07:00

Today I attended the Rackspace Cloud Event at the ACLU stage, just a couple floors above the ACTLab. I'll skip the marketing and get right to what startups and developers might care about.

Rackspace bought Slicehost
Rackspace bought Jungle Disk
Their version of Google App Engine, previously Mosso, is now called "Cloud Sites"
Their S3 is called "Cloud Files" and is $0.15/GB per month
They will be offerering pay-as-you-go CDN for Cloud Files through Limelight for $0.22/GB

I've always had a problem with Mosso ("Cloud Sites") in that it promises to scale your app for you, but it never says what type of scaling it actually does. Luckily, one of the engineers talked to me afterwards and answered all of my questions. Here's how it scales:

A master load balancer which will be hot swapped in case of failure
A cluster of web server load balancers
A cluster of app servers
Master-slave replicated databases

So they're basically doing all the things that are good and reasonable. I didn't have a chance to ask about caching issues such as whether a memcached cluster or any other caching solution was available. Also, while they don't do automatic sharding of your database (which would be cool, but kind of insane), they do support apps with multiple master databases and will support separate replication for each master. It really beats services which just provide a cluster of app servers in terms of ultimate scalability.

Autocomplete Form Fields (with jQuery)

2008-08-15T12:55:00.000-07:00

I recently added a feature to Ringlight which lets you share a private file with a particular user. There is a field to enter the username that you want to share with and obviously this field needs to autocomplete as you type. This is a feature that users expect now and so it's not really optional.

jQuery makes it very easy to add this feature:

Download the autocomplete extension for jQuery. You'll need both the javascript and CSS files.
Add an input field to your form:
Set the cacheLength option to speed up responsiveness: options={cacheLength: "20"};
Call the autocomplete extension: $("#youAutocompleteMe").autocomplete(url, options);
You'll need a server-side script installed at the url you pass to autocomplete to return matches.

The autocomplete function will call the given url, passing the text currently in the input box in the q parameter. Your server-side code should return a newline-delimited list of matches.

That's it! It's pretty easy. The key to performance is setting that cacheLength parameter as it default to 1, which doesn't provide much caching at all.

AJAX File Upload Progress Bars (with jQuery)

2008-08-08T08:39:00.000-07:00

I recently added progress bars (actually, a percentage instead of a bar, but it was the same to implement) to Ringlight uploads and downloads. I was suprised to find that the available server-side libraries for dealing with file uploads seemed to be inadequate for adding this functionality to my website.

The basic technique for adding progress bars is relatively simple. With jQuery:

Install the Ajax File Upload extension.
Install the periodic execution extension.
Register some Ajax event callbacks to reveal the progress bar on the page when the upload starts and also to catch errors.
Call the $.ajaxFileUpload function with the URL of the upload handler script, the id of the file input element, and the callback function to handle output from the upload handler.
Have the upload handler return a Json object with an id for the upload.
Call your progress bar update function with the periodic execution model: $.periodic(updateProgressBar);
The updateProgressBar function should fetch the download status from a server-side script, supplying the upload id and a callback function: $.getJSON("fetchProgress", {id: id}, function(data) {/* update progress bar with data.percentDownloaded*/});
The fetchProgress script should return upload progress information in a Json object. I return percentDownloaded, but you can include anything you'd like, such as upload rate. You should also provide error information here, such as if the upload failed.
The callback function for fetchProgress should update the page to reflect updated progress. For instance, updating a percentage to completion could be as simple as $("#percent").empty().append(data.percentDownloaded);

This was all very simple to implement and jQuery made it possible in very few lines of code. The difficulty was in providing a percentDownloaded value. The difficulty comes from the fact that it is common for browsers to not include the Content-Length field for uploaded files. The file upload handling libraries generally solve this problem by either 1) not providing a content length or 2) loading the whole file into memory (or disk, in some cases) and then finding the length of it. Either way, not very useful for a progress bar! This total failure to handle streaming files is a common problem in libraries and if you could avoid it in the libraries you implement then the world would be most appreciative.

In the meantime, there are a number of action items that require your attention. First, calculate the file length by taking the HTTP request Content-Length field and subtracting the size of everything which is not the file in order to yield the file length. I did this with the following shoddy algorithm:

Extract the MIME boundary from the Content-Disposition field.
Subtract the size of the MIME boundary twice (there is a boundary on both sides of the file).
Subtract 2 because the second boundary has a trailing "--".
Subtract 4 because each boundary has a trailing two-character newline (\r\n).
Subtract 8 because my numbers were always off by exactly 8. I'm not sure where this additional 8 is coming from.

As I said, this algorithm is shoddy, a kludge not fit for use in production. However, it works for now! It needs extensive testing and tweaking on a variety of browsers. The next steps:

Improve algorithm so that it's robust enough to work with most browsers
Submit a patch to python's FieldStorage class to support this algorithm
Submit a bug report to Mozilla requesting that they supply content length in file uploads

For now, my upload progress bars are working, so I'm happy.

Flash Crowd Preparation - Load Testing With Siege

2008-07-29T08:34:00.000-07:00

The official launch of Ringlight is approaching and so it's time to prepare for launch day flash crowds, in particular from Slashdot (or Digg, etc.). On the bright side, these sorts of Flash crowds are not as fearsome as the used to be, in that most website hosting these days provides adequate bandwidth and CPU to handle the load. The weak point is most likely your application itself, so it's worth load testing it.

So how much load is a Slashdot flash crowd? On the order of 1-10 requests per second. If you can handle 100 then you're more than fine. Additionally, this load only lasts about 24 hours. These are small numbers as compared to the continuous load you can expect for a popular site, but scaling up for your post-launch flash crowd is good preparation for the traffic to come.

The first step in scaling is to load test your site. Don't go writing a level 4 load balancer in C with async I/O just yet. First, find out what is slow and just how slow it is. I like to use siege for this because it let's you start simple.

First, apt-get install siege. Then, run siege.config to make a new config file (you can edit it later).
Then, try the simplest possible test: siege http://yoursite.com/

Don't run siege on someone else's website as they are likely to think they are under attack (and they are!) and block your IP.

Siege will launch a bunch of connections and keep track of core statistics: availability (should be 100%), response time (should be less than 1 second), and transaction rate (you're shooting for 100 transactions/second).

By default, siege will just keep attacking your server until you tell it to stop (ctrl-C). To run a shorter test, use -r to specify the number of repetitons. Note, however, that this is not the number of transactions that will be made. siege uses a number of simultaneous connections (15 by default, set it with -c), so if you specify -r 10 -c 10 for 10 repetitions with 10 simultaneous connections, then there will be 100 transactions.

Other important options are -d to set the random delay between requests and -b to use no delay at all. The use of -b is less realistic in terms of real load, but will give you an idea of the maximum throughput that your system can handle.

I tested my site with siege -b -c 100 -r 100 in order to hit it hard and see the max throughput. I found that most pages could handle 100 transactions/second, but one page as doing a scant 5 t/s. Unacceptable! I added memcached caching to that page for caching some of its internal state and it now benchmarks at about 90 t/s. That's perfectly acceptable for the launch crowd, but this benchmark relies on cache hits. With 100% cache misses, I'd be back to 5 t/s. So the real performance is somewhere in the middle, depending on the ratio of cache hits to misses. What this ration is depends on real traffic patterns as well as the size of the cache and the number of pages. This makes it hard to say, but I can give it a shot.

The way to simulate something like this with siege is to put a number of URLs in a file called urls.txt and then run siege -i. siege will then pick URLs to hit randomly from the file. Put multiple copies of the same URL in the file in order to simulate relative weighting of the URLs. By providing a file containing all of my caching-dependent URLs, weighted based on estimated popularity, I can see how well my cache is holding up and tweak caching settings as necessary to get adequate performance.

Startup Camp Austin

2008-07-11T12:04:00.000-07:00

There's been a lot of discussion lately about Austin's startup culture.
Is Austin a good place to do a startup? Should you head to California instead? What sort of startsup work best in Austin? Where should you look for funding? Is Austin Ventures the only game in town? Why doesn't Austin have an equivalent of Tech Stars or Y Combinator?

There's a growing movement to make Austin a great place to do startups. Projects like the Startup District and various coworking spaces, both commercial and noncommercial are examining what we need to provide in order for Austin startups to succeed.

In the spirit of these disucssions, I'm organizing Startup Camp Austin, a one-day event on August 2nd where Austin's startup community will come together to discuss the issues, challenges, and advantages of having a startup in Austin.

RSVP on the Facebook Event page and sign up to give a presentation, moderate a discussion, demo your product, or pitch your idea on the BarCamp wiki.

Java and Memory Leaks

2008-06-30T13:38:00.000-07:00

People are often surprised that I prefer Java as the primary language for coding serious applications. The assume that it must be ignorance of other languages, enforced slavery by my employer, or simple insanity. I assure you, however, that while I have experience in programming with 15 different programming languages and I enjoy many of them, I still choose Java for doing real work.

This is because while first class functions, closures, and metaobjects are all very cool and fun, I don't think these are the important factors when writing, say, a web application that you need to scale up to lots of users. What really matters are the libraries and the tools. These will save you more time than not having to type semi-colons at the ends of lines.

An example of what I mean is memory profiling. I recently wrote a handy load testing tool for Ringlight which generates an up-to-date google sitemap by hitting every URL on the website, comparing hashes to see if it's changed, and updating the sitemap's lastmod field. There are currently around 200,000 pages and hitting them all at once is a good test of the memory cache, database responsiveness, average page load time, etc. Interestingly enough, this process causes the server to run out of memory and crash.

Excellent! The load test revealed a memory leak, just what a load test should do. If the application was in, say, python, I would do is run the code under the primitive profiler that's available. This would spit out a stats file. I could then write some code using the bstats.py library to sort the stats in various ways looking for area which are consuming lots of memory.

Luckily, the server is in Java, so I can get a 15-day trial of YourKit Java Profiler (there are free ones as well, but YourKit is the best). My code runs on the server, the user interface runs on my local machine. They automatically communicate over the network so that I can get realtime graphs of memory consumption as my app runs. I can take snapshots of the memory state, run tests scenarious, compare the snapshots to see only the memory retained between tests, drill down through paths of method calls that look suspicious, check for common memory allocation gotchas, etc. It's an excellent tool and it makes finding memory leaks easy.

In this case, the memory leak seems to be inside the Java database access layer (JDBC). It appears that this is because the MySQL JDBC driver intentionally does not garbage collect results. You must wrap the use of any ResultSet objects with a finally clause that will attempt to close them. Of course, this is just good coding style anyway and I had already done this in my applications database access methods. Unfortunately, I later decided that I didn't like the way one of these methods was called and so added an additional database access method and this time I had forgotton to add the finally clause. As this new method because more prevelant in my code, the memory leak got worse.

Of course I'm sure that you, dear readers, would never be guilty of such inconsistent coding practices. This memory leak is a result of my fast and loose coding style. Some might say that it is this style which leads me to use languages with good tools. Others I know prefer to write their own memory profilers, object graph inspectors, and even syntax style checkers from scratch. Personally, I prefer to spend my time writing applications, at least while engaged in the professional business of writing applications. When not at work, I enjoy inventing my own impractical languages as much as anyone.

Mini-Bio

2008-06-25T11:29:00.000-07:00

I realized as I was e-mailing an introduction to a new business contact today that I never really took the time to properly introduce myself. Here's the same miniature biography that I sent to my new contact:

I have worked for the last ten years in open source and peer-to-peer software, but in community projects and tech startups. I founded a number of open source peer-to-peer software projects, including Freenet, Tristero, Alluvium, and Project Snakebite. I've also worked in peer-to-peer Interent video delivery at Swarmcast as Senior Engineer and then at BitTorrent as the Director of Product Management.

I currently have a startup here in Austin called Ringlight, where I make social file-sharing software. It can be thought of as "peer-to-peer meets Web 2.0" or "google for your desktop". It indexes all of the files on all of your computers and makes them available on a website, so you can access your files from anywhere that has a web browser, send links to your friends, search, tag, bookmark, etc.

I also do some consulting in the areas of product design, product management, and engineering architecture.
I'm always happy to meet anyone in Austin with a startup and I'd love to hear what you're working on. Let me know if you'd like to have lunch anytime this week or next. Also, I will be at Jelly this Friday and most any Friday, so feel free to stop by if you'd like. Jelly is a co-working group that meets on Fridays at Cafe Caffeine. It's a great place to meet other people in startups.

Other things to check out if you want to know more about me: blog, personal website, twitter, LinkedIn profile.

Twitter Integration for your Website

2008-06-06T10:01:00.000-07:00

Social sharing of content is a popular feature for websites. It seems like every blog post these days is accompanied with a list of bookmarklet buttons: Digg, StumbleUpon, del.icio.us, etc. However, what about adding the ability to post links to Twitter from your website? It's not quite as simple as a bookmarklet, but it's still pretty easy to do.

The Ringlight website is written in Java (client is in python), so when I picked a library for twitter access, I picked jtwitter as it has no external dependencies.

It's so easy to use that it's almost too easy. Check out my code:

Twitter twit=new Twitter(username, password);
Status status=twit.updateStatus(message);

So easy, right? Now you can go to any file on Ringlight, click on Share on Twitter, and the link is posted on Twitter. You don't even have to be logged into the Ringlight website, so any random user on the Internet can start twittering links to my copy of Accelerando.

Cross-Platform Monitoring of Filesystem Events

2008-06-04T10:53:00.000-07:00

A recent problem with deployment of the Ringlight client has been that users with a large number of folders have been experiencing annoying amounts of CPU usage. This is because the most fundamental functionality that Ringlight provides is periodically rescanning your filesystem to automatically find changes to the filesystem. Rescan too infrequently and changes won't appear on the website when expected, confusing users. Rescan too frequently and users will complain about too much CPU usage. There are many applications that require rescanning the filesystem, from virus scanners to automatic backup programs, and they all have to deal with this problem.

An attractive alternative to rescanning the filesystem is to use filesystem monitoring events. Rather than periodically scanning to see if anything has changed, instead let the operating system notify you when something has change. Very efficient! Unfortunately, unlike a simple recursive traversal of directories, this approach has to be implemented separately on each major platform and each OS has its own pitfalls and gotchas. I will focus primarily on building this in python, although the same underlying mechanisms can be used in any language with appropriate bindings.

On Windows, filesystem events are available using the Python for Windows Extensions. It is not a particularly simple API to use, being a direct binding to the Windows system calls.

On OS X, PyKQueue offers a binding to kqueue, which is also available on BSD.

On Linux, there are two kernel interfaces, dnotify and inotify, depending on whether the kernel version is lesser or greater than 2.6.13. You can call inotify directly with pyinotify. You can also use a more generic library such as Gamin, which will use either inotify or dnotify, whichever is available. Of course, really old versions of Linux don't even have dnotify, and you'll have to fall back to periodic rescanning.

Every platform's filesystem monitoring API is different and each has different issues, however they generally share a common set of issues as well:

Network drives don't generate events - you'll have to use periodic scanning for these
Every folder to be watched must be registered separately - you can't request notifications for an entire directory tree. You can to register all the directories separately and when a new directory is create you have to remember to register it as well. You sometimes need to keep a file descriptor around for each directory you're watching, so watch out for running out of file descriptors.
No special handling is done for shortcuts, aliases, or symlinks - if you're monitoring, say, a directory, and that directory has a shortcut (or the equivalent for that OS), you need to monitor two objects: the shortcut itself (in case its target is changed), and the targeted file or directory.
Sometimes deleting a directory won't send deletion events for files in that directory or subdirectories. You have to maintain a copy of this information yourself and perform a virtual recursive delete on your database when the parent directory deletion event is received.

The Ringlight client, being a cross-platform application the primary function of which is to monitor filesystem changes (and report them to the server, where the real fun happens), naturally takes all this into account. I am planning on release the filesystem monitoring core of the Ringlight client as open source, as there are no good cross-platform filesystem monitoring libraries available and it's really a shame that people have to reimplement all of this for their applications.

By the way, the users seem quite happy with the new version of the client that users filesystem monitoring events. No complaints about excessive CPU usage anymore!

Adding Search (with Lucene)

2008-05-28T14:54:00.001-07:00

The time has finally come for me to add search functionality to Ringlight. There is enough content now that just clicking around is getting tedious. After all, it's up to almost 200,000 files now.

There are a number of considerations to make when adding search to your site. For instance, you can usually get by pretty well with just integrating Google search into your website. This is fast, easy, and doesn't require messing with your backend code at all.

However, this is not really what I want. I want to let users search for files, not web pages, and I want the results integrated nicely with everything else. For instance, it would be cool to use a search query as a radio playlist like you can do on Hype Machine. So I'll need to build my own search engine.

This is not really that hard to do. I would recommend you read some articles and then download Managing Gigabytes for Java. Those articles are by Tom from AudioGalaxy. You may remember AudioGalaxy as the best thing to happen and unhappen to music in my lifetime. I know do. More importantly, it was deliciously scalable and for the most part it was just a search engine. So don't go writing one without learning some tips from the best.

I'm sure that a little engineering and MG4J could produce a highly scalable search engine. However, I didn't really want to spend that much time on it, so I went with a higher level solution in the form of Lucene for Java. There is also a popular version for Python. I would recommend waiting a while if you're considering using Lucy (Lucene in C with Python and Ruby bindings) because I don't consider it mature. I'd also stay away from layers on top of Lucene like Solr because if you're looking for tools to make Lucene easier to use then you're missing the point that it's already easy to use.

Download the official Lucene release and you'll see that it comes with source code for a demo. One class, IndexFiles, shows you how to add information to the search index. Another class, SearchFiles, shows you how to search the index to retrieve items. Like most demo code, it's both too simple (doesn't provide enough use cases of the library to let you fully understand it) and too complex (has a bunch of command line arguements that it has to parse and such). However, it will do. I have working search for my whole site after two days of fiddling around and working on it part-time.

Napster DRM-Free, The Future and Past of Online Music

2008-05-21T10:21:00.001-07:00

As of today, Napster now offers DRM-free MP3 downloads. In one sense, it seems that the war is over, and we won. Let's take a look at the history and future of online music distribution.

First there was Napster, one of the original three peer-to-peer systems (along with Freenet and Gnutella). Unlike the other two, it was a commercial venture focused on exclusively on music distribution. It became the controversial flagship of the peer-to-peer movement and the conversation about p2p became focused exclusively on the online music distribution debate. While I was going to conference to speak about Freenet, I would introduce it as a censorship-resistant platform for protecting the freedom of speech of political dissidents and the audience would ask me, "How are artists going to get paid?", the jingoistic slogan propagating by the Recording Industry Association of America. You could argue that that wasn't the point, but by then you'd already gone off the topic of how to revolution media distribution to be efficient and fair and were now talking about the online music debate again. It wasn't even really possible to distribute mp3s over Freenet, but yet it kept coming up again and again.

After Napster, Grokster, and most notable AudioGalaxy (the best thing to happen and unhappen to music in my lifetime) were shut down, the p2p hackers went mostly underground, becoming assimilated into a number of small startups acquired by big companies. No one wanted to touch music at all. Personally, I stopped buying CDs as I had no desire to contribute funding the the organization attacking my friends, my livelihood, and everything interesting going on in my field. I listened exclusively to my collection of unsigned, obscure bands from AudioGalaxy (bands that had paid to be listed on AudioGalaxy so as to get more exposure) for many years. Have you ever heard of The Blood Group? So good.

Online music distribution was dead, and then Apple stepped in, with a not particularly new, but well-executed plan to make money off of online music sales by selling an expensive player for their exclusive DRM format (the iPod). This was only the first step of the plan, however. Step two was to bring in DRM-free MP3s at a higher price than DRM files, offering the RIAA what they really wanted from MP3 sales, which was making even more money selling mp3s than CDs. This is, of course, insane, as CDs contain much more value than MP3s in the form of cover art liner notes, a physical object to hold, and of course the costs of CD distribution are higher as well. Having a higher price for something which has both lower value and a lower cost is just not sustainable. However, it must have sounded quite appealing to the record industry that while CD sales were dropping they could actually make more money that ever before. I'm sure those power point charts went over very well in meetings. Then step three from Apple, readjust the price of DRM-free MP3s to be the same as DRM tracks. As iPod sales decline (because everyone already owns an iPod), Apple switches to a business model not dependent on the iPod and becomes the number one online music distributor. Apple has become what Napster and AudioGalaxy tried to be, the future of music distribution.

So in that sense, we won. Go p2p hackers! Oh but wait, this has nothing to do with p2p hackers, does it? This is just online music distribution. And in this sense, we've lost, or at least are still at war. Peer-to-peer became demonized because of its use for online music distribution. Now that online music distribution is okay, peer-to-peer is still getting picked on. The content industries would have you believe that p2p is only for morally corrupt kleptomaniacs bent on stealing and never paying. This doesn't explain why Limewire is still around as a commercial entity, making money off of Limewire Pro, or how services like TorrentFreedom manage to exist charging $17/month (more than an Napster subscription when they were doing subscriptions).

The sad fact is that the focus has been on not what you are doing (distributing music online), but how you are doing it (peer-to-peer). This is tragic because peer-to-peer distribution is a technological revolution as important as the printing press, whereas online music distribution is only moderately interesting in the long term, being just one example of the future of media distribution being over the Internet.

Download Party

2008-04-25T09:30:00.000-07:00

There will be a Ringlight Download Party this coming Wednesday, April 30th, at 7pm at Tech Night at Monkey Wrench Books (110 E. North Loop).

You can check out the full event info on the Ringlight news blog.