organized chaos

What's happening?

2010-05-05T20:24:00.001-07:00

Worked 12 hours then swam 3000 yards after running 4 miles at a 7minute pace. Wrapping up some emails then crashing.

Ask me anything, even anonymously

are you a hacker?

2010-05-05T13:42:00.001-07:00

I'm a hacker in the MIT/FSF sense of the word. The Jargon File defines hacker as "A person who enjoys exploring the details of programmable systems and stretching their capabilities, as opposed to most users, who prefer to learn only the minimum necessary." I think this is an apt description of me.

Ask me anything, even anonymously

Google Wave is to the Internet what Lost is to TV

2009-10-19T21:43:00.000-07:00

I remember when the TV show Lost first came on the air. People loved it. People who hadn't even seen it loved it and spewed their praise. Years went by and that excitement dwindled but you still had the die hards, screaming for more. Eventually, not wanting to be entirely left out, I sat down and watched an episode. I just didn't get it. I mean I really didn't get it. I wanted to like it but I ultimately just had no clue what was going on.

Google Wave is kind of like that. I got my invite last week and hopped on, extremely excited. But there were no people to interact with. It was like that big desert island in lost. There were supposed to be some people, somewhere on the island, but you don't know their intentions or who they are. I just kind of wandered through the mass of with:public waves trying to catch one that could hold my interest but found nothing.

Unlike Lost though, I think Google Wave just needs critical mass to succeed. I remember when I was an undergrad at Purdue I did the math on GNUnet and determined that in order for it to succeed as a straight P2P network it would require content to be seeded. I went as far as determining that all major P2P networks must have been seeded at some point.

Fortunately Google has the ability to seed Wave with plenty of interesting content and people, more than enough for it to succeed. It will be interesting to see whether Google sees this as another play thing and discards it, as it has with Dodgeball, Notebook, and several other products or lets it play outside the sandbox with all the other big kids like Search, Mail, Maps, etc.

I am bmatheny on Google Wave.

Migrating from Akamai to CloudFront

2009-09-23T20:08:00.000-07:00

There seems to be some confusion on basic usage of S3 and CloudFront, and how they are related. There are also some gotchas when it comes to using the services that may not be obvious at first glance. I recently moved some data from Akamai to S3/CloudFront and had to 'translate' concepts for former Akamai users. Below are some of the items I addressed.

Concepts

S3 is a simple key/value store with an HTTP interface.
A key is just a string but you can think of it as the filename.
The value associated with a key is referred to as an object.
The top level container for objects on S3 is referred to as a bucket.
CloudFront is a CDN that uses S3 as an origin server.
CloudFront has the notion of a distribution. A distribution is simply an S3 bucket that can be served via CloudFront.
A CloudFront distribution is associated with a single S3 bucket.
CloudFront is a CDN, S3 is more like an HTTP based file server

Gotchas

CloudFront offers no purge mechanism like Akamai does.
You will need to use different keys (filenames) for CloudFront objects that change since they will not expire.
S3 has no real notion of users or roles, natively. An access key and secret key are used for authentication. If a user leaves your organization who had these keys, you will need to reset them.
Because the URI for an object on S3/CloudFront refers to a key, the string must be an exact match. This also means that there is no native way to handle double slashes. If you are prone to referencing files with a double slash, this can be a problem.

Best Practices

Create a CNAME (different ones) for both your CloudFront distribution and your S3 bucket if the objects contained in them will be consumed by a web browser. This will help you be flexible if you ever want to point your CNAME at another CloudFront distribution.
Enable logging if you are publishing static assets, in particular JS/CSS that are subject to change. This will help you determine whether content is still in use.

Resources

Really the only additional thing we needed, besides some user education, was the notion of actual users with roles. We needed regular users that were tied to a particular bucket and had basic rights (upload, download, etc) and admin users that could create new buckets/distributions, etc. We decided to go with Bucket Explorer for this. Bucket Explorer essentially sits on top of S3 and provides an explorer like interface for users. In addition, it allows you to create BE users with their own roles (admin,user), usernames and passwords. The nice thing about this is I don't have to hand out the access and secret key to a large number of users.

Back at it

2009-09-22T19:54:00.001-07:00

Back at the end of July I ended my tenure as VP of Product at Compendium Blogware. I had a wonderful experience with that company and had the good fortune of working with a really talented, smart group of engineers. Despite that, as we neared the end of our first OEM integration, I found myself looking for a new challenge. Compendium had gotten to the point of being on auto-pilot from a product perspective and the big challenges that I originally joined for had been tackled.

Towards the middle of July I was contacted by a recruiter with local Q&A/Search service ChaCha. ChaCha was looking for a replacement for their recently departed VP of Operations which didn't sound like a good fit for me but I was interested in the scale and the problem space and agreed to talk with them. After several rounds of interviews I ended up accepting a position as VP of Engineering and started on August 10th.

Since joining ChaCha I have also taken on the operations, QA and helpdesk teams as well as some PM and UI responsibilities. The issues and problems that I expected to have to tackle are as challenging as I thought they would be but of a completely different nature. ChaCha has an excellent team and although I'm still the "new guy" I've been able to make (from my perspective) a lot of positive changes, particularly with respect to team structure and product development processes.

So, now that I'm not at a blogging company I'll be getting back to my trusty blogger blog and writing again. Looking forward to it.

Oracle does NOT enter the AWS Cloud

2008-09-22T14:43:00.000-07:00

Okay, seriously? Did the announcement that oracle was entering the AWS cloud really get sent out today? Don't get me wrong. I think Jeff Barr is great and I love AWS, but let's be clear about what this announcement really means.

When I saw this headline I thought, "Awesome, Oracle is lowering the barrier to entry for SMB customers." This isn't the case. It's true that Oracle is making it easier for people to boot up a presumably properly tuned Oracle instance in the cloud. It's also true that Oracle will support certain EC2 hardware configurations in the cloud (up to 8 virtual cores although you can license it for 16). However, when Jeff Bar makes a statement like the following I get confused.

The variability and flexibility of cloud-based licensing has perplexed users and vendors for some time now. Now that a large software vendor has made a clear statement of direction here, we should see more and more cloud-compatible licenses before too long.

Let's dig into the Oracle licensing terms for a minute. If you read the licensing terms it becomes clear that two things have happened:

Oracle will support certain versions of their flagship product running on certain hardware configurations in the cloud.
Oracle will license certain versions of their flagship product running on certain hardware configurations in the cloud.

I'm sorry, but does Oracle just not get it? This type of a license is the exact opposite of the utility model. While the hardware (and software, in the case of Red Hat, S3, SQS, etc) is "pay for what you use", Oracle has decided that you will pay whether you are using the software or not. On Red Hat's Cloud Computing page we get the following quote:

Cloud computing changes the economics of IT by enabling you to pay only for the capacity that you actually use.

This is what I always believed was the power of cloud computing. Pay for what you use. I hope Oracle does not become a model for other software vendors as Jeff states. I hope companies take a page from Red Hat's book in this case, especially if they are looking to enter and stay competitive in the SMB market.

Digg: Still not interesting

2008-09-19T18:06:00.000-07:00

As some of you know, during my job search last year one of the places I interviewed that ended up making me an offer was Digg. Joe Stump, who is now the lead architect at Digg was someone I knew from the Seattle PHP community and one of the folks that interviewed me. Being interviewed by someone you know is nice because you have a shared set of experiences that allow you to ask good questions that can help you make a better decision about whether or not to accept an offer.

One of the questions I asked Joe was, "are you learning?" to which Joe essentially responded (and, I'm paraphrasing) that he wasn't learning much but things operated on the largest scale at which he had been involved which made things interesting for him. Some of the technology points I inferred at that point, knowing Joe's background and that he's a bright guy:

Typical LAMP stack
No web services
Database sharding
'Legacy', organically grown code base

This was my opinion after talking with several engineers. Having worked in that environment for several years, it wasn't exactly what I was looking for and I ultimately ended up declining the offer.

In a series of recent blog posts, Digg engineers including Joe have begun describing the system and software architecture.

One of the things that strikes me is that from a technology perspective, not a lot has changed in a year. The typical high traffic LAMP system still consists of:

Caching (Memcache)
Distributed file system (MogileFS)
Monitoring (Nagios)
Asynchronous Processing (Gearman)

It's about as vanilla as it gets from an architecture perspective. But what's wrong with that?

Clearly Digg has been successful and as such their approach to technology has obviously worked. Anyone that has been tasked with scaling a web application is going to recognize the building blocks that Digg is using. However in not building a distributed system (as Digg has decided to do) you will run into some of the following issues: increased coupling of software components, longer ramp up for new developers, inability to update individual system components, difficulty in parallelizing development tasks and additional risk in new releases.

Let's use a Unix pipes analogy for a minute. Assume that each component in a software system is a unix tool; ls, grep, tail, etc. Imagine the command you are running is:

    ls /bin/ | grep cat | tail

Each of these applications handles a very specific piece of functionality. You can use each application in isolation. You can upgrade any of these applications without affecting another. Different developers can work on each application in isolation. There are some obvious advantages to the Unix approach. This is one way you can think of a distributed system but instead of pipes you're using an IP based transport (probably) and instead of command line options you're using a well defined API.

Now imagine an application called lsgreptail. It's a single application that handles all of the above functionality. You lose the ability to use each part of the application in isolation (no mixability). The code base is larger so it's more difficult for developers to get up to speed on it or become an expert with it. Making a change to functionality in directory listing (ls) requires reinstalling the entire application. Tracking down a performance bug becomes more difficult due to the lack of component isolation. There are some obvious drawbacks to this approach in software development. This is how you can think of how Digg (and many LAMP based sites) has built their system.

The point is this; there is more to scalability than the number of simultaneous users you can support. As your business grows and becomes successful, scaling your development team is just as important and becomes increasingly difficult on a monolithic lsgreptail style application. Digg is digging their own hole (no pun intended) by continuing to build their system in this fashion.

AWS CDN - Super Sweet

2008-09-18T08:31:00.000-07:00

One of the very first things I did with AWS (Amazon Web Services) was to use S3 (Simple Storage Service) and EC2 (Elastic Compute Cloud) to build a CDN on top of it. A CDN is essentially a way of distributing static content to your users rapidly in a scalable fashion. I built mine by publishing data to S3, using UltraDNS to distribute users requests to an appropriate availability zone in EC2 (east coast, west coast) based on their geographic location, and serving the request out of S3 but from EC2. Many people choose not to go this route for reasons of simplicity and just serve content out of S3. Well, now Amazon is going to do all the hard work for you.

If you are on the early warning radar of Amazon Web Services, I'm sure that you received the following email this morning just like I did:

...we are excited to share some early details with you about a new offering we have under development here at AWS -- a content delivery service.

This new service will provide you a high performance method of distributing content to end users, giving your customers low latency and high data transfer rates when they access your objects. The initial release will help developers and businesses who need to deliver popular, publicly readable content over HTTP connections. Our goal is to create a content delivery service that:

Lets developers and businesses get started easily - there are no minimum fees and no commitments. You will only pay for what you actually use.

Is simple and easy to use - a single, simple API call is all that is needed to get started delivering your content.

Works seamlessly with Amazon S3 - this gives you durable storage for the original, definitive versions of your files while making the content delivery service easier to use.

Has a global presence - we use a global network of edge locations on three continents to deliver your content from the most appropriate location.

You'll start by storing the original version of your objects in Amazon S3, making sure they are publicly readable. Then, you'll make a simple API call to register your bucket with the new content delivery service. This API call will return a new domain name for you to include in your web pages or application. When clients request an object using this domain name, they will be automatically routed to the nearest edge location for high performance delivery of your content.

Why is this significant?

Lowers the barrier to entry for small businesses wanting to use a CDN
Reduces the need to do DNS based geo-distribution on your own
Allows you to take advantage of something you are already using (AWS S3)
Allows you to simply 'enable' the service for existing items stored in S3

Given the expense of CDN services from companies like Akamai, Limelight and Level3 as well as the term commitments (you typically negotiate a rate in a fashion similar to bandwidth), many smaller companies have often avoided using a CDN. This is despite the fact that using a CDN is one of the easiest ways to significantly improve perceived page load times for end users. By allowing users to pay for a CDN via the utility model that has become so popular with AWS, this opens the door for Joe Developer to simply start out using a CDN and not have to make those kinds of trade offs.

Very cool stuff. If anyone from Amazon happens to read this, please add me to your beta group :)

Practical TDD

2008-09-16T20:01:00.000-07:00

It took me several years to drink the TDD kool-aid but now that I have I'm addicted. It's not that I didn't want to automate my testing, it's just that it wasn't particularly practical for me to do so. Having worked at startups over the past several years, I have never been able to find that balance between producing new code and appropriate test coverage for that code.

The problem has typically been that the testable API changed frequently enough that I spent as much time or more updating tests as I did writing new code. This was the problem for unit tests and as such I never seem to get around to writing them or using continuous integration tools like CruiseControl. However, at my current job we have managed to create a TDD methodology that works particularly well for us. It essentially works like this:

Agree upon web service API
Write Unit Tests that cover the new service API
Iterate on code until all service tests pass

The primary difference between this and any other testing methodology is that we focus on testing our web services as opposed to the underlying API's. This gives us a few very concrete benefits:

The front-end team can begin coding against the service API immediately.
Increased test coverage with less tests due to service dependencies.
Immediate feedback on work in progress.
Breaking API changes caught immediately, reducing impact on customers.

The introduction of continuous integration via CruiseControl along with 100% service coverage has allowed us to immediately see the benefits. The number of bugs introduced into our production environment has been reduced a measurable amount since creating the test framework.

Resources

MySQL Multimaster Replication in an Asynchronous Environment

2008-09-15T18:11:00.000-07:00

By design, MySQL replication occurs asynchronously. That is to say that the replication on a slave doesn't necessarily occur at the same time as on the master. In a multi-master replicated environment (assuming two masters), each master is a slave to the other master. There are a few gotcha's to consider when creating or editing data asynchronously in a multi-master environment. You can get bitten by these issues if using AJAX, threads, or even loading images that are database protected or backed. It's even possible to run across these in an entirely synchronous environment if the replication lag time is high enough.

Let's assume for this discussion we have the following table:

CREATE TABLE users (
    user_id INTEGER UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    user_name CHAR(64) DEFAULT '',
    user_password CHAR(64) DEFAULT '',
    user_last_login DATETIME DEFAULT '0000-00-00 00:00:00',
    UNIQUE KEY (user_name)
);

Let's also assume that we have two servers, master1 and master2.

Auto Increment Conflict

Assume that the following is submitted to master1:

INSERT INTO users VALUES (0, 'user1', 'a609316768619f154ef58db4d847b75e', '1979-09-23');

and let's label this as e1 (event1) and assume that master1 assigns user_id 1 to 'user1'.
The following is then submitted to master2:

INSERT INTO users VALUES (0, 'user2', 'f522d1d715970073a6413474ca0e0f63', '1984-01-02');

and let's label this as e2 (event2) and assume that master2 assigns user_id 1 to 'user2'.
Oops. Now when replication occurs on the other slave, we end up with 'user1' and 'user2' having different values for user_id on each server.
This topic has been covered in depth elsewhere so I won't go into details. For fixes and more information see Advanced MySQL Replication Techniques. Note that I recommend using a UUID/GUID instead of AUTO_INCREMENT to avoid this type of problem however the MySQL function UUID() doesn't work with statement based replication.

Uniqueness Conflict

Let's assume that you have some AJAX code which attempts to create a user, 'user1' which results in the following SQL statement:

INSERT INTO users VALUES (0, 'user1', 'a609316768619f154ef58db4d847b75e', '1979-09-23');

being submitted to master1 and let's assume that for whatever reason you attempt to create the user a second time (timeout occurred, side effect causes another create to happen, etc) against master2.
You would think that the UNIQUE constraint would prevent the creation from occurring the second time however whether or not the statement is executed on both servers and in what order depends on a variety of factors including server lag (the amount of time between a statement being executed on one server and replicated and executed on the second server). This problem means that you can end up with user1 being created on both master1 and master2 but having two different user_id's.
How can you avoid this pitfall? Avoid asynchronous, identical create statements. Make the call synchronous. Additionally you can configure your application to only write to one database for certain statements, essentially from an application level reverting to a typical master-slave replicated environment.
Yes, I have run into this in a production environment.

Update Conflict

Let's assume you submit the following update request to master1:

UPDATE users SET user_last_login = '2008-09-15 17:58:30' WHERE user_id=1;

which is executed on master1 at the time set in user_last_login plus 1 second (at 2008-09-15 17:58:31).
Now the following is submitted to master2:

UPDATE users SET user_last_login = '2008-09-15 17:58:32' WHERE user_id=1;

which is executed on master2 at the same time set in user_last_login plus 1 second (at 2008-09-15 17:58:33).
Lastly, the update on master2 is replicated to master1, and the update on master1 is replicated to master2.
Here is what was executed on master1:

UPDATE users SET user_last_login = '2008-09-15 17:58:30' WHERE user_id=1;
UPDATE users SET user_last_login = '2008-09-15 17:58:32' WHERE user_id=1;

and on master2:

UPDATE users SET user_last_login = '2008-09-15 17:58:32' WHERE user_id=1;
UPDATE users SET user_last_login = '2008-09-15 17:58:30' WHERE user_id=1;

Now we have different values on each server. Uh oh.

Delete Conflict

This can occur when a delete occurs on master1 and before that delete is replicated to master2, an update to the PK used for the delete in statement1 on master1 is executed on master2. Imagine the following is executed on master1:

DELETE FROM user WHERE user_name='user1';

and the following is executed on master2:

UPDATE user SET user_name='user3' WHERE user_name='user1';

Now we have inconsistent data on each server. Uh oh.

Summary

In short, there are a variety of challenges to overcome in a multi-master replication setup. These problems are exacerbated by asynchronous operations on your data set. A few bullets of advice:

Reduce complex transactions to be written to a single server.
Monitor server lag.
Be prepared for failure. The more you distribute your data set and scale your service, the more you will need to deal with failures.

References

Google Code as Personal Wiki/VC Tool

2008-09-07T19:43:00.000-07:00

I figured I would put this up as some folks might find the idea useful. For a long time I've wanted an externally available, free, reliable, hosted environment that had a personal wiki and version control. I would have loved to find a Trac+Subversion environment but I didn't trust any of the free ones out there. I've got tons of documentation that I create that seems to get lost in a slew of .txt files in my home directory. Likewise I've got lots of sample code that gets created as Foo.java, foo.js, foo.php etc and ends up disappearing. Google Code is a hosting service that has version control via subversion, issue tracking, wiki pages and a bunch of other features that I didn't really need. Today I noticed the link on Google Code saying, "Create a new project", so I thought, "What the heck?" Looking at the TOS and FAQ, there is nothing that prevents me from using this as a personal wiki and version control system. I already keep all of my source under a friendly LICENSE and have no problem doing the same for scripts and documentation as well. I put everything I would need to checkout my home directory (bash scripts, vim files, etc) into subversion. Perfect for when I hop on a new machine. Also good for synchronizing changes between environments. I also threw up a few small java projects that have been sitting out of source control for too long, I'll keep adding more as they come up. I also got started putting a bunch of the files in my local doc/ directory into the wiki. Fortunately I've been using the Trac/MoinMoin syntax for a long time so it shouldn't be too difficult to make the transition from local storage to remote. One of my favorite features so far is that I can check out my wiki onto my local workstation and make changes there with my editor of choice. Very cool. If you're interested, here are some links:

So far I only see two downsides. First, I have to be very careful not to commit anything sensitive to the repository as it's public. This generally isn't a problem but I do double check my commits. Second, unless I switch from Ant to Maven in the near future, jar files are going to send me over the 100MB limit sooner than later. I wonder why I don't have storage similar to GMail or Picasa. Oh well.

Beliefs and Programming

2008-02-17T07:51:00.000-08:00

I don't post too often on here these days. I've moved to blogging with my current employer, Compendium Blogware. You can find new posts here. Occasionally though, there is a post that doesn't quite belong or isn't quite appropriate for the corporate world. This is one of those posts. Michael Kimsal put together a survey called Religious affiliation and software development languages, which you can also discuss on his blog here. I downloaded the data set and made the following further analysis:

Found the top 25 languages by the number of people who filled out the survey
Found the top 5 religious affiliations for each language
I normalized all Christian religions into the affiliation 'Christianity'
I grouped agnostic and atheist declarations into 'AA'

Given that only ~3815 people took the survey, not a whole lot could be drawn from the numbers. However, here is what I found, draw your own conclusions.

The top 10 languages in order were: Python, C, C++, Java, Javascript, Ruby, PHP, Lisp, Perl, Haskell
The top two affiliate declarations were: AA (Atheist,Agnostic), Christian. After that Buddhist was most common.
Without normalization, the top declarations where Atheism followed by Agnostic followed by some variety of Christianity.

I'm not sure what the target group was. I didn't even know about the survey until after it was closed. However, I would say the results are inline with what I have observed in my own geeky social circles. Analytical people tend to question doctrine.

Amazon Opens Up EC2

2007-10-16T04:56:00.000-07:00

Amazon today opened up its EC2 (Elastic Cloud Computing) web service in an "unlimited beta". What does that mean? It means if you have an AWS account, you can now sign up for EC2 without a long wait to become part of the beta group. I haven't been blogging much lately as I joined a startup in Indianapolis about a month ago. Between the move from Seattle and the settling in I've been pretty busy. Now that things are starting to slow down a bit, one of the things I'm investigating is leveraging EC2/S3 for my company. If you don't know those acronyms, S3 (Simple Storage Service) is essentially storage on demand and EC2 is compute on demand. The pricing uses a utility model, e.g. you pay for what you use and that's all, and the pricing is competitive. I did a cost analysis of hosting with EC2 vs a standard leased colo situation and EC2 was significantly less expensive, like, 75% less expensive. I still don't like the bandwidth pricing but I hope that will change over time as more users get on EC2. I've played with EC2 for the past few months but hadn't found the perfect app for it. One of the challenges I'm currently facing is that at my new company, users upload lots of content. That content can vary in size (up to 10MB), is immutable, and is displayed/downloaded potentially many times. I don't want to have to worry about storage and backups of this content. Once the amount of data starts to move into the TB range, it becomes costly to effectively backup and store that much content. Enter EC2/S3. You can imagine several EC2 instances that handle storage and retrieval of content from S3. A simple web service on top of this system allows for users to upload content essentially right to S3, and anyone can view content. Since no ACL is needed for the content, and I don't have to worry about SSL, this is for me a perfect application. I am also currently aware of another EC2 customer using the platform for logging web requests. When users load a web page, they download a web bug from EC2. Since web logging and analytics can be potentially a costly and compute intensive application, this is another great way to utilize EC2. We (developers, technologists, hackers) are only beginning to see the possibilities of having a platform like EC2/S3 at our fingertips. On another note, I am again hiring developers in Indianapolis. If you are, or know, bright hackers in the area or that would be willing to move to the area, please contact me.

One Time Passwords for Web Apps

2007-08-08T23:47:00.000-07:00

I recently did some traveling through Europe, and as I did I encountered my fair share of Internet cafe's and sketchy net connections. In the Internet cafe's I worried about keyloggers, screen capture utilities and rootkits. On the sketchy net connections in hotels I was primarily concerned about sniffers on the wire. In all, I got to thinking about one time passwords for web applications, and why they seemingly don't exist. One of the things I started thinking was, many people have a cell phone. Why not replace your SecurID card with a cell phone? When you go to log onto a site from an untrusted location, have an option where users can check a box and enter in a pin instead of their password. Once successfully entered, a user receives a text message with a one time password they can use for a short duration of time. The user then uses their pin, along with the one time password to gain access to the site. This would be easy and inexpensive to implement as a web service that you could offer to third parties, so why has no one tackled this problem? If you know, let me know.

Back from Defcon

2007-08-07T19:35:00.001-07:00

I spent this past weekend at Defcon 15. This is my 8th year going to Defcon and the conference keeps getting better. I got to meet up with some people I haven't seen in a few years so that was excellent. This year I mostly went to non-technical tracks, I went to what I would consider 'geek' tracks. My favorites were:

Creating Unreliable Systems, Attacking the Systems that Attack You
GeoLocation of Wireless Access Points and "Wireless GeoCaching"
Being in the know... Listening to and understanding modern radio systems

http://www.radioreference.com/

Hardware Hacking for Software Geeks

http://www.sparkfun.com/

Satellite Imagery Analysis

I got some great info from the "Hardware Hacking for Software Geeks" talk. I'm planning on building a micro-controller driven camera that acquires location via GPS and submits photos via Bluetooth. This probably already exists, but it should be fun to build. Wikimapia is also a really cool site. Highlights of the weekend included the first Defcon wedding and an undercover reporter being outed.

On the Ethics of Contracting

2007-06-26T22:09:00.000-07:00

For the past couple of months I have been doing contract work for a variety of local companies. When someone takes you on as a contractor, they have certain expectations about what you bring to the table. In particular, clients have an expectation that you bring particular expertise to the company and can help them solve a particular problem more quickly then they could do on their own with their given resources. This means you are, as a contractor, particularly well suited for startups, short-term projects, acquisitions and mergers. In many circumstances, you are brought in as a domain expert and simply asked to do the "best" thing for that company while still solving their problem. There are no or very few technology requirements. Herein lies a problem of ethics. If you determine that the best solution is a technology that you have very little or no experience with, do you have a responsibility to inform the company of that fact and should you charge them for the time it takes you to get up to speed? First, I'm not sure how you can recommend a technology if you have zero experience with it. Yet, I've seen it happen. If you have no experience, how do you know that the solution will meet their expectations? I don't care how much you've read about something, experience matters. Assuming that you take some time to work with the technology and then make a recommendation, you have a responsibility to let the company know what your experience level with the technology is. This mitigates risk, and allows the company to make an educated decision on how to move forward. Also ask yourself the question, "How much is this recommendation based on my own personal desire to become an expert on the technology?" If it plays a large part, do the right thing and at least reconsider the recommendation. If you genuinely selected the technology because it is the best fit, read on. If you make a recommendation to use a technology that you have very little experience with, should you charge the company for your learning time? Assuming you let them know that you aren't an expert, and they still want you to handle the development, I still don't believe you should charge for learning time. The only time I believe that is appropriate is when the technology is specified and you aren't an expert. It shouldn't happen, but it does. So, what do you think? Do contractors have an ethical (and perhaps legal) responsibility to disclose their level of expertise with a given technology and should they charge for the time they spend learning? Until now, I have felt alone in the thought camp of responsible disclosure and appropriate billing. What do other contractors out there do?

Distributed Computing Failures

2007-06-22T19:59:00.000-07:00

I went to a talk several months ago given by Alan Robins, a Principal Engineer in the distributed systems engineering group at Amazon. The title of the talk was something like, "Performance and Availability" but the focus of the talk was much more on the how and why of what distributed computing technologies had failed at Amazon. It was really interesting. I took about three pages of notes, and they're more or less verbatim below. I wish he had released the slides, there was a lot of really good information that I was unable to get down on paper.

Technologies

XA Distributed transactions (two phase commit)

TP monitors such as Tuxedo

DCOM, DCE, CORBA, RMI, EJB

Stateful Remote Objects

Many dimensions to consider beyond performance and availability

Performance (TPS/Host Latency, etc)
Availability: How many nines (time up/total time)?
Scalability: How much effort to scale?
Distributability: How much effort for multiple data centers?
Evolvability: Effort to extend and mutate
TCO: Hardware, licensing, dev or integration, operations and maintenance

Reconsider performance and availability relative to TCO!

Distributed Transactions: Atomic transactions across multiple transactional resources

Example: Customer changes primary address and hits customer and address db
Dark Side

Expensive. Reduces scalability of db server
Latency of commit is 5x more over normal transactions
Reduce throughput of application
If any resources are down, nothing can happen. Reduces availability

Alternate to XA

Be optimistic, commit what work you can
Do no harm: order commits such that if failure occurs you can live with inconsistent state
Compensate: undo previous commit or queue up rest of work for later
Design for failure: Minimize cross db foreign key refs, even denormalize
Tolerate dangling references and inconsistencies

Remote Procedure Calls: make a function call like it's local, but it's not

Example: calculate shipping charges on a customer order
Dark Side

Binary formats create dependencies
Evolving API forces client side rebuilds. Expect to evolve.
Service owners must run multiple versions of their software.
RPC tightly couples availability requirements.
Many fine grained requests have high latency over global distances

Alternative to RPC

Document passing paradigm

Self describing wire format (XML)
Evolution without affecting old clients possible
Good for asynchronous message passing

RPC model still possible

SOAP Problems

Large messages
Expensive to parse and build DOMs

Stateful Remote Objects (CORBA, EJB)

Problem being solved: supports ? for clients, clients can make many fine grained calls, keeps data model on server, complex data model not transferred
Dark Side

Mapping client session to stateful server is complex.
Servers must keep state for each client (reduces scalability)
Server failure fails a lot of clients (reduced availability)

Alternative: Stateless servers with persistence store

Servers handle each request independently
Use data in request to establish context
Return results to caller
Advantage: High performance, high availability, scales great
Disadvantage: Pushes state onto data store

Asynchronous messaging, Once-only delivery

Problem being solved: service developers don't worry about dupes. They can just do what the request wants. Reduces application logic and complexity of handling dupes.
Example: Customer 1-clicks on an item
Dark Side

Almost impossible to guarantee. In order to ? everything must be transactional

Double clicks happen all the time

Alternative to Only-Once

Idempotence (quality of something that has the same effect if used multiple times at once): dupes handled correctly with respect to application.
Advantages: simple, enables more scalability and availability. Simplifies clients.
Disadvantages: Requires services to check their db. Sometimes service has to build look aside cache.

In order delivery: service doesn't worry about temporal discontinuities

Example: Order adds A, adds B, adds C, deletes B, submits order
Dark Side

Very difficult for infrastructure to manage total ordering.
Tight coupling.
Can't deliver message until current message delivered.
Eliminates availability and scalability

Alternative: best effort delivery

Developers deal with out of order messaging. Requires event to have a time stamp or sequence id.
Advantages: high throughput, optimistic delivery policy (deliver events when you can), very high availability
Disadvantage: Application developers must deal with out of order messages

Stored Procedures

Easy for developers to write RPC type applications
DBAs can ensure db resources are used efficiently
Complex logic performed without moving data across the wire
Dark Side

Database resources are the most expensive
Creates scaling limitations
Low performance
Application now split between application server and database

Alternative

Use a database for what it's good for: relational queries and updates
Keep business logic on server

Centralized Database

ACID model is easy to program against, ensures consistency
Reads after write guaranteed to reflect write
Provides single synchronization point for all applications
Provides richest set of capabilities
Example: Customer information database
Dark Side

Doesn't scale
Doesn't lend to global distribution
Most labor intensive
Least available

Alternative: Lightweight operation datastores/caches (e.g. bdb)

Datastore distributed geographically
Updates propagate via asynchronous messaging
Read operations are done locally
Updates done locally then write back to central or peer
Disadvantages

Inconsistencies: Read after write not absolutely guaranteed
Partitions can cause multiple versions to exist on different peers
Requires distributed group management (DHT)

The nature of distributed systems

Nodes fail
Networks partition
Data centers go down

There is a tradeoff between availability and consistency

Via distribution and redundancy you gain availability, scalability and performance but lose consistency
Strive for eventual consistency

Embrace failure: build in availability
Accept inconsistency.

Apology oriented development

If you are a developer, deal with these things: Potential inconsistencies considering race conditions
Infrastructure cant hide
Model applications as event driven process. Include all info needed in each message. Prop, repl, cache. This provides high performance and high availability

There are talks of this nature fairly regularly at UW and Seattle University, I encourage you to go when you can. This was one of the most informational talks I have ever been to, and it was free.

Trac reminds me of Oracle

2007-06-22T04:03:00.001-07:00

I just finished getting a trac installation up on my site, you can find projects here. And, while I love Trac, the install process just reminded me of Oracle. It took several hours, had a bunch of dependencies, and to get it to work the way I wanted it required several more hours of customization. Granted, if you need an "enhanced wiki and issue tracking system" that has integration with subversion and works pretty well, Trac is tough to beat. But getting it running on a slightly older system was no easy task. If they want that tool to become more widely used the developers are going to have to fix the installation issues and create some type of an automated installer. Of course, I really have no right to complain considering I haven't written a single line of code for the project.

Martin Roesch on Snort 3.0 and Sourcefire

2007-06-20T10:27:00.000-07:00

Yesterday Sourcefire put on a two hour presentation at the EMP here in Seattle. With admission you got some swag including a calendar and a snort toy, admission to the Sci-Fi museum for the afternoon and an "ice cream social". Below are notes from the presentation, these are not my opinion. Overall I found the presentations pretty interesting with them covering the following topics:

Sourcefire & Snort; past, present & future
Demo of their RNA/ETM tools
Snort 3.0
Sourcefire 4.7

In particular, I wanted to hear Marty's thoughts on Snort 3.0 and where he is heading. Martin said that the 3.0 release would focus on the following areas:

Reduce Manual Tuning & Automate Configuration

"Tuning today is a failure"

We need dynamic defense for dynamic networks

Solve layer 3/4 evasion due to the IDS not being IP stack aware

Model the way an endpoint sees, model the IP stack

Normalize rules and configuration languages

Rules work well
Trivial to use for simple stuff

Ugly
Hard to do hard things
A bad rule can significantly impact performance

Snort is not a language project

LUA will be snort 3.0's next generation language processor
Snort 3.0 will include a command shell that will allow LUA commands to be executed

Take better advantage of hardware

We are getting more cores, not speed. Snort is single threaded, this is a problem.

Must multi-thread snort

Vendors are accelerating the wrong parts of Snort and have been for years

Need explicit locations for optimization.

Martin asserts that tuning, prioritization and evasion are the same problem. The root of this problem is a lack of knowledge of what is being defended. The solution is to impart knowledge about the operating environment directly into the engine. This allows for the engine to tune itself, automate anti-evasion and automate prioritization. Above is the snort 3.0 architecture as described/shown by Martin. I think of primary interest is the rearchitecture and threading. I will be surprised if Martin is able to release RNA as open source and integrate that into Snort. If that doesn't happen, it either means that the automation features won't make it into Snort or they won't work nearly as well as RNA.

Work in Progress: CopyBlog

2007-06-19T11:55:00.000-07:00

I used to have a Wordpress blog, and although there weren't many posts I wanted to import them into Blogger. I found a bunch of tools for importing from Blogger to Wordpress, but none that did the opposite. I found one tool that did what I was wanting (or says so), blogsync-java, but looking at the code it isn't very modular. Given that my technology tastes seem to change every other month, I really wanted a tool that would allow me to copy posts and comments between any two blog systems. Hence, CopyBlog. CopyBlog is a command line tool that allows you to copy posts and comments between any two blog systems, at least in theory. I spent some time yesterday and this morning writing some code to take Wordpress posts and comments and import them to Blogger. Although this currently just replicates the functionality of blogsync-java, the API is much more modular so you should be able to drop in a single class and you can immediately copy to and from that blog type. I should have a 0.1 version out the door this week if I can find some free time, and that would include full support for Blogger, Wordpress and LiveJournal. You can find source code up at http://mobocracy.net/code/CopyBlog

The Challenges of EC2

2007-06-17T14:10:00.001-07:00

I've recently been working on a project building out the development process, environment and tools for a startup client. This includes things like configuration management, release engineering, automated testing, version control, etc. In doing so, I've been wanting to create as part of this several images including:

A bootable and optionally installable build system (see this post)
A bootable development environment for each developer (preconfigured application server, version control data, preconfigured database, etc)
A bootable QA and Beta environment

I was hoping to use AWS for this, particularly EC2 for the development, QA and Beta environments. If you're not familiar with EC2, it is Amazon's Elastic Compute Cloud, and it allows you to essentially boot and run OS images in their cloud. You pay for hourly CPU usage and data transfer from the cloud and to the cloud. Let's compare a fully hosted, dedicated server solution from the planet with an EC2 image.

	The Planet	Amazon EC2
OS	CentOS	Linux w/2.6 Kernel
Data Transfer	1500GB	Unlimited
Disk Space	250GB	160GB
RAM	2048MB	1792MB
Cost	$147/month	$312/month

So, the above for bandwidth assumes that you use your full 1500MB a month (and the same in EC2) and that inbound traffic is only 25% of the total traffic, that number comes from my own experience and is probably overly generous. If you decreased the percentage of inbound traffic, the price increases. Also, it assumes 24/7 operation of a machine over a 30 day period. We also a amortized a $225 setup fee at the planet over a 12 month period. So, EC2 is more expensive for a front-end web server then a hosted environment however your downtime due to hardware related failures decreases to almost zero. You also have no setup fee and you can literally bring images up in minutes (so I'm told, more on that later). However, for an N-tier system, EC2 is a very inexpensive solution for your middleware application servers and backend servers since traffic between EC2 systems costs you nothing. Your cost for operating an EC2 image on a 24/7 basis and only doing inter-image traffic? $72.00/month. So, purely on a cost basis, EC2 seems like a platform for at least middleware and backend systems. Also from an downtime perspective, not having hardware to deal with should increase availability. In doing some research, I found the following troubling points. First, the internet is apparently full. This limited beta is currently full, a helpful error message tells me, and I will be notified when there is availability. Okay, that sucks. Second, EC2 doesn't natively have persistent storage for images. That is, if an image fails/aborts/shutsdown, any data stored on the local disk is lost to you. Apparently you can mount an S3 partition in an EC2 image, but S3 isn't really meant for random IO like you might have in a database for instance. I hope that EC2 opens up to new users in the near future, because it does solve many common problems (rapid horizontal scaling for the web tier, ephemeral environments for dev or qa). It is not however a silver bullet. Bandwidth to and from the cloud is expensive, and the lack of persistent storage makes many jobs impractical for the platform. If I ever get a chance to test, I'll put more information up.

SASAG Meeting for June

2007-06-15T18:24:00.000-07:00

Last night was the monthly SASAG (Seattle Area System Administrators Guild) and was the first I had been to. As I'm not a system administrator but a software engineer, I wasn't sure what I would gain from going but it turns out a lot. Last nights topic of discussion was, "Project Success: Science, Magic or Luck?" and was presented by Leeland Artra who is currently a PM at Qpass. Essentially, Leeland was suggesting applying software development techniques to system administration. For example, he suggested using test first development methodologies (I always knew them as TDD, Test Driven Design) to works towards a functional system. He also recommended using scrum to manage projects. My experience with systems engineers is that their programming experience is limited to systems programming, and things like functional and unit tests are foreign to them. If this isn't true, by all means let me know. So the suggestion for non-programmers to write programs to validate systems seemed counter intuitive to me. However, I liked the idea of using TDD to move forward on a project. I'm not sure why Leeland didn't suggest simply using Nagios or another monitoring system as your test platform. You can imagine going and writing all your monitors for Nagios which all start off red, or non-functional initially and as parts of the system come online and become functional monitors go to green. This seems like a much more straightforward, and intuitive way to use TDD for non-development projects. Regardless, the idea of applying agile methodologies to non-development projects is an interesting one and one I hadn't seriously considered before. I'm not sure how well it would apply to projects with serious capital expenditures such as hardware acquisitions, but the ideas should apply pretty well for any project. Leeland also showed off an interface he had developed for testing complex systems, which just seems unfair since I know it's not open source. On another note, I had my all time longest interview today. 7 hours and 8 people including the CTO, the hiring manager, two developers, one system programmer, one system engineer, the ops manager and the HR person. I really enjoy interviews that are challenging because you know you're going to be working with other good people. I mentally collapsed in my last technical interview though, so who knows what they thought. Updates ahead.

What do you want to be when you grow up?

2007-06-14T22:17:00.000-07:00

I've been doing software development, system engineering and architecture for almost 10 years. I've worked at large companies, small companies and tiny companies. And after leaving Mixxer in February with the intention of going back to school, for the first time in many years I felt a bit lost in terms of "What do I want to do now?". So I spent about 8 weeks traveling, saw family, did some vacationing, and got back feeling refreshed with what I had hoped would be a new perspective on things. I didn't have that though. What I had instead was the desire to go back to work, but no idea about what I wanted to be doing. So I started interviewing with everyone, 19 companies to be exact, doing everything from embedded C & C++ development to Ruby on Rails at companies ranging from fortune 100 to pre-funding startups. During the interview process I have kept busy consulting for small startups; helping with software development, architecture and direction. In doing some consulting it helped me figure out not what I want to do, but the characteristics by which I will be able to identify what I want to do. Characteristics of the right job include:

A company that believes they are improving the quality of life for its users
Coworkers who are really smart and passionate about the company mission
A startup
People who get the philosophy behind the technology they are using
The hacker ethos is prevalent
Decisions are made based on merit, not ego
A technical community based on meritocracy, not seniority

After determining that the above list would help me classify the right company for me, my list of companies dropped from 19 to 4. I'm wrapping up interviews with those 4 now, although the way I crumble in my in person interviews it could drop to zero pretty quickly :) In any case, being able to identify what it is exactly about working that you love is crucial to being able to find the right job. Sometimes you don't have a choice, you have responsibilities that drive you towards finding the first available and well paying position. I choose to wait. If you think you fit the environment I described above, and based on my resume I look like a good fit, send me an email.

Botnets and the Convex Hull

2007-06-14T22:05:00.000-07:00

Over the past few months I have worked on some computational geometry problems which required computing the Convex Hull for some set of points. I have been using it for some pattern recognition work and in doing so thought to myself, how could you map an IP address to a real vector space? And, if you could, is it possible to track an attacker or adversary? More importantly, can you estimate size or infer the location of a master in a botnet? Now, I realize that the location of a compromised host has no bearing on the location of the attacker. However, the latency between the compromised host and the attacker (or botnet master) does have a bearing on location. Likewise, there are a number of other useful metrics such as how recently the machine was compromised, the difference in times for two zombies to receive the same command, etc. Take one of these metrics, and assign it to each node you are aware of. Now use that metric as the distance to an arbitrary point P. Now compute the convex hull. Perform this same series of steps for each of the metrics you have chosen and overlay the convex hull for each metric. My assumption would be that your arbitrary point P could be identified in each one, and that may help identify a master. Also, it may help estimate the size of the botnet. The above writing is very hand wavy, I realize. However I'm curious if any work has been done until now to determine botnet topology via a similar mechanism. If anyone is aware, please let me know.

Reviving WADE

2007-06-14T08:56:00.000-07:00

I couple of years ago I started a small project called WADE. WADE stands for "Wireless ADvertising Engine", the goal of which was to enable coffee shops and other sites providing free wi-fi with the ability to earn money from advetising to help offset the cost of wifi. The technology would essentially allow sites to insert advertising in place of existing ads on a page, with ads that they get a payment for. More specifically, using something like the Adblock filter list, apache, mod_rewrite/mod_proxy and some dns magic to instead of removing ads, replace them with perhaps more relevant ads from local businesses. When I went to a friend at the EFF, he informed me that there may be an issue with Copyright law. As in, the content and layout of a page are protected under copyright law. I'm not sure if I'm liable for providing the software, the coffee shop is liable for using the software, or if it is a non-issue. In any case, I have received a few emails about WADE over the past couple of months and have thought I would revive the software and send it to a few friends who can use it. The question is, does something already exist and I should just point people there?