I, Hacker

So You Want To Be A Breaker, Pt. 1: Web Security

Sun, 17 Mar 2013 00:00:00 GMT

In my opinion, there's nothing more interesting and challenging than the security field. When you come toe to toe with a particularly strong piece of software and you just have to break it, the only thing I can liken it to is a good game of Go. Every time you make a move, your opponent -- the designers and developers, by proxy -- will respond in kind. You know there's a way to win and it will take every part of your brain firing on all cylinders to find the weak link in the chain.

I live for the moment when I strike the right point and the whole system comes tumbling down. You will too.

Preamble

The goal of this blog post is not to be comprehensive; in fact, the opposite is true. My goal is to teach you enough about security -- specifically software security -- that you can go forward and learn the other parts on your own. To ease in that learning, I've sprinkled links to resources throughout the post; Google is also very much your friend here, as there are a million resources out there for most of these topics.

Once you read the General section, you can jump to any of the others (including the other blog posts, as they come out) as you see fit as they're written independently.

This post is focusing on web security. Posts on native security and crypto will be coming soon.

General

Prerequisites

It's rather difficult to come up with a list of prereqs for security, simply due to the breadth of the field. However, the following are the must-haves:

Know how to program in one or more languages at a fair degree of competency (no one would expect you to edit a novel if you couldn't write the language yourself). If native security is your interest, at least one of these languages must be low-level; C is best.
Understand what a computer does and how it does it. You don't need to be able to put together a CPU with tin cans and string, but you should be able to explain:
- (Web) How HTML is parsed and how JS interacts with the page.
- (Web) How HTTP works.
  - In fact, write a proxy. You'll end up doing a lot of that. (This is actually important enough that I'll probably write a blog post about this one day.)
- (Native) How pointers work, how the stack is laid out, what the heap is, and (for bonus points) how syscalls work.

Where needed, I'll talk about more specific prerequisites.

Mindset

Before I can continue, I have to touch on the mindset of a breaker. Your goal is not to preserve user interactions or the intended flow of data through a program. Rather, your goal is to answer one simple question: Under what circumstances does this not do what it's intended to do?

Perhaps the best example of this is the cross-site scripting attack that's nearly everpresent in web apps; an attacker is able to output content to the page without proper filtering, allowing her to insert arbitrary HTML content and run scripts in the context of the page. The normal flow is preserved -- the data makes it from the user to the page -- but because that data contains HTML, the page is completely compromised.

The difference between intended and real behavior is what you're looking for in every case, whether that's a standard bug as described in the next sections, or a logic flaw unique to the problem domain you're testing.

Web Security

Must-have Tools

Below is a list of tools that you should have in your arsenal. Some of them are categories with more than one tool -- unless explicitly mentioned, they're roughly interchangable.

HTTP(S) proxy -- This you will use constantly, as it will allow you to analyze and modify data in-flight
- Burp Proxy -- Unless you have good reason not to, you should use this. It's the industry standard and by far the shallowest learning curve.
- mitmproxy
- Charles Proxy (UI is terrible -- only use it for AMF)
DirBuster -- Useful for finding 'hidden' files/directories
PadBuster -- Useful for exploiting padding overflow vulnerabilities
Python or Ruby -- For general automation of tasks

Cross-Site Scripting (XSS)

XSS vulnerabilities are among the most common you'll see in web applications. The concept is simple: data from an attacker is put into a page without proper filtering, allowing the attacker to execute code in the context of the page.

Reflected XSS is when data from an attacker (e.g. passed in the query string) is directly output on the page. This is the most common form and is frequently used in conjunction with CSRF attacks (see next section). They are protected against by default in many modern web frameworks.

Stored XSS is when data from an attacker is stored -- generally in the database -- and then output without filtering at a later point in time. These are less common than Reflected XSS vulnerabilities but have a higher impact normally, due to the ease with which they can be used to target other users. Like Reflected XSS, these are protected against by default in many modern web frameworks.

DOM-based XSS is when data is inserted by JavaScript on the browser side without filtering. These are fairly uncommon but are not protected against by most web frameworks; they are becoming more common relative to the other forms of XSS vulnerabilities due to the protections afforded those.

In all cases, they are protected against by filtering user data before output, but it's important to encode for the right case. For example, HTML encoding won't serve to protect anything if the data is being output inside of a script tag; in such cases, JS string output encoding is needed.

Cross-Site Request Forgery (CSRF)

CSRF vulnerabilities are simple but highly effective and common. The basic idea is that an attacker points the victim (automatically with a redirect, or manually with a simple link e.g. in an email) to a page on the target website. The link to that page contains request data to perform an action, e.g. make a bank transaction. To the server, this looks just like a legitimate request and it's performed as if the victim had done it themselves.

Protection against CSRF takes the form of random tokens that are put into the form bodies of pages and then submitted with each request; the server checks the token and ensures it's a match for the user's current random token in the session and rejects requests that don't match. Most modern web frameworks have built-in CSRF protection but it remains a very common vulnerability.

SQL Injection

SQL injection is, at this point, one of the most well-known and prevalent vulnerabilities present in web applications today. The premise is that data from an attacker will end up being put into a SQL query without proper escaping. This leads to the attacker being able to break out of a string (most commonly) and add their own code to the query.

Depending on the code and the database configuration, this can range from bad (blind injection allowing slow retrieval of data) to super critical (arbitrary code execution, file access, data destruction, etc). Most modern frameworks are protected from this due to their use of an ORM; everything else should use parameterized queries where possible. When a raw query with attacker data mixed in is the only route, proper escaping must be used.

Shell Injection

Shell injection is fairly uncommon but almost always critical when it happens. When user data is fed into a shell command without filtering, it's possible -- through the use of backticks, semicolons, ampersands, etc -- to execute other commands at the same time.

Code calling out to a shell command using user data should always prefer execution methods where an array of arguments is provided (this is naturally immune), and use the appropriate shell escaping functions when raw command strings are built.

Wikipedia

Directory Traversal

The most common forms of directory traversal are those where user data is directly used to construct a path to a file on the system. Without proper filtering in such cases, it's possible to insert a reference to the ".." directory entry, to walk up the directory tree. This can allow arbitrary reading/writing/deletion of files and is very frequently a critical vulnerability.

Wikipedia

Insufficient Authorization

Insufficient authorization refers to cases where access controls are not put on certain pieces of functionality or data. A good example is a properly access-controlled administration console which has links to admin-only functions that do not have proper controls.

This is often seen in conjunction with forced browsing, in the case of improper controls on data.

OWASP

Forced Browsing

Forced browsing -- also known as direct object reference -- is a class of vulnerabilities where an implementation detail is exposed to users in a modifiable way. Most commonly, this takes the form of an easily modifiable id parameter in requests, allowing an attacker to simply enumerate all possible identifiers to access/modify data.

OWASP

Others

The topics above are a great starting place and they're things that I see constantly in testing real-world applications. However, the list is less than comprehensive, so here's a list of a bunch of other topics that you should read up on:

Practice

I've recently been pointing everyone with an interest in websec to Natas. That wargame will let you use a good number of the exploitation techniques mentioned in this article, and some more advanced techniques that I didn't mention here. It also happens to be a whole lot of fun.

Other good resources include:

Going Further

Done practicing and want to dig your teeth into something real? Well, many companies are now running bug bounty programs where you can not only legally test their software, but get paid for the bugs you find. You can find a list of such programs here.

Happy hacking,
- Cody Brocious (Daeken)

Benjen v1

Sun, 17 Mar 2013 00:00:00 GMT

I've just released the first version of my new blog engine -- powering this site! -- for public consumption. It's a super simple Python script, allowing for a huge amount of customization without so much as touching the actual blog engine.

This was built in response to Posterous' announcement to shut down, which I had been using for this blog previously.

How it works

At its heart is 92 lines of Python code which reads in entries from your blog directory, splits them into index pages, and runs everything through Mako templates that you provide.

This all enables a lot of flexibility and power, despite that it can fit on a single sheet of paper.

Example

You can see this blog on Github here: daeken/benjen

Next Steps

I'm hoping to package this nicely as a git post-receive hook, so that you can just push your blog to a repository and have it automatically rebuilt on the server. (Update: This is now done and documented in the README.)

Update #2: Benjen v1.1 is out with RSS support.
Update #3: Benjen v1.1.1 is out with some additional magic to support proper ordering of posts within a given day.
Update #4: Benjen v1.2 is out, switching to Markdown metadata for title/date, simplifying the code and making it more extensible for the future.

If anyone has other ideas, shoot me an email and let me know: cody.brocious@gmail.com

Happy hacking,
- Cody Brocious (Daeken)

Distributed Balance

Thu, 21 Feb 2013 00:00:00 GMT

So, I've been quiet for long enough and it's time to discuss my plan. I've long said that we need a revolution, but that I didn't know the form in which it would come; I know that now. This is Distributed Balance.

Background

Despite their major flaws, we've seen the edges of the revolution for years -- forces pushing us to transparency: Wikileaks, Anonymous, etc. But the inherent problem is that we have a tiny portion of society helping out with these things; only a tiny portion has the skills to gain access to confidential documents or works in roles where that sort of access is possible. At the same time, we're in a position where we have so much data that it's impossible to catalogue and index it all.

In addition, centralized organizations such as Wikileaks are an inherent bottleneck when it comes to document releases, making it difficult for information to propagate effectively. This does, however, have the benefit of allowing them to redact names and the like from documents; whether this is desirable/acceptable to you may vary. Aside from that, it's very easy to strangle such an organization -- the US government has done so very effectively by cutting off their donations. This is not an acceptable risk.

What's Next

I'm designing a next-generation platform for distributing, indexing, and tagging large data dumps. Because the nature of such data may be objectionable to many, the platform is being designed with the assumption that it will be under constant attack legally, technologically, socially, and economically; it must defend against all of these points.

To this end, I'm going to discuss each of the design decisions and their purpose separately.

Federated

The system is designed to be inherently federated. By this, I mean that there will never be a central system containing data, receiving leaks, or hosting content. Rather, anyone will be able to set up a system to connect to the network, anyone will be able to add or tag documents, and anyone can publicly host the content.

This solves a few problems at once:

No centralized choke point for leaks. Since anyone can push content into the system, it makes it significantly harder to track who is leaking what.
The cost of distributing the content is spread out over a large number of hosts, rather than a single organization; this means that even if donations were sought out, it'd be far harder to shut down all of those payment endpoints. That said, because of the distributed nature, it's unlikely that any one host will end up with costs that are too high.
It becomes far easier for innovative thinkers to build new interfaces to data.

Web of Trust

Because of the federated nature of the system, some trust protocol needs to exist; without it, anyone could poison the well by distributing illegal and off-topic content. As such, the design calls for each system to be linked to others and given trust ratings. The exact meaning of these trust ratings is still up in the air, but the concept is straightforward:

Trust is ranked for each peer
- How much you trust what they're adding
- How much you trust what they're deleting
- How much you trust what they're tagging/annotating
Trust is transitive, e.g.:
- There are three peers, A B C, with A connected to B, B connected to C
- Peer A trusts peer B 50%
- Peer B trusts peer C 75%
- Because of the transitive trust, this means that A trusts C 37.5% (50% of 75%)

Details on this are still very much in the air.

Volunteer Mobilization

As mentioned earlier, part of the goal is to get volunteers involved. Right now there are a lot of people willing to sign petitions and all that, but what if we took that energy and got people actually organizing data?

In the first incarnation, volunteers will simply go to a node and receive a random document. They'll read through it and tag it appropriately, to the best of their abilities. One thought is that a short questionnaire might be the best way to go, with the tags being derived from those answers; the questionnaire would simply be written ahead of time by the leak uploaders, to make it situationally appropriate.

Getting volunteers to share documents and rope others into helping catagorize documents would be a big, big win for the platform; this is something that really hasn't been seen before. The people are there, they just need something to do, whether for 5 minutes or a week.

Search Engine

Journalists in particular have expressed the difficulties they have with dealing with the massive troves of information released by the likes of Wikileaks. As Google can tell you, organizing data into a form that people can drill down into to find what they want is very, very important. As such, everything in the system will be full-text searchable, as well as being able to filter by tags and things like that.

The technology for this is fairly straightforward, especially when you're talking about searching individual leak troves; there's a lot of data for humans, but not for computers.

Global Changelogs

To keep everything in check, we really need a system by which we can see all the changes across the system, at least for a few hops on the web of trust. This may not happen for v1, but this will be necessary as things scale up. Something akin to Wikipedia's vandalism bots would not go amiss here as well.

Problems

Below are the problems I see at the moment:

How do we keep one node from becoming the canonical reference for the network? This will drive up costs for that node, make them a large target, and add a single point of failure for trust.
How do we stop things like music, movies, and pornography from being stored on the network? The purpose is not general file-sharing, but specifically document storage and indexing.
How do nodes find each other? One idea that has been bounced around is using IRC to match up nodes.
How do we stop rogue nodes from flooding the network? One way is to use trust ratings to determine how big a change a given node can make in a certain period of time.

"Competition"

Services like Freenet enable people to publish and share documents at the moment. However, due to the nature of the system: documents are immutable, searching isn't possible without pulling down the contents of the archive, and annotations have to be external.

The same tradeoffs exist when, say, publishing a torrent full of leaks. A federated system like this is complex, but it allows for these fundamental issues to be dealt with.

Getting Involved

Simply put, things are very early stage. If you want to get involved, join #distributedbalance on irc.freenode.net. Design is underway, and bodies are needed.

There's also a Github repo, primarily for tracking issues and designing in the wiki.

Happy hacking,
- Cody Brocious (Daeken)

Holy crap, it's 2013

Wed, 09 Jan 2013 00:00:00 GMT

It's a new year, so it's time for a quick wrapup post.

Last Year

Obviously the big thing was my Black Hat USA talk on breaking hotel locks. The attention it's received has been insane; I never expected anyone outside the lock industry to care about it, but it's been featured in every major publication and even on national TV. Quite an experience.
I spent February-December working for Mozilla on the Boot2Gecko/Firefox OS project. Lots of interesting experiences and a great team.
In November I built QtzWeb, the first Quartz Composer implementation not by Apple, compiling it to JS WebGL. I released the first demo using it at Kindergarden, and dropped the source on Github recently.

This Year

As always, I have something new and interesting brewing, which I'll be showing off in the next few months.
I have an upcoming series of posts on getting into security, of which you can read a preview.
I started work at Accuvant LABS, where I'm a Senior Security Consultant.
I'm going to be splitting my time between the middle of nowhere in Connecticut, and Barcelona, starting in April.
QtzWeb -- the previously mentioned project -- is getting a web-based editor to cut Quartz Composer out of the picture entirely. This has a lot of potential for interactive 3d content on the web.

I'll be posting some more in-depth blog posts soon, but in the meantime have a happy 2013, and thanks to everyone for their support over the last year!

Happy Hacking,
- Cody Brocious (Daeken)

Responsible Disclosure Can Be Anything But

Thu, 06 Dec 2012 00:00:00 GMT

The last few weeks were full of mixed emotions for me. Reliable reports of hotel break-ins seemingly utilizing the vulnerability I disclosed at BlackHat are coming out, and I'm both horrified and vindicated.

As I explained in my paper, the decision to release this information in the way I did was not an easy one. I've been full of doubt ever since I first made the call, and while I was inclined to think I did the right thing, I really couldn't tell for sure. But now, my doubt is gone.

Ever since the first story broke about these issues, I've taken significant flak from the security community and others for my lack of adherence to Responsible Disclosure. Hopefully this post will show that that's not always the responsible approach.

Background

When I set out to reverse-engineer the Onity HT system, my goal wasn't to examine or undermine its security at all. Rather, the goal was to understand how it works and create a replacement for the Onity front desk system (primarily the encoder that makes the keycards). In the course of developing that, I found the vulnerabilities but didn't think much about them for a while.

In 2010, we (the startup I was running with friends at the time, UPM) decided to license the opening technology to a locksmithing company for law enforcement purposes. Quite honestly, I don't know what I can or can't say beyond that, due to contracts; at some point I'll be able to talk more openly about it.

Going Public

In 2012, we decided that it was time to make these vulnerabilities public, but we were unsure how.

Concerns

These were the characteristics of the vulnerability that were considered when weighing the release:

Severity and Impact

The vulnerability itself, in a vacuum, is quite severe. Any Onity HT lock can be opened in less than a second with a piece of hardware costing effectively nothing. The hardware can be built by someone with no special skills for only a few dollars and utilized with no real 'training'.

Combine this with the fact that there are 4-10 million of these locks (Onity's own estimates -- my guess is that this is locks in use versus locks shipped) in use protecting people for 20 years now, and you have the perfect storm.

Mitigation

Since the locks are not flashable, the only real way to fix them is to either prevent access to the jack (a non-solution) or replace the circuit boards in the locks. Either way, you're talking tens of millions of dollars to fix all the locks. Neither the individual hotels -- primarily independently owned with very low margins -- nor Onity can afford this.

Ease of Discovery

The major vulnerability (the memory read) is trivial to discover -- it's used as part of the normal operations of the lock. That combined with the time these locks have been on the market gave me a strong feeling that I was not the first to discover it. In fact, it's blindingly obvious to anyone looking at the way the lock's communication functions that this vulnerability would be present; how did Onity not know of it?

I've since heard from various people that someone inside of Onity may have built an opening device themselves at one point, but I can't confirm that, let alone that it operated on the same principles as mine.

Release Scenarios

Contact Onity

The standard 'Responsible Disclosure' approach would be to notify Onity and give them X months to deal with the issue before taking it public. While this is tried and true, there are several issues with this approach.

Onity, after 20 years and 4-10M locks, has a vested interest in this information not getting out, as it makes them look bad and costs them a significant amount of money. As such, it's likely that without public pressure -- which we've seen in the form of unrelenting press coverage -- they would have attempted to cover this up. Cases of security researchers being sued by vendors are well known in the industry and not uncommon.

Due to the difficulty in mitigating the issue, it's entirely possible that only a tiny fraction of hotels would've been fixed by the publication deadline, and without such a deadline applying pressure, there's no reason for Onity to continue to make strides to fix the issue.

Don't Release

This was a genuine option for a long while. While it's likely that it's been discovered and exploited long before I even looked at these locks, it was not a well-known attack.

However, I decided that the long-term benefits of this being fixed outweighed the problems certain to be faced in the short term while the flaws were being mitigated.

Full Public Disclosure

The last approach is to simply release all information to the public in the most visible way possible. This dramatically increases the odds that someone will use the attack for malicious purposes, which is why it was always a big concern for me.

However, by making it as visible as possible, it puts significant pressure on Onity and the hospitality industry as a whole to fix the issues and get hotel guests back to a safe position. At the end of the day, this seemed like the approach most likely to get a swift response to the problem.

Priorities

The key problem with Responsible Disclosure -- aside from the implication that any other means of disclosing vulnerabilities is irresponsible -- is that it puts the focus on security researchers' responsibility to vendors rather than customers. In most cases, the vendor of a vulnerable product is not the victim of attacks against that product, which makes this focus misguided in many cases.

In the case of the Onity vulnerabilities, the vendor (Onity) is not the victim, and neither are the hotel owners (Onity's customers). In this case, the hotel's customers are the true victims. While hotels or Onity could be liable in certain cases as a result of these vulnerabilities, hotel guests are the ones with their property and lives on the line.

Putting the focus on responsibility to the vendor, as nice a situation as it may be for vendors in many cases, can leave the true victims in a dangerous spot.

The focus for security researchers should always be on the customers, not the vendor; this is what Responsible Disclosure gets wrong. While it's true that in the majority of cases -- especially when dealing with software -- the quickest way to make customers safe is to work with the vendor to fix the issue, Responsible Disclosure advocates tend to ignore the edge cases in favor of a dogmatic adherence to this method.

When Is Responsible Disclosure Irresponsible?

This is a hard one, as Responsible Disclosure is usually the best approach and knowing when it falls short is tough. In general, I'd say that if the majority of the conditions below are met, Responsible Disclosure might not be the best way to minimize customer risk:

The product is widespread (note: in-house software almost always immediately merits Responsible Disclosure)
A fix for the vulnerability/vulnerabilities is costly or difficult to implement
The risk to customers is severe
- This must not simply be the vulnerability's rating, but the risk rating weighted with the impact to customers and the payoff for malicious actors
The likelihood of independent discovery is high
Replication of the attack requires little skill relative to the payoff of exploitation
The vendor is likely to have severe damage to their reputation or customer base as a result of the vulnerability becoming public knowledge

In cases where all or most of these are met, Responsible Disclosure may not be a responsible approach and may lead to customers remaining insecure and unsafe for years to come.

Response

The response to the Onity HT vulnerabilities has been bigger than I ever thought it would be. The press has been unrelenting for months -- it was even featured on the Today Show this morning (December 6th, 2012), more than 4 months since the original release.

Onity has been vague on how and when they're fixing the issue. In August they published a plan for mitigating the flaws, which I responded to; Forbes picked up on this and Onity subsequently removed all trace of this plan from their site. They are now stating that they are paying for/heavily subsidizing their fix, only after months of battering from the press.

This would never have happened without public pressure; hotel guests would have remained vulnerable for a long time, rather than a few months. When all is said and done, disclosing this fully and publicly will have lead to increased safety for hotel guests the world over.

Takeaway

Cases like this are never easy. In fact, it downright sucks to have to go through this with a vulnerability that could cause severe harm to customers. But at the end of the day, you have to decide one thing: is the customer more important than the vendor? In most cases, I'd say that they are.

The world is not black and white, and a dogmatic adherence to Responsible Disclosure makes us less secure and less safe as a whole. I urge all security researchers to think long and hard about how to disclose vulnerabilities, for the sake of everyone impacted, not just the vendor.

Happy Hacking,
- Cody Brocious (Daeken)

Confusing Crypto Blobs

Sat, 20 Oct 2012 00:00:00 GMT

Introduction

In 2011, CCBill ran a bug bounty program. While they didn't pay much per-bug, there were enough low-hanging fruit available to make it fairly lucrative; you could find a couple bugs in an hour and make some quick money. So for a month or two, I'd dedicate one or two nights a week to hunting down bugs and reporting them. As you can see from the rewards page that worked out pretty well. However, one of the most interesting bugs I found isn't on there. This is the story of that bug.

Interest

Early on in the bug bounty, one of my friends (Neal Poole, also testing CCBill at the time) pointed me in the direction of the 'Forgot Password' function and thought I might be able to find something there, specifically a padding oracle type attack. When I requested a password reset for my account, I'd receive a long link in my email, e.g.:

https://admin.ccbill.com/updatePassword2.cgi?enc=c8d2d3
d66ac13a865d64276623d8260352616e646f6d49566e31e063163e5
bd501fbca5d6f1c71998148a12edb637a2cf36492b646d8ad9ab811
9c6214432afe27c3c3030c2e6b43a2a8e9452923ad18298f9255578
d639c07afe41c1e82935d1cdba3709bec565e3eefb08ad27f33127a
d3daeb51d88d16209c706de526fc1c63f9e760e39da6fecbc9b5734
a8bb8a8348a07c2aac7645a799586a0f4cac7fa075862f4cac6dc17
137ad3840d43ea1d7a04d324cff6aab1917b4ac3779a98f3fdb8476
f25f350ae9bc4863fa1e11b4eacfba44b4dded55e6589568b576758
90635da75c13e22234

Now, when I see a blob like that, my first step is to take it from hex back to an ASCII representation. In the header of that blob, I saw RandomIV. A quick Googling of that string told me that it was generated by Perl's Crypt::CBC. Outside of an irrelevant old vulnerability report about the usefulness of the IVs (here), it looked solid. CBC mode, HMACed properly (seemingly), etc. So tampering with the blob wasn't much use.

I gave up after trying a couple different attack routes on it, as it seemed outside my cryptographical expertise.

Obsession

Of course, giving up and forgetting about something are two very different things, especially for me. Something about those blobs didn't seem right, and I started seeing them in other places. For instance, requesting reports about your account would lead you to a viewer that took another -- much larger -- encrypted blob, presumably containing either a query for the report data or the data itself.

So one night I found myself laying in bed and thinking about this issue, and whether I had missed something. I'd tried everything I could think of to break the blobs themselves, but there was some idea in the back of my mind.

Source confusion

They use the same blobs in many places. What if they're using the same key in each place? If they are, then a blob from one place will decrypt properly in another. If you could control what goes into the blob, you could compromise the other components. He who controls the source, controls the universe.

I immediately got up and took one of my report tokens and put it into the forgot password page. Up popped up a password reset dialog.

Of course, I had no control yet -- I had no idea what was in the blob, how it was constructed, or how any of the pages actually used the data in the blob -- but that proved to me that they used the same key in each place, making this very likely exploitable.

Gaining control

There was one place I could control the data going into the blob: report generation. While I couldn't ever tell just how much control I had over the final blob, I was able to put some arbitrary content into it, and that was good enough.

However, another interesting problem popped up soon after I discovered that: they were compressing the data going into the blob. If I put in a string of 'A's, I would only add a byte or two to the total size. That makes it harder to gain total control, but if you can put in any data you want, you could poison the compression enough to put it into a bad state -- one where effectively nothing compresses properly, and you end up with your own data completely in the clear.

The compression also could be used to figure out what else is in the blob other than your data. I didn't go down that rabbit hole, but this isn't unprecedented; in fact, it's the same way CRIME works.

Impact

Through trial and error, you can determine what impact your data has on the page where you're using the blob. This is an odd one in that depending on what's done with the bug, it could be anywhere from useless to super critical. In this case, it was determined by CCBill's security staff that it was worth two high bugs, or $1000. I never did find out what exactly could have been done with this bug, but I do know that you could compromise the forgot password form, and cause some query failures with the reporting facility.

Conclusion

If I were an attacker, this could've been a big, big win; in theory, this could've given me the keys to the castle. However, as a bounty participant it was worthless outside of scratching my own itch, as I took probably 5 or 6 hours to find that $1000 bug, rather than spending that same time to find ten $300 bugs. That said, it was definitely a more interesting result than the others.

Advice for developers/security folks

These days it's well known how to get encrypted blobs right (even if most people don't): use sane crypto in a good mode with a random IV, HMAC appropriately to prevent tampering, and don't leak information to enable oracles. However, even if your blobs are perfectly sound, using them in multiple places with the same key makes it possible for attackers to use a blob from page A on page B instead. If the attacker can control the data inside the blobs at all, you're likely to be compromised. The solution is to use different keys for each page that uses a different blob, or to put a 'blob type' flag at the beginning of the plaintext data.

In most cases this will not be a good way to attack a site -- there's generally something juicier and easier -- but they're really easy to miss and can be critical.

Advice for bug bounty admins

When determining pricing for bugs, make sure you aren't creating perverse incentives. A SQL injection vulnerability could cost you far more than a CSRF bug in a random user-facing page -- price accordingly. CCBill priced theirs at $300 for low, $400 for medium, and $500 for high bugs, though both Neal and I received one or two 2*high payouts. If you can find three or four low bugs ($900-1200) in the time it takes you to find a single high bug ($500), it absolutely makes sense to focus on those low-hanging fruit; the pricing is simply broken in that case. Not that I mind.

Happy Hacking,
- Cody Brocious (Daeken)

P.S.

Want me to find bugs like this in your own products? Check out my portfolio and contact me about my consulting services.

Onity's Plan To Mitigate Hotel Lock Hack

Fri, 17 Aug 2012 00:00:00 GMT

Onity responds

On August 13th, 2012, Onity released their plan to mitigate the vulnerabilities in the Onity HT hotel lock line that I released at BlackHat this year. My previous post on the subject is available at .

Their statement is available at and is reproduced below in case future edits are made:

August 13, 2012 - This is an update to Onity's previous communications regarding the hacking of certain models of Onity hotel locks. We want to assure you that Onity is working on providing you with a solution that will address any potential risks related to the alleged vulnerability of these locks.

Onity is going to implement a two tiered approach. The first approach will include providing a mechanical cap, free of charge, to our customers, who have the Onity HT series locks. This mechanical cap will be inserted into the portable programmer plug of the HT series locks. With the existing battery cover in place the mechanical cap will not be removable without partial disassembly of the lock. This will prevent a device emulating a portable programmer from hacking the lock. To further enhance the security of this fix, we will also supply a security TORX screw with each mechanical cap to further secure the battery cover in the lock. This solution is currently going to production, and should be ready for deployment starting the end of August. The second solution Onity will offer to our customers, if they choose to use this option, is to upgrade the firmware of the HT and ADVANCE series locks. The firmware is currently complete for the HT24 lock, and by early next week should be complete for the entire HT series of locks. By the end of August we should have the firmware complete for the ADVANCE lock as well. The deployment of this second solution, for HT series locks, will involve replacement of the control board in the lock. For locks that have upgradable control boards, there may be a nominal fee. Shipping, handling and labor costs to install these boards will be the responsibility of the property owner. For locks that do not have upgradable control boards, special pricing programs have been put in place to help reduce the impact to upgrade the older model locks. If you are interested in pursuing this solution, have additional questions or require further information, please contact Onity at 1-800-924-1442. Thank you again for your business and your trust in Onity over the past many years of our relationship.

July 25, 2012 - At the Black Hat conference on Tuesday July 24, a hacker presented alleged vulnerabilities of certain models of Onity hotel locks.

The hacker showed an open-source hardware device that he built, which mimics a portable programming device. He claimed that by using this device a plug can be inserted into the locks’ DC port, which may result in opening the lock.

Onity understands the hacking methods to be unreliable, and complex to implement. However to alleviate any concerns, we are developing a firmware upgrade for the affected lock-type. The upgrade will be made available after thorough testing to address any potential security concerns that you may have.

Onity places the highest priority on the safety and security provided by its products.

For additional questions or for further information, please contact our office. See below for your nearest location.

Thank you for your continued business!

Before going further, I want to say this: Onity should get credit for responding to these issues and taking steps to mitigate them. This situation is not a simple one, and taking steps to secure their hardware could not have been easy.

However, I feel I must throw in my two cents on the mitigation plan as it stands, as I do have serious concerns.

Physical security

Onity plans to provide a mechanical solution to the problem to all customers, in the form of a mechanical cover on the inside of the battery compartment. This cover will be mounted inside the battery compartment, covering the portable programmer port, and they plan to switch the screw on the battery panel with a TORX screw for added protection.

This -- as much as it is security-through-obscurity -- is actually a great temporary fix. Don't get me wrong, it will not take long at all to open the panel and use an opening device to pop the lock open, but it will raise the bar and make it more likely that the attacker is caught in the process.

However, I wonder how they're going to do this for the ADVANCE series locks; I don't have one handy, but as far as I can recall, the portable programmer port stands on its own and could not be covered up in this way -- they even say the HT locks are the only ones the cover is for. This seems to leave the ADVANCE lock owners (who are admittedly few) in the rain.

"Firmware update"

In addition to the port cover, Onity is working on what they are calling a "firmware update" for the locks. This is where things get hairy.

Not a firmware update

This is not really a security issue, but it is a credibility and honesty issue. I feel it's very deceptive to say to customers "we are preparing a firmware update" when you really mean that you're preparing a hardware update. They may be changing the firmware on the lock, but to make use of this, customers are required to replace the whole main circuit board.

This is equivalent to Apple telling customers "we're releasing a software solution for this issue" and then going on to say that they're doing it by replacing your laptop's motherboard.

Issue mitigation

I have not seen nor tested the updated locks in any way, shape, or form; what follows here is speculation based on my knowledge of their system and the vulnerabilities in question, and what they announced in their statement above.

At BlackHat, I announced two vulnerabilities: an arbitrary memory read and initial work into their flawed cryptography for key cards. The important thing to keep in mind is that neither of these sit in isolation; the arbitrary memory read happens as part of the protocol between the portable programmer and the lock, and the crypto is flawed between the encoder and the lock.

As such, I cannot imagine a fix for both of these issues which does not consist of replacing not only the lock circuit boards, but that of the portable programmer and the encoder.

If the protocol on the lock is changed in any substantial way -- as would be needed to fix the arbitrary memory read -- then the portable programmer would no longer communicate with the lock, causing the system as a whole to fail to function. Likewise, the cryptography cannot be changed on the lock side without also changing it on the encoder side.

I would absolutely love to be wrong about the lock protocol issue; if they can fix this at the lock level alone, and fix it well, then the impact on customers will be lower and the chances of the issue being fixed are higher. However, I find this highly doubtful. It seems far more likely to me that they have mitigated this issue at the lock level simply by shifting data around in memory or something along those lines, which would serve to break existing opening devices but not hold up to even the slightest scrutiny.

Responsibility

While it's great that Onity seems to be taking these issues seriously, the fact remains that such blatant vulnerabilities existed in their massively distributed product line for nearly a decade. As such, I believe that Onity has a greater responsibility to their customers than they are currently taking on.

Fix cost

Onity plans to distribute the mechanical caps free of charge to HT customers, but the cost of fixing the actual vulnerabilities will fall to the customers. Even if this were to cost only $5 per lock (between the hardware itself, shipping, and installation), at 4-10 million locks in the wild that means a cost of $20-50MM to the hotel industry as a whole; this will not be insignificant, given that the majority of hotels are small and independently owned and operated. Given that it won't be a low cost endeavour, it's not hard to imagine that many hotels will choose not to properly fix the issues, leaving customers in danger; this is on top of those that will simply not have heard of the fix, if Onity does not contact all of their customers directly -- I do not know if they plan to do this or not.

If such a significant issue were to exist in a car, customers would likely expect a complete recall at the expense of the manufacturer. I can't help but feel that Onity has the same responsibility to their customers, and to customers staying in hotels protected by Onity locks. Whether they have such a responsibility from a legal point of view or not, I can't say; but from an ethical point of view I believe they do.

Auditing

Releasing a fix for such a serious vulnerability without having the complete system (fix and all) audited by independent security professionals is merely asking for the issue to persist, or even new issues to come up. For the safety of hotel guests everywhere, Onity must have an audit performed; it's simply the only way to know that they aren't releasing another horribly vulnerable product onto the market.

This isn't a panacea -- things can and do slip by security audits every day -- but it's the best solution we have right now to at least catch the low hanging fruit, which I would very much consider the vulnerabilities announced to be.

Closing

The plan announced is not perfect, but it's absolutely a step in the right direction. I look forward to seeing it put into action and hope that the issues I raised in this post will be addressed.

As always, Happy Hacking,
- Cody Brocious (Daeken)

Blackhat paper

Tue, 24 Jul 2012 00:00:00 GMT

Well, my talk for Blackhat (My Arduino can beat up your hotel room lock) is over. Things could've gone better in terms of execution -- went through it too quickly and ended up using 30 minutes of my 60 minute slot. But people really enjoyed it and I spent a good hour or so answering questions.

Now it's time to release everything. There's still work to be done on the paper, but that will happen in time.

Paper: http://demoseen.com/bhpaper.html
Slides: http://demoseen.com/bhtalk2.pdf

I'll write more on all this in the near future, but now it's time for sleep and all that.

Happy hacking,
- Cody Brocious (Daeken)

Edit #1: I've created an IRC channel for the ongoing research in this stuff. It's #lockresearch on irc.freenode.net. Feel free to join if you want to keep track of the work, or participate hands on.

2012

Wed, 29 Feb 2012 00:00:00 GMT

Well, been a while since I've written anything here. Things have changed quite a bit lately: new year, new job, tons of random projects, etc. Figure it's time to write about it.

New Year

Holy shit, it's 2012. Last year was pretty insane:

Finally found an awesome girlfriend
Met a lot of amazing people
Traveled at least once a month
Released a few cool projects
- Kivlad -- Android decompiler I wrote at Matasano
- Windowpane and the associated article Superpacking JS Demos
Completely and utterly dominated the CCBill bug bounty program. That was freaking fun. List of rewards here.

But honestly, I feel like I missed the mark last year. I want to release a good bit of fun stuff this year.

New Job

As of the beginning of the month, I'm working for Mozilla Corporation as a hacker on Boot2Gecko. While I love the people at Matasano and will miss working with them, it was just time to do something new, so I'm initially working on graphics optimizations in Gecko. I'm also doing some fun stuff in terms of demos to exercise new features, and I'm now a member of Khronos and will be submitting various extension specs to WebGL.

While I'll be working out of the various offices from time to time, I'm primarily working remotely from Manhattan still, which is really nice. Finally turning my apartment into a Real Adult Home (TM), with the help of my girlfriend. There's even real art on the walls for once!

Projects

I'm working on a couple things for this year, and hopefully they don't fail quite so hard as a number of the projects I started last year (e.g. my completely stillborn hardware project, Gotham Eyewear).

QuestCompanions -- Real-time market for temp labor in MMORPGs. Launching publicly shortly.
Webgl-inspector -- An attempt to make WebGL canvases transparent, so you can investigate textures, shaders, etc and perform modifications in real time. Very early, but should be releasing a beta build in the next couple weeks.
Browser.js -- A complete browser written in Javascript. Note that this isn't intended to be a real browser, but rather one for experimentation and learning. Now that I'm working on the lower levels of Gecko, I really just want to learn how all the parts of the browser works, so I figure I'll write a simple browser to learn it over time. It should be an interesting experiment.
GenShaders -- Genetic algorithms applied to shader generation. Very much a work in progress, but I'm writing a series of articles on it for Displayhack, which will be linked/reposted here.

Goals

Above all else, my goal this year is to travel outside the country more. The girlfriend and I are planning on going to Greece and Turkey around July, which I'm looking forward to in particular.

This ought to be a good year; I'm hopeful that I'll learn a lot and have a lot of fun.

Happy hacking,
- Cody Brocious (Daeken)

Superpacking JS Demos

Wed, 31 Aug 2011 00:00:00 GMT

Background

The techniques I'm going to be describing here were created and/or implemented to pack my entry to the Mozilla Demoparty: Magister. Huge thanks to my friend Nicolás Alvarez for helping squeeze every last byte out of this.

Getting started

So you have a demo in JS. It's pretty. It's perfect. It's 3k in a 1k competition. Well, damn.

You start with the obvious and run it through a minifier and you shorten all your variable names to a single character, you get it to 2500 bytes. Great, that's progress, but you've still got 1476 bytes to go. You merge some functions together, fold a couple loops into each other, and soon you're at 2200 bytes. Long way to go.

Rule #1 of shrinking demos: removing a byte becomes more difficult every time you remove a byte. It starts off trivial and runs very quickly into a brick wall.

Quick wins

These are a couple useful techniques to trim the fat at this point.

var a = g.getAttribLocation(x, 'pos');
g.uniform1f(g.getUniformLocation(x, 'time'), t);
g.vertexAttribPointer(a, 2, g.FLOAT, 0, 0, 0);
g.enableVertexAttribArray(a);

What's wrong with this picture? Well, if we ignore whitespace (minification takes care of that): we're using var (pfft, correctness), we have a shader attribute with a 3-byte name and a uniform with a 4-byte name, we're using g.FLOAT (more on this in a moment), and we've got a lot of zeros.

g.vertexAttribPointer(
    a = g.getAttribLocation(x, 'p'), 
    2, 
    5126, 
    g.uniform1f(g.getUniformLocation(x, 't'), z), 
    0, 
    0
)

Much better! We killed var removing 4 bytes, we removed 2 bytes by inlining the assignment to a in the first use and killing the semicolon, we eliminated a zero (the return of uniform1f is treated as a zero by the WebGL API in this case) and a semicolon for a savings of 2 bytes, we changed our shader attributes/uniforms to have 1-byte names (saving 5 bytes), and we inlined the FLOAT constant for a savings of 3 bytes. That's 16 bytes removed from a block of 151 (after whitespace removal), or a reduction by 10.59%! Let's do it again.

Before we move on, I just want to make a note that WebGL constants really are just that. You can inline them and trust them not to change.

Abusing globals

Look at the latest code above. Count the instances of 'g.'. Four instances just in that tiny little snippet. If we could get rid of those in a small way, this could be a huge win, so let's do it.

What is the global namespace in JS? Normally, it comes from the window object. So when you call foo(), it's going to look in window if you haven't declared it locally. So why don't we shove our methods into the window object? Or even better, into top (in our case, they're identical)?

Right now the declaration of g is as such:

g = z.getContext('experimental-webgl')

But let's change it to:

for(k in g = z.getContext('experimental-webgl'))
    top[k] = g[k].bind && g[k].bind(g);

So we walk over every key in the WebGL context and put it into top, but only if it has the bind method and we can bind it to the WebGL context. Without that, it tries to treat the window as a WebGL context. The window doesn't like this.

At this point, we just cut all instances of 'g.' out of the picture. This was a win of around 30 bytes (counting the cost of the globalization code) in my demo's case, but we can go much, much further.

What's in a name?

Let's look at a list of all the WebGL methods I called in my demo:

attachShader
createProgram
linkProgram
createBuffer
bindBuffer
bufferData
viewport
createShader
shaderSource
compileShader
useProgram
getAttribLocation
getUniformLocation
uniform1f
uniform2f
vertexAttribPointer
enableVertexAttribArray
drawArrays

Holy moly, that's a lot of bytes! But these are defined in WebGL, how can we change this?

Welllll... let's look back at our globalization code. We get a name, k, and then bind g[k] into top[k]. But we control what k goes into top. Having a map of long name into short name would be expensive, but what about a regex? I'm going to spare you the gory details, but after a while of tinkering with it, I determined that the optimal code for this is:

t[k.slice(1, -5).replace(/[ntalruicoh]/ig, '')] = 
    t[k] = g[k].bind && g[k].bind(g);

In this case, t is top (because hey, 4 bytes!). We still assign the original name into t, but we also put the sliced and diced name into it. Why? Because this mangles two names we need to disambiguate: uniform1f and uniform2f. But that's fine.

So what do the names look like after this?

S
eeP
kP
eeB
dB
ffe
e
eeS
deS
mpeS
seP
eb
efm
f
f
eexbP
beVeexb
w

Night and day difference, as you can see. You'll obviously want to change the regex if you do this yourself, since your code will have different balances.

Other random wins

Arrays of digits tend to be pretty wasteful:

new Float32Array([0,0,2,0,0,2,2,0,2,2,0,2])

What about this instead?

new Float32Array('002002202202'.split(''))

The more digits you have, the bigger the win from this replacement is. Until we get to compression and everything changes.

Compression

Ok, we all know HTTP requests are trivially compressible, and this can improve load times and blah blah blah. Doesn't matter when it comes to demos: if the size on the wire is 1k and the size on disk is 3k, your demo is 3k.

In the past, people have compressed their demos into images. This is an easy way to get a good size reduction -- PNGs are just zlib'd data with a little header, basically. But these demos all had a PNG and an HTML JS file that would load the PNG, draw it to a canvas, pull the pixels out of the canvas as a string, and eval that string. How can you make this into a single file demo?

Introduction to PNGs

The PNG file has an 8-byte signature followed by a series of simple chunks. The chunk format is as follows:

4 byte length
4 byte chunk type
X byte chunk data
4 byte CRC

Most of these are pretty clear, except chunk type. Chunk type is a FourCC, e.g. 'IHDR', which is mostly generic, with some exceptions which I'll talk about shortly.

I'm not going to go into detail on the IHDR chunk (we can't really mess with it, but I will say I use greyscale for simplicity purposes), but here's what we start with.

8 byte signature
13 byte IHDR with 12 byte chunk header
X byte IDAT (covered below) with a 12 byte chunk header

The IDAT format is dead simple: 1 byte filtering method followed by your zlib deflated data.

Abusing PNGs

We start by defining our own chunk type (we're allowed to do that!) before the IDAT chunk.

4 byte length
4 byte "jawh" (Just Another WebGL Hacker (TM))
X byte bootstrap
4 byte CRC

What is the bootstrap? Well, it's what turns our PNG into code and runs it. Here's the one I use:

The 4968 here is really the size of the decompressed data in bytes times 4 -- there are 4 components to each pixel (red, green, blue, alpha) but we're only using grayscale, so we need to offset that. If you look carefully, you'll also notice that it walks backwards across the data, which should create a string in reverse, but I compensate for that when I compress the data, and reverse it before doing so. This saves a couple bytes. The image source is also important: rather than hard-coding the image filename and wasting space, it uses #, enabling it to treat itself as a PNG.

So what do we have now? We have a PNG containing some HTML. The browser first opens it as HTML (file extension is important here), then sniffs the MIME type and figures out it's a PNG when it gets loaded into the image tag. We have a self-extracting PNG. But we can do better.

Why specs don't matter

We're currently using a chunk type of "jawh". Clever and a nice little insider reference, but that's 4 wasted bytes! The chunk type comes right before the data, so why don't we make the chunk type into <img instead?

The spec says on the subject:

Four bits of the type code, namely bit 5 (value 32) of each byte, are used to convey chunk properties. This choice means that a human can read off the assigned properties according to whether each letter of the type code is uppercase (bit 5 is 0) or lowercase (bit 5 is 1). However, decoders should test the properties of an unknown chunk by numerically testing the specified bits; testing whether a character is uppercase or lowercase is inefficient, and even incorrect if a locale-specific case definition is used.

It then goes on to tell you that bit 5 of each of the bytes is:

First byte: 0 = critical, 1 = ancillary Second byte: 0 = public, 1 = private Third byte: reserved Fourth byte: 0 = unsafe to copy, 1 = safe to copy

Since we could call it <iMg or <imG or any other combination of uppercase and lowercase, clearly we only care about the impact of < on the first byte. But if we look at bit 5 of <, we see that it's already set high, so it's an ancillary chunk -- we're in the clear and just saved 4 bytes!

Wait, no, neither Chrome nor Firefox load the image anymore. What the hell? This is the point where you realize: the spec doesn't matter. No browser out there follows the PNG spec, whatsoever. Perfectly to-the-spec images won't work, and horribly broken images will work. So let's horribly break it. Note: Both Chrome and Firefox use libpng for image parsing, so you could go through the code and look for ways in which it's mishandling data and go down that path, but for my purposes I found that just experimentally changing things and seeing how they break was Good Enough (TM).

Breaking PNG for fun and profit

Messing with the chunk type is out due to browser incompatibility, so what other options do we have? Well, who cares about CRCs.

Set the CRC to c=#>, save, refresh. Hey, it works in Chrome and Firefox! 4 bytes saved.

Well, we can't push back towards the front due to the chunk type, and we can't push forwards towards the IDAT chunk because of the size. But how do browsers handle chunk size mismatches?

If we set the size of the IDAT atom to c=#>, that unpacks to a size of 1042496867, which is obviously more than the size of the IDAT chunk. We shift ) sr into the CRC and set the IDAT size to c=#>, save, refresh. All browsers will set size = min(reportedSize, endOfFile - startOfChunk)! That saves us another 4 bytes.

So sure, we couldn't get rid of jawh, but who cares? We just saved 8 bytes!

Compression caveats

Now you know how to make the PNG container small, but this says nothing about actually making your demo small. A few of the tips above are actually harmful in the context of compression. Why? Because repetitive data compresses really, really well. For instance, the Float32Array trick is nice when you're dealing with uncompressed data, but ends up eating a couple extra bytes after compression. Reusing function and variable names is also very, very important. The more repetition you can introduce, the better off you'll generally be.

Wrapping up

All in all, these tips (and general insanity) will let you build really small demos. My demo, Magister, went from around 3k down to just over 900 bytes when all was said and done. I ended up adding a message in the demo ("1k should be enough for anybody.") to get it up to exactly 1024 bytes, which was my goal. Feels good to have to work up rather than down.

I hope this helped shine some light on the subject.

Happy Hacking,
- Cody Brocious (Daeken)

Further Freedom Attacks

Sat, 27 Nov 2010 00:00:00 GMT

Please, redistribute this far and wide. Share it, repost it, print it out and hand it to people you see on the street.
Further Freedom Attacks by Cody Brocious is licensed under a Creative Commons Attribution 3.0 Unported License.

Today I've been vindicated; everything I've said for the last 5 years has come true. If you think I'm happy, you couldn't be further from the truth. As of this writing, the US government -- specifically Immigration and Customs Enforcement, a division of the Department of Homeland Security -- has seized 70 domain names from private citizens and corporations, not as part of legal proceedings, but on the basis of suspected illegality. The content of the sites, mostly owned by companies suspected of selling knockoff items, is inconsequential; even if their actions are illegal (which hasn't been proven in court, as they're not in court), they have the same right to publish their thoughts and feelings on the subject as anyone else does, due to the freedom of speech enabled by the first amendment.

Background

In 1998, President Bill Clinton signed into law a bill called the DMCA, or the Digital Millenium Copyright Act. The DMCA, among other things, was intended to: make it easier to deal with content infringing content; make the distribution of tools for circumventing copy protection illegal; make it illegal to link to sites that distribute infringing content. Now, this may seem innocuous, but there are a few very scary things here.

First and foremost, let's look at the last bit of that: due to the DMCA, it is illegal to tell someone where to get content that's thought to infringe copyright law. Note: this has nothing to do with actually giving someone illegal content, but just telling them where they can get it.

More relevant to myself is this: the anti-circumvention provisions of the DMCA make it illegal to tell someone how to use a piece of music in a way that the distributor doesn't want you to. If you purchased a song on iTunes in the past, you could only listen to it using iTunes or an iPod. People who made it possible to listen to your own purchased music on other devices were attacked on the grounds of the DMCA. This hits home for me, as one of my previous companies was shut down by Apple in this way, despite simply helping legitimate users.

Freedom of speech

There are two sides to the freedom of speech issue here: the DMCA's anti-circumvention provisions and their relation to source code freedom, and the direct freedom of written speech.

Source code

Source code, used to write programs, is a mechanism for communicating knowledge, much like normal speech is. For this reason, many people have made the suggestion that source code ought to be a protected form of speech. Courts in the US have been having this battle for ages, and they tend to lean heavily towards the argument that source code is more utility than artistic form, and thus does not get the same protection that speech does.

However, think about this from a slightly higher level: it's legally protected speech for me to tell you how to do X in steps the exact same way that the source code for doing X would. However, as soon as you translate that into something the computer can directly understand -- regardless of its ability to be understood by people as well -- it's no longer constitutionally protected.

If you take the stance that source code is -- or ought to be -- protected speech, then the use of the DMCA to stifle the distribution of code is a clear violation of the first amendment.

Written speech

The other big side to this is that, in seizing these domains, they have not just taken down the potentially illegal content, but the owners' own words. That is, they didn't just take down what they felt might be illegal, but everything written on these sites. This is such a ham-fisted, unconstitutional approach to the problem, I can't even begin to express my sadness. Whether or not you agree with what any one of these sites were doing, everyone has the right to speak their mind. With this action today, we've seen another in a long, sad trend towards restrictions on "bad" speech. This absolutely cannot be allowed to stand.

What can we do?

First and foremost, donate to the EFF. The EFF, or Electronic Frontier Foundation, is the foremost organization working to defend our digital rights. They've done more for the cause than anyone else out there, and they need our help.

Secondly, write to your elected officials, at the Local, State, and Federal levels. While only the Federal level is directly involved here, the more people who understand the issues and concerns, the better. This is absolutely crucial.

Lastly, spread the word. Share this article, news articles on the subject (a list of references are below), and any other relevant materials with people around you. Make sure that everyone you know knows about these issues.

We cannot take this sitting down. This must not be allowed to stand.

Stay free,
- Cody Brocious (Daeken)

References

http://torrentfreak.com/u-s-government-seizes-bittorrent-search-engine-domain-and-more-101126/ -- U.S. Government Seizes BitTorrent Search Engine Domain and More
http://www.osnews.com/story/24074/US_Government_Censors_70_Websites -- US Government Censors 70 Websites
https://writerep.house.gov/writerep/welcome.shtml -- Write Your Representative
http://eff.org/ -- Electronic Frontier Foundation
https://www.eff.org/issues/dmca -- EFF Issues: DMCA

New Job

Thu, 04 Nov 2010 00:00:00 GMT

After a lot of interviewing and a lot of deliberation, I decided to take a job. Starting Monday, I'm the new senior security analyst for Matasano Security. I'm really looking forward to working with these guys, and I hope to post some fun stuff on the Chargen soon!

Thanks to everyone who contacted me in response to my post about looking for work -- there were a lot of interesting options, and it took a while to narrow it down.

Happy hacking,
- Cody Brocious (Daeken)

Emokit status, JsDataFlowEditor, and looking for work

Sat, 25 Sep 2010 00:00:00 GMT

Before discussing new stuff, a little update on the progress of Emokit. Emokit is now completely able to parse the EEG data coming from the Consumer and Developer headsets, and thanks to skadge now contains proper Linux support, the beginnings of a C library, and quite a few other fun features. At this point, the only things left to reverse-engineer are the battery meter and contact qualities. Huge thanks to everyone who's supported us -- it's been invaluable!

Now for new stuff. I've just released JsDataFlowEditor, a Javascript library (based on Raphael and JQuery) for building and manipulating dataflow graphs. This can be tremendously useful for everything from image compositing to animations to audio work to visual programming. I'm really excited to see what people do with it. It's still fairly early stage, but it works well as it stands.

Super simple example use: . If you build something with it, shoot me an email at and let me know -- I'll link to it from the Github repo.

On another note, I've decided to get out of the full-time startup thing for a while. If you know of anyone looking for a developer/reverse-engineer in the NYC area, shoot them a link to please.

Happy hacking,
- Cody Brocious (Daeken)

Looking for work

Fri, 24 Sep 2010 00:00:00 GMT

Update: I've taken a job, but I'm going to leave this post here for historical reasons.

I've decided to put myself on the market for full-time work. I'm primarily looking for work in New York City, but other locations (or telecommuting) will be considered.

Most of you reading this will have a general idea of the work I do from other posts on my blog, but for those not in the know: I'm primarily a reverse-engineer and compiler developer, but I work on anything that happens to catch my interest. I'm always looking for a new challenge and new learning experiences.

My resume is available here, in PDF or in HTML format. It's very reversing-heavy as that's what the majority of my interesting commercial work is, but I've done everything from game development to kernel hacking to web development.

If I sound like the right person for the job, let's talk. Give me a call at 1.678.636.9323 or shoot me an email at .

Happy Hacking,
- Cody Brocious (Daeken)

The Hardware Hacker Manifesto

Tue, 21 Sep 2010 00:00:00 GMT

Edit: I've decided to release this post under a CC Attributed license. Share it, copy it, edit it, whatever you feel like, just attribute it back to me.
The Hardware Hacker Manifesto by Cody Brocious is licensed under a Creative Commons Attribution 3.0 Unported License.

This post is a long time in coming. For years we've been seeing an active fight against the right to utilize hardware the way the owner wishes to use it, but it wasn't until this week that it got personal, driving me to write this. Please, share this far and wide; hardware hacking is essential and we're losing ground to those who would love to see it done away with.

The Hardware Hacker Manifesto

My name is Cody and I'm a hardware hacker. It started at the age of five, taking apart a toy computer to figure out how it worked. I live for that thrill of discovery and rush of power that I feel when I figure out what makes something tick, then figure out how to bend it to my will. This has led to me hacking everything from game consoles to phones.

It used to be that this was what people did: if something was wrong with a device, it was acceptable to take it apart, figure out how it worked, and fix whatever was wrong with it. That's no longer the case; we're still there -- in growing numbers, to boot -- but what's changed is that it's no longer acceptable. As companies have made devices more and more locked down, making hardware hacking even more important than ever, there's a growing segment of the population that believes we're pirates. Who are we to modify these devices against the company's will?

It all comes down to one simple question: once you've purchased something, do you own it? While this may seem like a silly question, it's the entire crux of the argument for hardware hacking. If you believe that the purchaser owns the good, then they have the right to do with it what they want.

I exercise that right on a daily basis, whether with my jailbroken phone, my Wii running homebrew media player software, or -- now -- my hacked brain-computer interface. The last case is interesting, because it's the first time I've ever been called a pirate by a representative of the company producing the hardware I hacked:

Piracy is a vexed question but in its worst form it is still basically taking what someone has spent a lot of time and money on, and denying them some or all of the rewards for doing it. If the developer is being reasonable about it then it's tough to justify piracy. It costs a lot to get something developed and into the market, and next to nothing to copy or crack it. It discourages people from taking the risks in the first place, and we're all the poorer for the things that didn't get done because they would be too easy to steal.

In this case, I purchased a brain-computer interface outright, then proceeded to reverse-engineer it and release details of how to communicate with it. In the week since I released this, I've been called a selfish pirate more than I'd like to recall. All of this because I decided to exercise my right to use my hardware the way I want.

Why should we have to ask permission to use what we've spent our money on? Let's see an absurd extension of this logic: Why should Ford lose out on the rewards of building the car, when you don't go to an authorized service station to get your oil changed?

Let me make this crystal clear: once you sell me something, I will do whatever I want with it. Period. I'll take it apart, I'll patch it, I'll make it do things you never imagined, and I'll tell everyone who will listen exactly how to do the same. It's mine, and every device you've purchased is yours too; don't let anyone tell you otherwise.

I am a hardware hacker and this is my manifesto. We've always been here and we will always be here; you can fight to keep us out, but we'll fight even harder to get back in. I assure you we'll win.

Happy hacking,
- Cody Brocious (Daeken)

Emokit: Hacking the Emotiv EPOC Brain-Computer Interface

Mon, 13 Sep 2010 00:00:00 GMT

Update

Emokit fully supports the Emotiv headset and has functional Python and C interfaces. It's now maintained as part of OpenYou under the loving care of Kyle Machulis. The official repo is now at . Many thanks to everyone who helped out with the project.

Intro

I've been interested in the Emotiv EPOC headset for a while; a $300 14-sensor EEG. It's intended for gaming, but it's quite high quality. There's a research SDK available for $750, but it's Windows-only and totally proprietary. I decided to hack it, and open the consumer headset up to development. Thanks to donations I got some hardware in hand this weekend.

I'm happy to announce the Emokit project, an open source interface to the EPOC. The goal is to open it up to develompent and enable new products and research. For the first time, we have access to a high-quality EEG for $300 -- this is huge.

Code

The code is available on github on the daeken/Emokit repository. There's a Python library for interacting with the EPOC, as well as a renderer that will graph the sensor data.

Where things are right now

You can access raw EEG data from the Emotiv EPOC on Windows, Linux, and OS X from Python. While it's not known exactly which sensors are which in the data (read below for more info), it's close to being useful already. Word of warning: this project is less than 48 hours old (I just got hardware in my hands Saturday night) and has only been run by me on Windows due to a dead Linux box. It's very much alpha quality right now -- don't trust it.

How it happened

The first step was to figure out how exactly the PC communicates with it. This part was straightforward; it's a USB device with VID=21A1, PID=0001 (note: from walking through the device enum code in the EDK, it seems that PID=0002 might be the development headset, but that's totally unverified). It presents two HID interfaces, "EPOC BCI" and "Brain Waves". Reading data off the "Brain Waves" interface gives you reports of 32 bytes at a time; "EPOC BCI" I'm unsure about.

Next step was to read some data off the wire and figure out what's going on. I utilized the pywinusb.hid library for this. It was immediately apparent that it's encrypted, so figuring out what the crypto was became the top priority. This took a couple hours due to a few red herrings and failed approaches, but here's what it boiled down to:

Throw EmotivControlPanel.exe into IDA.
Throw EmotivControlPanel.exe into PeID and run the Krypto Analyzer plugin on it.
You'll see a Rijndael S-Box (used for AES encryption and key schedule initialization) come up from KAnal.
Using IDA, go to the S-Box address.
You'll see a single function that references the S-Box -- this is the key initialization code (not encryption, as I originally thought).
Use a debugger (I used the debugger built into IDA for simplicity's sake) and attach to the beginning of the key init function.
You'll see two arguments: a 16-byte key and an integer containing 16.

So that you don't have to do that yourself, here's the key: 31003554381037423100354838003750 or 1x005T8x107B1x005H8x007P. Given that, decrypting the data is trivial: it's simply 128-bit AES in ECB mode, block size of 16 bytes.

The first byte of each report is a counter that goes from 0-127 then to 233, then cycles back to 0. Once this was determined, I figured out the gyro data. To do that, I broke out pygame and wrote a simple app that drew a rectangle at the X and Y coords coming from two bytes of the records. I pretty quickly figured out that the X coord from the gyro is byte 29 and the Y coord is byte 30. The EPOC has some sort of logic in it to reset the gyro baseline levels, but I'm not sure on the details there; the baseline I'm seeing generally (not perfect) is roughly 102 for X and 204 for Y. This lets you get control from the gyro fairly easy.

That accounts for 3 bytes of the packet, but we have 14 sensors. If you assume that each sensor is represented by 2 bytes of data, that gives us 28 bytes for sensor data. 32 - 28 == 4, so what's the extra byte? Looking at byte 15, it's pretty clear that it's (almost) always zero -- the only time it's non-zero is the very first report from the device. I have absolutely no idea what this is.

From here, all we have is data from the sensors. Another quick script with pygame and boom, we have a graph renderer for this data.

However, here's where it gets tough. Figuring out which bytes correspond to which sensors is difficult, because effectively all the signal processing and filtering happens on the PC side, meaning it's not in this library yet. Figuring out the high bytes (which are less noisy and change less frequently) isn't terribly difficult, and I've identified a few of them, but there's a lot of work to be done still.

What needs to be done

Reversing-wise:

Determine which bytes correspond to which signals -- I'm sure someone more knowledgable than myself can do this no problem
Figure out how the sensor quality is transmitted -- according to some data on the research SDK, there's 4 bits per sensor that give you the signal quality (0=none, 1=very poor, 2=poor, 3=decent, 4=good, 5=very good)
Figure out how to read the battery meter

Emokit-wise:

Linux and OS X support haven't been tested at all, but they should be good to go
Build a C library for working with the EPOC
Build an acquisition module for OpenViBE

Get involved

Contact us

I've started the #emokit channel on Freenode and I'm idling there (nick=Daeken).

How you can help

I'm about to get started on an acquisition module for OpenViBE, but someone more knowledgable than myself could probably do this far more quickly. However, the reversing side of things -- particularly figuring out the sensor bytes -- would be much more useful.

Summary

I hope that the Emokit project will open new research that was never possible before, and I can't wait to see what people do with this. Let me know if you have any questions or comments.

Happy Hacking,
- Cody Brocious (Daeken)

Hacking the Belkin Network USB Hub

Sun, 05 Sep 2010 00:00:00 GMT

I've recently been hacking the Belkin Network USB Hub. It allows you to connect USB devices to the network and share them between your computers. The problem? It's Windows-only and totally undocumented. The solution? Reverse-engineering and a Python library, of course. Btw, if you pick one up, consider getting it from Amazon using my referral link:

You can find all of the source and some logs in the pybelkusb repository on Github. Note that this post is largely stream of consciousness, having been written mostly during the reverse-engineering process, in an effort to show how I actually went about it it, and as such there are certainly inaccuracies and unknowns. If anyone knows what some of these bits are or manages to find a case that breaks my assumptions, though, drop me a line.

I got it hooked up to my PC via a crossover cable, plugged in some devices (for testing I used an FTDI USB-RS485 cable, a generic flash drive, and a device with a Prolific PL2303 in it), and installed the Belkin software. Once I'd played around with it a bit -- it works remarkably well, for the record, although the default software is fairly buggy -- I dug in to start hacking it.

I started by breaking out Wireshark and refreshed the device list in the Belkin software, looking to see some basics: protocol (UDP), port number (19540), whether or not encryption/compression is used (unlikely or used sparingly -- plenty of plaintext). Now, I could've continued using Wireshark, but when I'm reverse-engineering I tend to write my own sniffer using the awesome SharpPcap wrapper for libpcap; this lets me annotate the logs, as well as quickly write parsers so I can validate my assumptions on the fly. For those of you playing along at home, it's in the Logger directory of the pybelkusb-reverse repo.

So the first thing to figure out was how the Belkin software discovered hubs on the network. This occurs via a broadcast UDP packet on port 19540:

0000 | 00 04 95 06 00 00 00 00  00 00 00 00 00 00 00 00  | ..?..... ........ 
0010 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0020 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0030 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0040 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0050 | 00 00 00 00 00 00 00 00                           | ........ 
0058

This, as far as I'm aware, is 100% constant. What the values are, I really can't say however. The hub responds by sending a UDP packet to the source of the broadcast:

0000 | 00 02 95 06 00 00 00 00  00 00 00 00 00 00 00 00  | ..?..... ........ 
0010 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0020 | 00 00 00 00 4e 65 74 77  6f 72 6b 20 55 53 42 20  | ....Netw ork.USB. 
0030 | 48 75 62 00 00 00 00 00  00 1c df cc 15 dc 00 00  | Hub..... ..??.?.. 
0040 | 00 00 c0 a8 89 2c 00 00  56 65 72 20 32 2e 32 2e  | ..???,.. Ver.2.2. 
0050 | 30 00 80 80 00 00 00 00  00 00 00 00 00 00 00 00  | 0.??.... ........ 
0060 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0070 | 00 00 00 00 42 4b 2d 4e  55 48 43 43 31 35 44 43  | ....BK-N UHCC15DC 
0080 | 00 00 00 00 31 2e 32 2e  30 00 00 00 00 00 00 00  | ....1.2. 0....... 
0090 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
00a0 | 00 00 00 00 00 00                                 | ......
00a6

I then worked through the data to figure out the following bits:

0x24 [16 bytes] -- Name (null term)
0x38 [6 bytes] -- MAC address
0x42 [4 bytes] -- IP address
0x48 [10 bytes] -- Some version string (null term)
0x74 [12 bytes] -- Serial number
0x84 [6 bytes] -- Some version string (null term)

Once a hub is discovered, the PC opens a connection (as much as you can with UDP, of course) on port 19540 and sends the following packet to enumerate devices:

0000 | 00 01 00 01 47 45 54 20  55 53 42 5f 44 45 56 49  | ....GET. USB_DEVI 
0010 | 43 45 5f 4c 49 53 54 00  00 00 00 00 00 00 00 00  | CE_LIST. ........ 
0020 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0030 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0040 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0050 | 00 00 00 00 00 00 02 00                           | ........ 
0058

The structure of that message is pretty clear, but the response less so. Here are a few responses with varying numbers of devices:

No devices:
0000 | 00 01 00 00 00 00 00 01  00 00 00 00 00 00 00 00  | ........ ........ 
0010 | 00 00                                             | ..
0012

1 device:
0000 | 00 01 00 00 00 00 00 39  01 64 65 76 30 31 00 55  | .......9 .dev01.U 
0010 | 53 42 32 2e 30 20 46 6c  61 73 68 20 44 69 73 6b  | SB2.0.Fl ash.Disk 
0020 | 00 00 00 00 00 80 81 12  21 32 34 48 31 34 00 00  | .....??. !24H14.. 
0030 | 00 00 00 00 00 00 00 00  00 00 00 00 08 00 00 00  | ........ ........ 
0040 | 00                                                | .
0041

2 devices:
0000 | 00 01 00 00 00 00 00 74  02 64 65 76 30 31 00 55  | .......t .dev01.U
0010 | 53 42 32 2e 30 20 46 6c  61 73 68 20 44 69 73 6b  | SB2.0.Fl ash.Disk
0020 | 00 00 00 00 00 80 81 12  21 32 34 48 31 34 00 00  | .....??. !24H14..
0030 | 00 00 00 00 00 00 00 00  00 00 00 00 08 00 00 00  | ........ ........
0040 | 00 64 65 76 30 32 00 46  54 44 49 20 55 53 42 2d  | .dev02.F TDI.USB-
0050 | 52 53 34 38 35 20 43 61  62 6c 65 00 00 00 00 00  | RS485.Ca ble.....
0060 | 80 81 04 03 60 01 46 32  33 00 00 00 00 00 00 00  | ??..`.F2 3.......
0070 | 00 00 00 00 00 00 00 ff  00 00 00 00              | .......? ....
007c

At first I was a bit thrown off by the varying packet size, until I realized that it actually just uses null-terminated strings. The structure from there is pretty straightforward, although certain bits are still unknown/assumed constant:

0x00 -- Header (00 01 00 00 00 00 00)
0x07 [1 byte] -- Packet size - 8
0x08 [1 byte] -- Number of devices
For each device
- Device node, null terminated string (devXX)
- Device name, null terminated string
- 4 byte IP of computer connected to device, or nulls if it's open
- 80 81
- 2 bytes Vendor ID
- 2 bytes Product ID
- 22 bytes of unknown data

After a short break to implement hub discovery and device enumeration (quite simple with Python), I started capturing data for device connection. The following is a request to connect to the flash drive, on device node dev01.

0000 | 00 01 00 01 45 58 45 43  20 53 43 41 4e 4e 45 52  | ....EXEC .SCANNER 
0010 | 20 43 4f 4e 4e 45 43 54  3a 01 06 00 1c df cc 15  | .CONNECT :....??. 
0020 | dc c0 a8 89 01 64 65 76  30 31 00 4c 54 00 00 00  | ????.dev 01.LT... 
0030 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0040 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0050 | 00 00 00 00 00 00 02 00  00 00 00 00 00 00 00 00  | ........ ........ 
0060 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0070 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0080 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0090 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
00a0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
00b0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
00c0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
00d0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
00e0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
00f0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0100 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0110 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0120 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0130 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0140 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0150 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0160 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0170 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0180 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0190 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
01a0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
01b0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
01c0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
01d0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
01e0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
01f0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00        | ........ ......
01fe

Holy nulls, batman! As you can see, most of this packet is unused, and it's almost entirely constant:

0x00 [4 bytes] -- 00 01 00 01
0x04 [21 bytes] -- EXEC SCANNER CONNECT:
0x19 [3 bytes] -- 01 06 (06 could be the length of the following field, a MAC address)
0x1A [6 bytes] -- MAC address of the hub
0x21 [4 bytes] -- IP address of the computer
0x25 [6 bytes] -- Device node string, null terminated
0x2B [3 bytes] -- LT followed by a null
0x2E [40 bytes] -- Nulls
0x56 [1 byte] -- 02
0x57 [423 bytes] -- Nulls

The hub responds as follows for success:

0000 | 00 01 00 00 00 00 00 03  4f 4b 00 76 30 31 00 55  | ........ OK.v01.U 
0010 | 53 42                                             | SB
0012

Or for failure:

0000 | 00 01 00 00 00 00 00 03  4e 47 00 76 30 31 00 55  | ........ NG.v01.U 
0010 | 53 42                                             | SB
0012

This one should be pretty self-explanatory, but regardless:

0x00 [4 bytes] -- 00 01 00 00
0x04 [4 bytes] -- 00 00 00 03 Potentially the size of the following string
0x08 [3 bytes] -- OK/NG followed by a null
0x0B [4 bytes] -- v01 followed by a null
0x0F [3 bytes] -- USB

Now it gets a bit tricky. You send the connection packet a second time, and the hub sends two UDP packets to port 19540 on the PC. Note: this is not where you sent the broadcasts from, it is a port you have to have hold open explicitly for device connections/disconnections. Also, a few things to be aware of if you're talking to the hub from Windows: The SXUPTP driver that handles the hub communication holds this port at all times, and if you connect to a device from, e.g. pybelkusb, and don't have the Belkin GUI up, the driver will crash and cause a kernel panic. This bit me in the ass repeatedly. Anyway, here's the packets it sends:

0000 | 29 08 02 05 c0 a8 89 2c  30 32 30 31 00 00 00 00  | )...???, 0201.... 
0010 | 01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0020 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0030 | 46 32 33 00 00 00 00 00  00 00 00 00 00 00 00 00  | F23..... ........ 
0040 | 12 01 02 00 00 00 00 08  04 03 60 01 06 00 01 02  | ........ ..`..... 
0050 | 03 01 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0060 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0070 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0080 | 09 02 00 20 01 01 00 80  96 09 04 00 00 02 ff ff  | .......? ?.....?? 
0090 | ff 02 07 05 81 02 00 40  00 07 05 02 02 00 40 00  | ?...?..@ ......@. 
00a0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
00b0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
00c0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
00d0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
00e0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
00f0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0100 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0110 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0120 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0130 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0140 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0150 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0160 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0170 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0180 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0190 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
01a0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
01b0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
01c0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
01d0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
01e0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
01f0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0200 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0210 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0220 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0230 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0240 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0250 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0260 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0270 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0280 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0290 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
02a0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
02b0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
02c0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
02d0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
02e0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
02f0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0300 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0310 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0320 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0330 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0340 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0350 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0360 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0370 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0380 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0390 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
03a0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
03b0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
03c0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
03d0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
03e0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
03f0 | 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  | ........ ........ 
0400

0000 | 00 03 00 00 00 00 00 00  30 32 30 31 00 00 00 00  | ........ 0201.... 
0010 | 01 00                                             | ..
0012

I never really figured out the meaning of either of these packets. The one thing I do know is that the fourth byte of the first packet there is some sort of device ID for the duration of the connection. You'll see it in use shortly.

Once these packets are receieved, you can make an actual connection to the device. You do this by connecting to TCP port 19540 on the hub. No initialization is required on the connection, and you can start sending commands. Let's see a bulk write:

TCP -> hub
0000 | 80 03 03 10 00 02 05 02  02 01 80 40 00 00 00 00  | ?....... ..?@.... 
0010 | 00 10 66 6f 6f 20 62 61  72 20 62 61 7a 20 68 61  | ..foo.ba r.baz.ha 
0020 | 78 21                                             | x!
0022

The data foo bar baz hax! was sent to endpoint 0x02 in this case, and the structure is as follows:

0x04 [2 bytes] -- Endpoint number (think this is two bytes, might be 1 at 0x05)
0x10 [2 bytes] -- Size of data
0x12 -- Data to write

Now, here I'll break from the narrative to explain the rest of the structure and a bit about the TCP "packets" themselves. Each TCP packet begins with two bytes indicating its type, e.g. 80 03 == bulk write. This is followed by a two-byte sequence, which starts at zero. Figuring out the rest of it took writing a parser that was capable of churning through a dozen logs I took over a period of a week. There are still unknown bits, but here's the best I managed to do:

0x04 [2 bytes] -- Unknown (I always send 00 03)
0x06 [1 byte] -- Device ID referred to above
0x07 [1 byte] -- Constant 02
0x08 [1 byte] -- Endpoint number
0x09 [1 byte] -- Constant 01
0x0A [2 bytes] -- Unknown (I always send 80 40)
0x0C [4 bytes] -- Constant 00 00 00 00
0x10 [2 bytes] -- Size of data
0x12 -- Data

The hub responds:

TCP <- hub
0000 | 80 03 03 10 00 00 00 00  00 00                    | ?....... ..
000a

Note that the type (80 03) and sequence number (03 01) match up with the packet sent to the device. The rest of the packet is null in every case I've seen. This may be used for errors in some case, but I've yet to actually see that.

I implemented this, then realized I needed control writes to actually initialize the device I intended to speak to, so that was up next. Here's an example control write:

TCP -> hub
0000 | 80 01 03 11 00 02 05 00  02 01 00 08 40 01 01 00  | ?....... ....@... 
0010 | 00 00 00 00 00 00                                 | ......
0016

Here you can see the type is 80 01. The packet structure is:

0x04 [2 bytes] -- Unknown (I always send 00 03)
0x06 [1 byte] -- Device ID referred to above
0x07 [1 byte] -- Null
0x08 [4 bytes] -- Constant 02 01 00 08
0x0C [1 byte] -- Request type
0x0D [1 byte] -- Request
0x0E [2 bytes] -- Value
0x10 [2 bytes] -- Index
0x12 [2 bytes] -- Null
0x14 [2 bytes] -- Size of data
0x16 -- Data

This one was particularly difficult to track down, as a lot of the values look the same and I had no idea which was which. It took installing a USB sniffer on my computer and capturing the actual device initialization and going over it field-by-field, matching the locations. I won't bore you with the details of that -- if you're interested, grab the demo of USBTrace and give it a shot; it's not hard.

The hub responds:

TCP <- hub
0000 | 80 01 03 11 00 00 00 00  00 00                    | ?....... ..
000a

Much like bulk writes, it's an empty packet with matching type/sequence.

With bulk writes and control writes out of the way, the matching two were dead simple, as almost everything is identical. A control read:

TCP -> hub
0000 | 00 01 00 05 00 02 05 00  02 01 00 08 80 06 03 02  | ........ ....?... 
0010 | 04 09 00 00 00 ff                                 | .....?
0016

The structure is identical to the control write message, with the exception of byte 0x5. In this case, I send 1 constantly, but I don't really think it matters. Of course, there's no data following this packet, since the size indicates the number of bytes to read.

Response:

TCP <- hub
0000 | 00 01 00 05 00 00 00 00  00 20 20 03 55 00 53 00  | ........ ....U.S. 
0010 | 42 00 2d 00 52 00 53 00  34 00 38 00 35 00 20 00  | B.-.R.S. 4.8.5... 
0020 | 43 00 61 00 62 00 6c 00  65 00                    | C.a.b.l. e.
002a

The structure here is dead simple, and you most likely could figure it out in under a minute, having made it this far:

0x04 [4 bytes] -- Nulls
0x08 [2 bytes] -- Size read
0x0A -- Data

Bulk reads are likewise simple as hell:

TCP -> hub
0000 | 00 03 03 6e 00 01 04 81  02 01 82 00 00 00 00 00  | ...n...? ..?..... 
0010 | 02 00                                             | ..
0012

Raise your hand if you're surprised that this is identical to the bulk write. None of you? Thought so. I won't insult your intelligence by breaking this down.

The response:

TCP <- hub
0000 | 00 03 03 6e 00 00 00 00  02 00 eb 3c 90 4d 53 57  | ...n.... ..?<?MSW 
0010 | 49 4e 34 2e 31 00 02 40  06 00 02 00 7e 00 00 f8  | IN4.1..@ ....~..? 
0020 | fd 00 3f 00 ff 00 00 00  00 00 00 20 3f 00 00 00  | ?.?.?... ....?... 
0030 | 29 22 96 1b 00 4e 4f 20  4e 41 4d 45 20 20 20 20  | )"?..NO. NAME.... 
0040 | 46 41 54 31 36 20 20 20  fa 33 c0 8e d0 bc 00 7c  | FAT16... ?3????.| 
0050 | 16 07 bb 78 00 36 c5 37  1e 56 16 53 bf 3e 7c b9  | ..?x.6?7 .V.S?>|? 
0060 | 0b 00 fc f3 a4 06 1f c6  45 fe 0f 8b 0e 18 7c 88  | ..???..? E?.?..|? 
0070 | 4d f9 89 47 02 c7 07 3e  7c fb cd 13 72 79 33 c0  | M??G.?.> |??.ry3? 
0080 | 39 06 13 7c 74 08 8b 0e  13 7c 89 0e 20 7c a0 10  | 9..|t.?. .|?..|?. 
0090 | 7c f7 26 16 7c 03 06 1c  7c 13 16 1e 7c 03 06 0e  | |?&.|... |...|... 
00a0 | 7c 83 d2 00 a3 50 7c 89  16 52 7c a3 49 7c 89 16  | |??.?P|? .R|?I|?. 
00b0 | 4b 7c b8 20 00 f7 26 11  7c 8b 1e 0b 7c 03 c3 48  | K|?..?&. |?..|.?H 
00c0 | f7 f3 01 06 49 7c 83 16  4b 7c 00 bb 00 05 8b 16  | ??..I|?. K|.?..?. 
00d0 | 52 7c a1 50 7c e8 92 00  72 1d b0 01 e8 ac 00 72  | R|?P|??. r.?.??.r 
00e0 | 16 8b fb b9 0b 00 be e6  7d f3 a6 75 0a 8d 7f 20  | .???..?? }??u.?.. 
00f0 | b9 0b 00 f3 a6 74 18 be  9e 7d e8 5f 00 33 c0 cd  | ?..??t.? ?}?_.3?? 
0100 | 16 5e 1f 8f 04 8f 44 02  cd 19 58 58 58 eb e8 8b  | .^.?.?D. ?.XXX??? 
0110 | 47 1a 48 48 8a 1e 0d 7c  32 ff f7 e3 03 06 49 7c  | G.HH?..| 2???..I| 
0120 | 13 16 4b 7c bb 00 07 b9  03 00 50 52 51 e8 3a 00  | ..K|?..? ..PRQ?:. 
0130 | 72 d8 b0 01 e8 54 00 59  5a 58 72 bb 05 01 00 83  | r??.?T.Y ZXr?...? 
0140 | d2 00 03 1e 0b 7c e2 e2  8a 2e 15 7c 8a 16 24 7c  | ?....|?? ?..|?.$| 
0150 | 8b 1e 49 7c a1 4b 7c ea  00 00 70 00 ac 0a c0 74  | ?.I|?K|? ..p.?.?t 
0160 | 29 b4 0e bb 07 00 cd 10  eb f2 3b 16 18 7c 73 19  | )?.?..?. ??;..|s. 
0170 | f7 36 18 7c fe c2 88 16  4f 7c 33 d2 f7 36 1a 7c  | ?6.|???. O|3??6.| 
0180 | 88 16 25 7c a3 4d 7c f8  c3 f9 c3 b4 02 8b 16 4d  | ?.%|?M|? ????.?.M 
0190 | 7c b1 06 d2 e6 0a 36 4f  7c 8b ca 86 e9 8a 16 24  | |?.??.6O |?????.$ 
01a0 | 7c 8a 36 25 7c cd 13 c3  0d 0a 4e 6f 6e 2d 53 79  | |?6%|?.? ..Non-Sy 
01b0 | 73 74 65 6d 20 64 69 73  6b 20 6f 72 20 64 69 73  | stem.dis k.or.dis 
01c0 | 6b 20 65 72 72 6f 72 0d  0a 52 65 70 6c 61 63 65  | k.error. .Replace 
01d0 | 20 61 6e 64 20 70 72 65  73 73 20 61 6e 79 20 6b  | .and.pre ss.any.k 
01e0 | 65 79 20 77 68 65 6e 20  72 65 61 64 79 0d 0a 00  | ey.when. ready... 
01f0 | 49 4f 20 20 20 20 20 20  53 59 53 4d 53 44 4f 53  | IO...... SYSMSDOS 
0200 | 20 20 20 53 59 53 00 00  55 aa                    | ...SYS.. U?
020a

Again, the structure screams out at this point.

And really, that's about it. I didn't cover disconnection because it's boring and dead simple (see the source in pybelkusb if you're really curious, or just log the disconnection yourself and look at it), and I haven't yet looked into error handling or timeouts whatsoever (they'll make their way into pybelkusb soon enough), but I think this gives you a good idea of how the protocol works and how easy it really is to reverse-engineer protocols like this.

Happy Hacking,
- Cody Brocious (Daeken)

Dotpack Beta 1

Fri, 09 Apr 2010 00:00:00 GMT

Edit: Due to a high troll rate on this post, I've temporarily disabled commenting. If you'd like to leave feedback, please email me at cody.brocious@gmail.com or drop a comment on the Hacker News thread .

I'm proud to announce the first beta of Dotpack, a packer for .NET executables. I've been working on this for the last week or so and finally have something fit for public eyes. It started out of my desire to build 64k demos on .NET and for that reason it's very small -- as of this writing, it's sitting at 5331 bytes overhead. It also achieves high compression ratios due to its use of LZMA; average size reduction for the files I've tested has been 60-80%.

At the moment it's fairly straightforward, not tampering with the original binary at all, but future versions will bring obfuscation and an array of code transformations to drop the filesize even further.

This version has several known issues I simply didn't have time to deal with: Silverlight packing is there (you can pass it a .xap and get one back), but it's very finicky and not particularly good so far. Packing binaries that use System.Reflection.Assembly.GetExecutingAssembly().Name will get back an empty string due to the way assemblies are loaded after unpacking, which can cause major issues. I'm going to fix these for the next release.

Other future features which will be coming, in no particular order:

Merging of assemblies. I was originally planning on releasing with ILmerge support as a stopgap until I finished my own prelinker, but licensing issues and a generally poor API made that less appealing. I'm working on a prelinker which, in addition to just merging assemblies, will perform dead code analysis to strip unused portions of the code away.
Obfuscation. Not only will this make it more difficult to analyze your binaries, but you'll get the benefit of less space being taken up by names. This can be quite substantial in a large binary.
Visual Studio integration. You'll be able to easily tie Dotpack into your Visual Studio workflow to produce packed binaries from your release builds.

Dotpack is freely distributable, but is under a non-commercial license. If you're using it in a commercial environment, even if you're not distributing your binaries, please purchase a commercial license. In addition to supporting Dotpack's development, you will also get builds ahead of the non-commercial users.

You can get the current beta build of Dotpack here: . To use it, simply run: dotpack.exe input.exe/.xap output.exe/.xap and you're off.

During the beta, a one-year single-user license is available at a discounted price of $250; once this goes gold, the price will go up to $500. Note that the year doesn't begin until the final release, so you're getting quite a deal. If you'd like to purchase a volume license, please contact me at . (Before the final release, I plan on getting a site up, but you know what they say about minimum viable products.)

Try it out, let me know how it works for you, and let me know if there are any features you'd like to see.

Happy Hacking,
- Cody Brocious (Daeken)

Demo a week, week 2: Armitage

Fri, 02 Apr 2010 00:00:00 GMT

Well, this demo is coming a bit late. After days of fighting with OpenGL and GLSL, I finally have a demo that's fairly presentable. It's still not great (by any means), but it's an improvement over last week's demo and that's the goal. Sadly, this week's demo has compatibility issues out the ass -- I suddenly understand why everyone goes D3D for the demoscene. My next demo will almost definitely be D3D10, due to familiarity and geometry shaders. We'll see.

Anyway, you can get the build at . If you give it a shot, let me know what video card you're using and whether it worked or not. I'll be releasing the source in the next day or two if anyone wants to play with it, and as always, the NFO is below.

Oh, and note: I couldn't make it less dependent on framerate this time around, but it's a ton faster than the last demo, so I'm not going to beat myself up over it. Next demo will be framerate-independent, I promise...

Happy Hacking,
- Cody Brocious (Daeken)

Armitage by Straylight
                    - April 2, 2010

Week 2 of my demo-a-week project is upon us.  I tend to think this is a much
better demo than the last one, even if it is short.  It has known compatibility
issues (older ATI cards, like the one in my desktop, seem to dislike my vertex
shaders...) and it'ss fairly short, but I like it a lot more than Waveride.

I was planning on making this one cross-platform, but after dealing with tons
of compatibility issues, I figure it's better for my mental health if I didn't.
The source will be made available in case anyone feels like doing something fun
with it.

Waveride used Miriel by Nightbeat as its music, on suggestion from a friend of 
mine.  Oddly enough, Nightbeat produced a song called Straylight -- I got the 
name from Neuromancer, but I can only take that as a good sign.  Anyway, enough
rambling.

Compatibility:
Tested on a GeForce 9400M, but should work with most modern cards.  Requires
SM3.0 (I believe) and rendering to a floating point framebuffer object.  If it
dies on start or you don't see the pretty spheres, one of those things is 
likely the issue.

Credits:
Code and art by Daeken
Music by Novoxide

Greets to:
ASD
SVatG
Conspiracy
Fairlight
Kewlers

Until next time,
- Daeken

Straytex: Online procedural texture studio

Sat, 27 Mar 2010 00:00:00 GMT

In the course of working on my next demo for my demo-a-week project, I'm doing a lot of procedural texture generation. The ability to quickly test modifications to my algorithms is crucial, so I decided to whip up a simple texture editor, Straytex (named after Straylight, my demogroup).

It likely only works on the latest Chrome dev channel build (edit: also works on Safari and Firefox, but I'm not sure which versions) and is ugly as sin (if anyone wants to help out with either of these issues, it'd be greatly appreciated!), but it gets the job done. At the moment, the interface is pretty simple: you can select the texture size, change the random seed, modify the source, and tag your changes. The tag feature is especially handy -- you often accidentally stumble onto cool effects, so the ability to tag these to go back to them later is insanely useful.

You can see it live here, and check the source out on Github. I hope you enjoy it.

Happy Hacking,
- Cody Brocious (Daeken)

Demo a week, week 1: Waveride

Thu, 25 Mar 2010 00:00:00 GMT

For a long time, I've sat on the sidelines of the demoscene, generally watching in awe as ASD, Farbrausch, Conspiracy, and others produce amazing works of art. I recently decided that I'd jump in and see what I can do, and to accomplish that I've decided to start releasing a demo a week. My goal is to get my feet wet and learn how to actually build something beautiful.

Today I'm releasing my very first real demo ever, Waveride, which I decided to hack together over the course of the last few hours. There's not much to it -- a wave plane consisting of spheres and droplets falling onto it with locations based on the music data -- but I don't think it's bad for a first shot. I'm planning on improving it over the coming weeks, before moving on to something new. At the moment, it's Windows-only, but pure OpenGL -- it should work fine in Wine. I'm going to port it to OS X and Linux before moving on.

Anyway, here's a link if you want to check it out in its current incarnation: . I've already received a good bit of information on what I need to fix (the nauseating rotation is absolutely first up on the list, I swear!), but please give me anything you can think of. This is an experiment for growth, and I can't grow if I don't know what I need to improve upon. The NFO is below.

Happy Hacking,
- Cody Brocious (Daeken)

Waveride by Straylight
                - March 25, 2010

Welp, not much to say.  I said I was going to create a demo a week until I had
a clue what I was doing.  Waveride is the first of these demos, and I'm pretty 
damn proud of it.  It's not perfect, and I'm not terribly happy with having 
used an off-the-shelf song for it, but eh.

Decided to sit down and write a demo, and I did it.  That was 1am, this is 8am.
Can't complain too much.

Developed and designed by Daeken.
Music is Miriel by Nightbeat (I believe this is cool -- if not, please, please 
let me know, and I'll change it at once.)

Greets to:
Nightbeat
ASD
SVatG
Lateralus
Kewlers

Until next time,
- Daeken

Daeken Discount Program

Fri, 26 Feb 2010 00:00:00 GMT

(Edit: The Daeken Discount Program is closed. While an interesting exercise, it resulted in only one sale, but it was of the 25% discount level so I can't complain too heavily. If nothing else, it was fun to try something.)

A long while back, I saw people selling shares in themselves. That is, you invest in the person and get a portion of their earnings and such, just like investing in a company. While I like the idea in theory, I can't see selling off a large (1 %) part of myself that I'll likely never get back. However, the idea's stuck with me. So while I was laying in bed last night, I came up with an idea.

The Daeken Discount Program

I'm launching an experiment. As of today, you can buy a lifetime discount on any of my products; rules and restrictions are below.

$50 gets you a 5% discount
$90 gets you a 10% discount
$200 gets you a 25% discount

I see this as a win-win. If no one buys into it, I'm out an hour worth of work and write this off as an interesting failure. If people do buy into it, I potentially lose some future profits, but the people who would do so are going to be more likely to buy from me anyway.

Why should anyone buy into this?

I think that people will see that I build cool things, and it's a small enough amount of money that everyone that buys into it will eventually at least break even.

How does it work?

The discount will be tied to the email you put in the Paypal purchase (although if you end up changing emails, I'll move it over to another, of course), and when you use it for purchases on anything of mine, you'll immediately receive the discount.

What does it apply to?

Any product purchase of mine, whether one-time or recurring. The obvious exception is if I don't have the right to do this, e.g. a product I build for someone else. However, all of my future companies will fall under this.

How long will you run this?

Probably no more so than a month or so. If it succeeds, I'd rather not have a million people with lifetime discounts on my products; if it fails, there's no point in leaving it up.

What have you built and what are you building?

Product-wise, I haven't built much; the biggest thing I've built was Alky, which allowed the conversion of Win32 binaries to run natively on OS X and Linux. It was a marginal success, although it ended up failing later for business reasons.

These days, I'm working on a few things:

Renraku: This will eventually be combined with
OpenBAMF/IREctive: Reverse-engineering platform and module store (My primary for-profit project right now)
Books
- The Emulator's Handbook: A book on building an emulator from start to finish. There's simply nothing there yet, which is a damn shame -- we need to get people involved here.
- So far unnamed: A book on reverse-engineering game protocols and emulating them.

In the future, I'd like to be developing and selling Eyetaps and other hardware around Renraku.

What if you never build anything interesting? / What if you never start another company?

In the (I hope, unlikely) event of one of these happening, I'm deeply sorry. I do my best to create things that people will want, but it's entirely possible that I'll fail. If I end up working a day-job for the rest of my life and someone buys into this program, I'll do whatever is in my power to give you the discount, even if I have to pay the difference myself.

Can I buy it multiple times?

No, sorry, but this does apply on top of any other discounts available.

What if you renege on the deal?

Then I'd hope that I'd be outted as a fraud and prevented from ever doing business again. Seriously, I wouldn't do business with someone who pulled something like that, and I'd sincerely hope that others wouldn't either.

Questions?

Post a comment or email me at cody.brocious@gmail.com. I'm curious to see whether or not this takes off; would be cool for it to do so.

Happy Hacking,
- Cody Brocious (Daeken)

Python Marshal Format

Sat, 20 Feb 2010 00:00:00 GMT

Earlier this week, I had to deal with some files in Python's marshal format (some .pyc files, specifically) in Ruby and discovered that the details of this format aren't documented. Since it's meant to be purely internal, the Python team has decided not to document it in any way.

The marshal format is used in .pyc files, lots of internal storage for random apps, etc. It's a shame that it's undocumented, as this means that there are, to my knowledge, no implementations for other languages. This also means that if you have a malicious marshal blob, you have to load it up with Python to play around with it; not a good idea.

Fortunately, you can read the source (Python/marshal.c) and figure out how it works pretty easily. However, to make it even easier, I've decided to write up some simple documentation on the format. I'll give types as int/uint where n is the number of bits. The object type indicates that this is a marshalled object.

Format

The marshal format in and of itself is very simple. It consists of a series of nested objects, represented by a type (uint8 -- a char, in fact) followed by some serialized data. All data is little-endian.

Note: I wrote all of this for the 2.x line. I don't know how much has changed in 3.x.

Types

Constants

These types contain no data and are simply representations of Python constants.

0 (TYPE_NULL) -- Used to null terminate dictionaries and to represent the serialization of a null object internally (not sure if this can happen or not).
N (TYPE_NONE) -- Represents the None object.
F (TYPE_FALSE) -- Represents the False object.
T (TYPE_TRUE) -- Represents the True object.
S (TYPE_STOPITER) -- Represents the StopIteration exception object.
. (TYPE_ELLIPSIS) -- Represents the Ellipsis object.

Numbers

i (TYPE_INT) -- Represents a int on a 32-bit machine. Stored as an int32.
I (TYPE_INT64) -- Represents a int on a 64-bit machine. Stored as an int64. When read on a 32-bit machine, this may automatically become a long (if it's above 2**31).
f (TYPE_FLOAT) -- Represents a float in the old (< 1) marshal format. Stored as a string with a uint8 before it indicating the size.
g (TYPE_BINARY_FLOAT) -- Represents a float in the new marshal format. Stored as a float64. (Thanks to Trevor Blackwell for noting that these are not float32 (along with TYPE_BINARY_COMPLEX).)
x (TYPE_COMPLEX) -- Represents a complex in the old (< 1) marshal format. Contains the real and imaginary components stored like TYPE_FLOAT; that is, as strings.
y (TYPE_BINARY_COMPLEX) -- Represents a complex in the new marshal format. Stored as two float64s representing the real and imaginary components.
l (TYPE_LONG) -- Represents a long. Haven't yet figured out how this works; I'll update shortly with that.

Strings

s (TYPE_STRING) -- Represents a str. Stored as a int32 representing the size, followed by that many bytes.
t (TYPE_INTERNED) -- Represents a str. Identical to TYPE_STRING, with the exception that it's added to an "interned" list as well.
R (TYPE_STRINGREF) -- Represents a str. Stored as a int32 reference into the interned list mentioned above. Note that this is zero-indexed.
u (TYPE_UNICODE) -- Represents a unicode. Stored as a int32 representing the size, followed by that many bytes. This is always UTF-8.

Collections

( (TYPE_TUPLE) -- Represents a tuple. Stored as a int32 followed by that many objects, which are marshalled as well.
[ (TYPE_LIST) -- Represents a list. Stored identically to TYPE_TUPLE.
{ (TYPE_DICT) -- Represents a dict. Stored as a series of marshalled key-value pairs. At the end of the dict, you'll have a "key" that consists of a TYPE_NULL; there's no value following it.
> (TYPE_FROZENSET) -- Represents a frozenset. Stored identically to TYPE_TUPLE.

Code objects

Code objects (like that in a .pyc file, or in the func_code property of a function) use the c (TYPE_CODE) type flag. Even in the case of the top level (as in a .pyc), they represent a function.

They consist of the following fields:

argcount (int32) -- Number of arguments.
nlocals (int32) -- Number of local variables.
stacksize (int32) -- Max stack depth used.
flags (int32) -- Flags for the function.
- 0x04 -- Has *args.
- 0x08 -- Has **kwargs.
- 0x20 -- Generator.
- This list is not all encompassing; certain __future__ declarations will set their own flags.
code (object) -- String representation of the bytecode.
consts (object) -- Tuple of constants used.
names (object) -- Tuple of names.
varnames (object) -- Tuple of variable names (this includes arguments and locals).
freevars (object) -- Tuple of "free" variables. (Can anyone clarify this a bit?)
cellvars (object) -- Tuple of variables used in nested functions.
filename (object) -- String containing the original filename this code object was generated from.
name (object) -- Name of the function. If it's the top level code object in a .pyc, this will be <module>.
firstlineno (int32) -- First line number of the code this code object was generated from.
lnotab (object) -- String mapping bytecode offsets to line numbers. Haven't delved into the details here.

RMarshal

I've implemented support for unmarshalling objects as well as reading .pycs in a Ruby gem called RMarshal. You can get it from Gemcutter or from Github.

Closing

Hopefully this will be of use to someone. One potential use is in studying malicious marshalled data; the Python guys strongly recommend against unmarshalling untrusted data, but we all know how well such notices are regarded. In addition, it may help you manipulate Python bytecode from non-Python languages.

Drop me a line if you do anything cool with it.

Happy Hacking,
- Cody Brocious (Daeken)

Renraku OS: Road to v1

Sat, 23 Jan 2010 00:00:00 GMT

Renraku has seen a renewed surge of energy in the past few weeks, and has made big strides. Here's what's going on.

Where we are

Renraku now runs in hosted mode on top of MS.NET/Mono. This enables a much, much quicker development/debugging cycle, and allows us to start working on the GUI while the low level details are still being worked out. There's still work to be done here, but it's a good start. We've also made the leap to Rake from NAnt; dealing with the bulky, inflexible build files just got to be too much. A few hours of hacking and Rake now fits our purposes perfectly.

After hosted mode came up, we made a push to get video/mouse services built on top of SDL, enabling us to start developing the GUI. You can see an initial GUI coming up and displaying our logo here:

In addition, Capsules (check out this blog post if you don't know what this is: Renraku OS: Networking, Hosted Mode, Moving Forward) are in progress and are taking shape; the implementation is still rapidly changing, but we seem to have a solid idea of what they're supposed to look like and how they interact with each other.

Where we're going

The project is now moving in parallel on two efforts: Userspace (GUI, applications, storage), and native (kernel, compiler, drivers). This will all converge over the next 6 months.

We're aiming to make two releases between now and July 4th, which will serve to get the code in the hands of developers leading up to v1.0. We now have a solid roadmap for Renraku v1.0, which we aim to release on July 4th, the one-year aniversary of the project. The roadmap is as follows:

Goal

The project goal is that v1.0 should go out on July 4, 2010. This marks the first anniversary of the project and will be the first major release.

What should it do?

You should be able to start Renraku in hosted or native mode (IA32 native only) and bring up a usable GUI. You should be able to browse files, start applications, edit configuration, and run UI tests. Depending on time constraints, we may or may not have additional applications (and a game?)

What needs to be done?

Compiler:

Exceptions
Generics
Array bounds checking
Emit proper class data for reflection

Gui:

Basic toolkit for controls
Image rendering
Font rendering
Vector support?

Graphical shell:

Window management
Desktop icons
Some sort of system tray/menu

Kernel:

Tracebacks on exceptions
Garbage collection
Memory management
Storage
- Low level storage service (hard drive access)
- FAT32 filesystem service
Networking
- TCP
- IP and UDP supporting fragmenting
- More robust DHCP
- DNS
- Routing
- Transparent object remoting
Video driver on IA32 (better than VGA)

Services:

Capsule implementation
Service documentation

BCL:

Migrate to Mono BCL
Implement more of the BCL, if migrating to the Mono BCL isn't possible/practical

Applications:

GUI console
File browser
Basic text editor
Image viewer
Some basic GUI game?

Build System:

Allow portions of the code to be tagged as platform-specific, rather than the large file lists in the Rakefile
Integrate building a LiveCD with a complete Renraku system.
Build Renraku installer (XXX: May get pushed back to v2)

How you can get involved

At this point, we need a few things: Application developers, service developers, kernel developers, and compiler developers. Want to write the first Doom port to run on a managed OS, a Twitter client, a completely new web browser, or a sane networking stack? Here's your chance.

No matter how you want to help out, or if you just want to ask some questions, join us at #renraku on Freenode IRC.

Points of contact:

Daeken (Cody Brocious, ) -- Kernel lead; Kernel, services.
ircubic (Daniel Bruce, ) -- Userspace lead; GUI, applications, storage. Documentation lead. Build system.
nrr (Nathaniel Reindl, ) -- BCL lead. Memory management lead.
dublindan (Daniel Kersten, ) -- Capsule lead.

Happy Hacking,
- Cody Brocious (Daeken)

Alky Postmortem

Sun, 27 Dec 2009 00:00:00 GMT

A lot of people have asked me what happened to the Alky project. The short answer is that we made a lot of bad business moves, but that answer glances over a lot of the fun little details. Having gained considerable knowledge from other stories of failed startups, I figure I'll throw one of my own into the ring.

History

The Alky project's history can be split into a few phases:

Conception

Alky began as an experiment to see how easily I could convert Windows PE files to run natively on Mac OS X (x86). If it were to work, it may make it possible for me to convert Windows games to run natively on OS X, which was my primary focus. I started by writing a converter that ripped the segments out of the original file and throw them into a Mach-O binary, then linking it against 'libalky'.

LibAlky was a reimplementation of the Win32 API. At first, I tested with files that just called a few simple things, like user32:MessageBoxA, and it worked spectacularly. It was at this point that I realized the basic concept was not only sound, it made a whole lot of sense.

Actual project creation

Once the initial prototype worked, it was time to get people interested. I went to Michael Robertson (who was my employer at the time) and gave him a rundown. He offered to buy the domain, host the project, and get some resources behind it, primarily PR-wise. Within a few days, the project started actually feeling real. We got the site up, wrote some copy to explain what we were doing, and then put it out on Slashdot.

Unsurprisingly, we received three types of responses:

This is impossible, it simply can't work from a technical point of view. (This was especially hilarious coming from a Transitive engineer, considering that what they did is considerably more complicated.)
This is possible, but it won't happen due to the immense amount of work involved. (Half right -- more on this later.)
Wine sucks, I hope you guys can do it better. (We couldn't -- more on this later.)

But more importantly than anything else, we got some developers involved. However, they ended up being driven away.

Mismanagement of the open source project

Alky was the first open source project I'd ever managed that consisted of more than myself and a few friends, and as a result it was mismanaged at every possible turn. It was unclear what people should've been working on, no effort was made to document the project, and no real effort was made to include the developers that were so interested in working on the project.

This was compounded by a rewrite and redesign, which I decided (foolishly) to take on entirely by myself. Some of the design was done as a group, but none of it ever left my hard drive, so work stalled on the existing codebase and the project started to wither.

Shortly thereafter, Falling Leaf Systems was incorporated to back the development of Alky. This further increased the rift between the open source developers and the "core" group (consisting of myself and one of the cofounders of the company). Originally, we planned to dual-license the codebase, but as we got more into discussions of the goals of the business, it became clear that closing the source was the right move. However, we couldn't have picked a poorer way to do it.

Rather than be upfront about the move to closing the source, we simply killed the IRC channel and took down the site. The open source developers were left wondering what happened, while we moved on the rewrite.

Prey and the Sapling Program

Alky was completely rewritten with the new design, and work quickly moved forward on getting the first game running. We released a converter for the demo of Prey (Mac OS X only at first), as part of our new Sapling Program. The Sapling Program was created as a way to get some upfront money for the company, so we could get needed hardware, go to the GDC (which was a horrendous waste of money, for the record), etc. We sold memberships for $50, which gained access to the Prey converter and all future converters. Of course, after we finished Prey for Linux, there were no more converters.

Loss of focus

After Prey was done, we'd planned on implementing DirectX 9 with hopes of running Oblivion, but we lost sight of this goal. Instead, we decided to go after DirectX 10. Along with this shift in focus came an even bigger one: we were no longer targeting Mac OS X and Linux primarily, but Windows XP. We saw that Vista was tanking, and DirectX 10 was only available there, so we decided that we only had a limited window to make money off of that.

Shortly after we made the change, we released a library to allow a few of the DX10 SDK demos to run on Windows XP via OpenGL, albeit with serious missing features (e.g. no shaders). It got some attention, but few people were able to actually get it working. After this was out, I started work on reverse-engineering the shader format and writing a recompiler that would allow Direct3D 10 shaders to work with OpenGL, and even more importantly, without DX10-class hardware. Geometry shaders were planned to run on the CPU if the hardware wasn't available to handle it, and work progressed quickly.

Alky for Applications

We discovered a project known as VAIO to allow Vista applications (non-DX10) to run on Windows XP, and after some talks with the developers, they (or more specifically, one of them -- we'll call him Joe) were sucked into Falling Leaf. We rebranded VAIO and it was released as Alky for Applications. After this, Joe was tasked with making Halo 2 and Shadowrun -- Vista-exclusive but non-DX10 games -- run on Windows XP. We were so confident in our ability to do this, we set up an Amazon referral system and made it such that anyone who purchased either game through us would get a copy of the converter for free.

At this point, I was working heavily on DX10 stuff, and was under tight deadlines to show things off to a company that was interested in our technology, but the clock was ticking. About a week before the planned release of the Halo 2 and Shadowrun compatibility system, Joe came to us and told us that he'd not been able to get anything working, and had very little to show for the time spent. In retrospect, it was my fault for not checking up on him (my job as his manager), but at that point it just made me realize there was no way it was going to be done in time.

I picked it up -- dropping DX10 in the process -- and attempted, feebly, to get it done. Of course, I picked the most difficult way to approach it; reverse-engineering the Windows application compatibility system. By the time I got anything remotely close to working, we'd already missed our deadlines for both the DX10 work and the Halo 2/Shadowrun converter.

The death of Falling Leaf

After all this went down, I fell into a bit of a depression. I became demoralized to the point of just not wanting to go on anymore, in the face of repeated failures, very much in public. Despite us not walking away with a dime -- we made approximately $7000 in total, none of which went to any of the founders of the company -- I felt that we'd ripped people off, despite the best of intentions. It wasn't long after this that Brian (one of my co-founders) stepped down as CEO, and I closed the doors on the company. The source to Alky and Philosopher (the compiler framework used in the shader work) were released at the same time.

Lessons Learned

If you're going to run an open source project, make it easy for people to contribute. Not only that, make it worthwhile for them to contribute and make them a part of the project, not just free workers.
If you're going to kill an open source project, be up front with the people involved. It's not only dishonest not to do so, you lose people who may well go to bat for you even if you are commercial. This is especially important for a project like Alky, where we faced nearly universal negativity.
If you're going to change focus, be clear with your users as to what's going on, and make sure you make it clear that the old stuff is dead and gone. If you don't, you come off looking terrible in the press, and it just makes you look like amateurs (which we were).
Make sure your employees are actually doing what they're supposed to be doing, especially if they're working remotely. This was really the final nail in the coffin for Falling Leaf.

Alky Reborn

Now, with all of that said, there's a light at the end of the tunnel perhaps. The source for Alky has been pulled into Github and it seems that development is picking up again. Perhaps I can shed some light into what design decisions were made, how it was implemented, and how I'd do it now if I were so inclined. I don't plan on working on Alky again (that time has passed), but I'd still love to see it succeed.

The old Alky design

Alky's original prototype had a very simple design, library-wise. There was one big LibAlky which contained all of the Win32 API implementations, each talking directly to native APIs. This design very quickly became unworkable, as we had tons of duplicated, unmaintainable code.

The new Alky design

Alky was redesigned such that we had the Nucleus and the Win32 APIs. The Nucleus was intended to abstract away the platform-specific details and expose a universal API that the Win32 APIs could sit on cleanly. While a good idea, it quickly broke down in implementation. I ended up writing code that straddled the Nucleus and the Linux/OS X APIs, rather than abstracting everything away. This led to slower development and an even more complicated code base.

Potential new design

Having done two implementations of Alky and quite a few other projects that relate to it in concept (IronBabel, Renraku (OS design plays a factor here), etc), I think I'm at a place where I can perhaps come up with a workable design.

The key point where both Alky implementations (and Wine, IMHO) failed is in maintainability. The codebase was a rats nest, much like the real Win32 APIs, and neither implementation managed to help this at all. I think this needs to be the absolute top priority, above performance, compatibility, and all else. If your code is maintainable, all the rest can happen.

With that focus in mind, here are the things I'd do with a new design.

Implement the APIs on top of Mono, taking advantage of the flexible marshalling that P/Invoke gives you. This will allow you to simplify things greatly, and will only have a marginal performance hit in most cases.
In cases where performance is critical, drop down and implement things in C, C , or assembly. If this chunk of the project is greater than 10% of the codebase, you've got bigger problems.
Abstract, abstract, abstract. Break things into the smallest chunks possible and reuse them. This is what we tried to do with the Nucleus idea, but it was easy to just ignore it for complex pieces.
Write thorough unit tests for every API that's implemented (public and internal). Regression testing would also make things really nice.
Rather than trying to get real games/applications running immediately, write your own simple applications that test specific chunks. This would've cut down considerably on the development time in the old revisions of Alky.
Write a great debugger to be able to step through applications cleanly. In the old days, I'd break out IDA Pro and step through a game on Windows, then compare the behavior to the Alkified version, but this was just downright painful.
Make it work on Windows, to allow easy side-by-side comparisons.

The suggestion that this should be built on top of Mono/.NET sounds ridiculous, I'm sure, but I do think it'd give the project a shot.

In Closing

I hope this has given you some insight into what went down with Falling Leaf, some ideas of what not to do (obviously, it's easy to completely overlook these things when you're down in the trenches, as we did), and where Alky could one day go. I wish the Alky Reborn folks the best of luck, and I hope some of my advice helps.

Happy Hacking,
- Cody Brocious (Daeken)

Designing a .NET Computer

Thu, 19 Nov 2009 00:00:00 GMT

Developing Renraku has led me to some interesting thoughts, not the least of which is the feasibility of a .NET computer.

Concept

Existing computers know only a few core things:

Move to/from memory and registers
Branch
Perform arithmetic

All the other smarts are built on top of that, leaving us with layer upon layer of complexity. What if we were to add some logic to the hardware and introduce a thin software layer on top of that?

Design

CPU

The CPU will execute raw CIL blobs and exposes hardware interfaces and interrupts. The most likely route is to keep the stack in a large on-die cache, doing a small lookahead in decoding to prevent multiple stack hits in arithmetic.

Since the CPU is going to be executing CIL directly, it's going to have to deal with object initialization, virtual method lookups, array allocation, memory manager, and many other tasks that existing systems handle purely in software. This is achieved by splitting the task between the CPU and a hypervisor.

Hypervisor

The hypervisor, like everything else, will be built completely in managed code. However, it will be a raw blob of CIL, instead of the normal PE container. The hypervisor will handle most of the high-level tasks in general, but it's up to the specific hardware manufacturer to determine what goes in the CPU and what goes in the hypervisor. The hypervisor will come up when the machine is booted, much like a BIOS. After performing hardware initialization, it will look for the bootloader.

The hypervisor provides a standardized interface to talk to drivers and memory, which are verified. In this interface, it will also provide hooks for the memory manager, allowing the OS to handle it in a nice way. Interrupts will also be hooked via this interface. The hypervisor instance will be passed to the bootloader, which then passes it on to the OS.

Bootloader

The bootloader is a standard .NET binary, and its job is simply to hand control off to the OS. Its function is effectively identical to that of a bootloader on an existing system.

OS

The OS will look very similar to existing managed OSes, with a few exceptions. Obviously, there's no need to write your own .NET implementation, and no need to write a compiler.

Because it exclusively talks through the hypervisor interface, it's possible to nest OSes as deep as you want, you simply have to expose your own mock hypervisor. You can handle resource arbitration here and run them very efficiently.

Unlike most OSes, it won't get free reign over the system. To access devices, it'll have to use verified channels with the hypervisor, to ensure that consistency is upheld. This is similar to the way that Singularity functions.

Is it a good idea?

Maybe, maybe not. I don't know how efficient a CIL chip would be or how much of the .NET framework can really be pushed onto a chip, but the core idea is really an extension of Renraku. A pure-managed system would be very powerful if done properly, but it's a pretty drastic change.

A nice middle ground would be utilizing an existing processor and using the hypervisor in place of the existing BIOS/EFI. With some code caching magic, the CIL->X compilation could be done at this level and be completely transparent to the OS. This may well be a better route than a CIL CPU, but it's tough without having more information on the hardware challenges involved with such a thing.

Happy hacking,
- Cody Brocious (Daeken)

Renraku OS: FAQ

Thu, 19 Nov 2009 00:00:00 GMT

This is the current Renraku OS FAQ. If you have a question that's not answered here, feel free to join us in #renraku on irc.freenode.net.

What is Renraku OS?

Renraku is a pure managed code operating system, written in Boo from the ground up. The goal is to produce a high-quality operating system without the traditional security, performance, and flexibility issues.

Is Renraku open source?

Yes, Renraku is distributed under the CDDL (Common Development and Distribution License).

Why not use X in place of .NET?

The .NET infrastructure is fairly simple, powerful, (very) easy to compile, and flexible enough for our purposes. Another tool may be better suited to the task, but we've found it to be a good match.

Why use Boo instead of C#?

Although C# is the more traditional choice for .NET work, it lacks macros and is simply much more verbose. Boo is very easy to pick up and significantly more powerful. However, our choice of Boo doesn't mean that the rest of the OS has to be written in Boo. Services, Capsules, and drivers can all be written in any .NET language.

Is it POSIX compatible?

No, by design. POSIX was great in 1989, but times have changed and we need to move forward.

What about legacy applications?

Virtualization is the only planned route for legacy code at the moment, as it's the most efficient and lowest cost. Perhaps we'll support running certain types of legacy applications in a more direct route in the future, but it's doubtful.

Won't this be slow?

For a while, yes. However, compilers are rapidly getting considerably better and compilers like Microsoft's Bartok are paving the way toward faster code. In addition, our lack of separate address spaces and privilege levels make system calls and task switches dramatically cheaper than on existing OSes.

No separate address spaces? What keeps applications apart?

Because we use pure managed code, we're guaranteed that applications cannot touch memory they aren't directly given. This means that the compiler has the last word on all memory access in the code, reducing the attack surface of the OS considerably.

Does it run X?

Unless the application you have in mind is playing with our shell, looking at our logo, or acquiring an IP address via DHCP, probably not. Renraku is still in its infancy, but we hope it'll be useful someday.

Renraku OS: Networking, Hosted Mode, Moving Forward

Thu, 19 Nov 2009 00:00:00 GMT

There hasn't been any news on Renraku in a while, so I figured I'd throw up an update on what's going on.

Where we are

Not a whole lot has changed since the last post. We have a clear direction on where we're moving and we're making progress, albeit more slowly than in the first few months.

The biggest new thing is networking. We have Ethernet, IP, UDP, DHCP, and ARP, although the IP and UDP implementations still need some love. We also have a driver for the AMD PCnet card used in VMware, so testing is fairly easy.

Outside of this, most of the work has been in design, where we're solidifying the way the system works.

Where we're going

Hosted mode

Renraku is soon going to be able to run on top of any existing .NET implementation, allowing considerably simpler development and debugging. The primary goal with hosted mode is to allow developers to tackle higher-level tasks (GUI work, filesystems, etc) while our .NET implementation, kernel, and drivers are built.

When you have a hosted Renraku kernel, you'll run the kernel like any other application, then all of Renraku's services will start up just like on the bare metal. At that point, you'll be able to run other applications inside it, with the benefit of the drivers being dealt with by the underlying OS. This should significantly lower the barrier to entry to hacking on Renraku, while making our code considerably better.

To achieve this, we're splitting Renraku into several pieces:

Compiler
Kernel
.NET Base Class Library
Services
Applications

When we do this, we need to split platform-specific details away from the rest of the code. Hosted mode will just be another platform to the OS, so this also gives us much-needed platform independence. (This will make it possible to write targets for ARM, x64, etc in the near future -- more on this soon.)

In hosted mode, certain services will run as-is, but others will talk to the host OS. For networking, a driver will be implemented that uses raw sockets, utilizing our stack above it. For storage, we can create a virtual disk in a file. For video and input (and sound, eventually), we'll likely backend to SDL.

Whenever possible, we'll use more Renraku services (e.g. not just implementing the 'tcp' service as a wrapper around the .NET TCP classes), which will keep the code from fragmenting too much.

Utilizing Mono's BCL

In the early days of Renraku, I pushed hard against the idea of using Mono's BCL, preferring that we write our own. While I'd like to go with our own BCL that's tightly tied to the Renraku kernel, cracks are quickly appearing in that idea. With us launching our hosted mode initiative, it's clear that the BCL needs to be separated as much as possible. At this point, it's starting to look like going with a modified Mono BCL may be the best route; at the very least, it's worth investigating.

This is simpler than writing our own BCL of course, but it's no trivial task itself. Integrating it with the Renraku kernel is going to be tough; most likely, the right way to do it is to strip the BCL down to the core components that we use and implement now, then move up from there. The other big challenge is compiler support; we currently don't support exceptions and several other things that the BCL uses heavily.

Regardless of the difficulty, this may well prove to be the right way for us to go. Hopefully it will work out, as it'll accelerate Renraku's development significantly.

Capsules

Capsules are our answer to the application and process concept in most OSes. A Capsule holds running tasks, connected events, and the service context. When a Capsule starts, its main class is initialized, which will set up services, throw up a GUI, or whatever. Unlike a process on most OSes, a Capsule does not have to have a thread (task) running at all times, acting much like TSRs did in the old days. Once initialization is completed, the main task can die off, leaving all its delegates and services up. Because of the design of the system, most Capsules will not have a main task, leading to a more nimble system.

The strength of Capsules is most apparant with a GUI or networking app. In a GUI, the main task will put up the GUI and connect events, then die off. When an event occurs, the GUI service calls the Capsule's delegate, which spawns a task inside the Capsule to deal with it.

A network service will work similarly, dying off after it registers itself, waking up only to accept a new client, then it'll wake up when communication happens. This makes writing network applications very, very simple and efficient.

One outstanding problem is how to handle a Capsule being force killed. We need to handle this in a way that guarantees that the system remains consistent, which is tough. Anyone have a proof debunking the halting problem?

Formal design documents

Currently the Renraku documentation is, to put it nicely, sparse. Code isn't commented at all and there's no documentation on how anything works. I've started writing real design docs, which will go live on this blog over the coming weeks. Here are the primary documents that are being worked on:

Capsule architecture
Input chain of command (how input gets from the hardware to the right place)
Storage architecture
Networking
Graphics
Compiler
Security

Capsules will definitely be the first to go out, as it's the most important (it blocks the most other elements of the system); the order on the rest is up in the air.

Closing notes

The work I've done on Renraku OS so far has been some of my best and certainly some of the most rewarding. I'd like to thank everyone who's contributed or just cheered on the team. Even if this never takes off (the likelihood is quite small), Renraku will be an OS we can all be proud of, because of you guys.

Happy hacking,
- Cody Brocious (Daeken)

How To Lose A Customer In Two Lines

Fri, 13 Nov 2009 00:00:00 GMT

You know, I'm not a picky customer. When I find a company that does the job well and without causing problems, I tend to stick with them for a long time. I also tend to advocate for good companies, because they're becoming less and less common. Today, a company just lost my business forever, and I doubt anyone else who sees this will ever deal with them either. I ordered some Adrafinil from Nubrain.com yesterday. The ordered was marked as completed in their site and I wasn't given a tracking number (didn't think about it at the time, but first class USPS was used, so there was no way to track it). As I was curious to see how long it'd take to arrive, I looked up where it was shipping from and found that they're located in Lawrenceville, GA, where I reside. Since the site didn't say it had shipped yet, I decided to send them an email to see if it was possible to pick up the order:

Hello, I was looking at your site to figure out where my order (redacted) was going to ship from, to get some idea of the time it'd take to ship, when I found that you're located in Lawrenceville as well. If my order hasn't shipped yet, would it be possible to pick it up instead? Thanks, - Cody Brocious

Not a terribly difficult request. If the answer was no, or it had already shipped, I would've thought nothing of it and moved on. Instead, I got this:

WE ONLY SEND BY US MAIL IF YOU WANT PICK UP CALL PAPA JOHNS PIZZA

Is it really that much of an inconvenience to send back "Sorry, but our store only delivers"? When you're in a market like shipping pharmaceuticals, a certain amount of trust is required, not that this would be appropriate in any market. If you want to keep your customers, you have to treat them well, not like you're being inconvenienced by them. If you don't want to do the job, don't do it; don't act like the customer is doing you a disservice by giving you money.

Cody Brocious

More Illegal Numbers

Thu, 29 Oct 2009 00:00:00 GMT

Texas Instruments has decided to C&D the UnitedTI forum for distributing the TI83 signing key (as reported here). This makes another beautiful illegal number. Distribute it far and wide -- no number should be illegal to distribute.

Gentlemen,

A mathematical morsel for your entertainment and edification.

The number
6,857,599,914,349,403,977,654,744,967,
172,758,179,904,114,264,612,947,326,
127,169,976,133,296,980,951,450,542,
789,808,884,504,301,075,550,786,464,
802,304,019,795,402,754,670,660,318,
614,966,266,413,770,127

is the product of
5,174,413,344,875,007,990,519,123,187,
618,500,139,954,995,264,909,695,897,
020,209,972,309,881,454,541

and
1,325,290,319,363,741,258,636,842,042,
448,323,483,211,759,628,292,406,959,
481,461,131,759,210,884,908,747.

RMS is a traitor to the Free Software community

Sat, 26 Sep 2009 00:00:00 GMT

Yesterday I read about a comment from RMS at Software Freedom Day in Boston. He referred to Miguel de Icaza as "basically a traitor to the Free Software community". Sorry, but this is too much, even for him.

RMS was a driving force early on and is responsible in large part for the open source movement. Since then, I believe he's done far more to harm the community than he's done to help it. From silliness like the famegrab that is the term GNU/Linux, to the attempts to fight Tivoization of Linux with the GPLv3 (notice that the Linux kernel is still GPLv2? This isn't an accident.), he's shown himself to be out of touch with the needs and desires of the community as a whole.

At a time when Free Software is bigger than ever and becoming more and more important in the commercial space, the last thing we need is a 'leader' who actively fights against anything that doesn't fit his exact model of freedom. At some point, you have to take a step back and realize that blind idealism does nothing but stop forward movement.

Miguel: Keep up the great work. From Gnome to Mono, you've shown an active interest in improving the Free Software desktop and haven't hesitated to make change where you thought it was necessary.

RMS: Start thinking objectively about the effect your words have on the community. I respect you for sticking to your guns, but your position requires you to see the big picture too.

Cody Brocious (Daeken)

Injecting Arbitrary Python Into EVE Online

Wed, 23 Sep 2009 00:00:00 GMT

I finally broke down and installed EVE Online and registered for a trial, since quite a few of my friends have been talking about it nonstop. Being the dork I am, I looked through the binaries that were installed, and I immediately noticed that they shipped Stackless Python 2.5 as a separate binary (python25.dll). Of course, this led me to an investigation of what I could accomplish with respect to modifying the client without actually patching any code. A few hours later, I was able to run any Python code inside of EVE. All of the code is available from the Github repository here:

Injector

First, you have to get some sort of code into the game process. My favorite tool for this stuff is EasyHook, as it lets you inject .NET code into processes. EasyHook, as the name implies, is very easy. You don't even need to hook anything here, just get some code running in the process. With a simple boilerplate NAnt build file and the source of an earlier hooking project, it only takes a few minutes to strip it down to what we need. EveInjector.cs finds the 'exefile' process for EVE and reads the given Python file into a string. It then uses RemoteHooking to inject EIInject.dll into the process. In EIInject.boo, you can see what exactly is going on. In the Run method, the following are called: Py_Initialize, PyRun_SimpleString, and RemoteHooking.WakeUpProcess. The Initialize call should've already happened, but this prevents the client from crashing if the injection happens very early in the load process. The PyRun call runs the code you gave, and the WakeUpProcess call causes the EVE process to come back to life, although it never seems to actually go to sleep. We now have control over the EVE process.

Logging

If you notice, there's a LogServer.exe binary in the EVE directory. This will display all sorts of interesting information from the EVE client. We need to be able to write to this. In my early experimentation, I noticed that traceback.print_exc() would go to the log server, so it was clear that stderr was tied to the log. By doing the following, you can print to the log directly: import sys sys.stdout = sys.stderr

Modules from compiled.code

All Python code in the client is Stackless Python bytecode and split up into a few files: lib/corelib.ccp, lib/corestdlib.ccp, lib/evelib.ccp, script/compiled.code. You can read about the details of these files on the EVEmu Wiki if you want to play with them. Anyway, the compiled.code file holds the bulk of the game code, but if you inspect sys.modules, you won't see anything from there. That's because it's handled by the 'nasty' module in libccp. To expose these modules to sys.modules (and thus make it easy to import them): import sys import nasty for key, value in nasty.nasty.mods.items(): sys.modules[key] = value

Services

In the 'svc' module, you can see all the service classes available on the client, but you can't directly instantiate any of them. Instead, you have to use the 'eve' instance. The 'eve' instance is the center of everything, client-wise. To get to services: from eve import eve gameui = eve.LocalSvc('gameui') You can see the methods available in the gameui instance via the svc module: from svc import gameui print dir(gameui)

Future

From here, getting to the bytecode of the functions is fairly easy, so it'd be nice to have an in-process decompiler that dumps the code of all modules, without having to mess around with the files directly. There are also a large number of bugs in the current injector, which I think stem from this code running in another thread from the normal Python interpreter. I'm fairly sure there's some way to acquire the GIL for the duration of the injected code running, which should fix it. All that said, I think there's some fun stuff that can be done with this. From here, it wouldn't take a huge amount of work to build something akin to Macroquest, where you can extend the UI, automate tasks, and generally make life better inside the client. With some tweaking, it's possible to make this fairly difficult to detect, but I'd be careful using this on an account you really care about early on. Happy Hacking,
- Cody Brocious (Daeken)

Reversing The Pokerstars Protocol, Part 1: Compression and transport basics

Wed, 19 Aug 2009 00:00:00 GMT

Let me start with a little background. I got into poker botting after reading James Devlin's great series on Coding The Wheel. I tried a few different attempts at the IO layer (hooking the poker client directly, screenscraping (wrote a cool little library), etc) before settling on hacking the protocol directly. However, I also started getting into actually playing poker heavily, and I realized a little while ago that there's no way this bot will be profitable (including my time) unless I throw years of work at it, so I've decided to release my knowledge publicly. I've not seen anyone reversing the PStars (or any other service, in fact) protocol publicly, so I'm hoping to help out there.

The likelihood of things staying the same after this is released is very slim, so for reference my work is being done on the latest client as of today, 8/19/09, version 3.1.5.8. All code is available on the Github repo, and I'll be doing code releases to correspond with the parts of this guide. This stuff will only work on Windows with Python 2.6 .

First things first, we have to go into this with a game plan. We need to end up with a client, but what's the best way to get there, since we need to be able to adapt quickly to changes? I cut my teeth on server emulators for MMOs, and I've found that building a server can allow you to find changes quickly. However, we also need to be able to see traffic, so we need a way of sniffing and displaying traffic. Therefore, I think the best route is to build the following: an MITM proxy with a web interface for viewing packets, a server emulator, and a client library.

Now that we have a plan, we can to dive right in. I made the assumption that everything was done over SSL, and quickly confirmed this with Wireshark. There's a lot of traffic and it indeed uses SSL.

I decided to implement everything for Python 2.6, to take advantage of the ssl library. With this, I built a simple MITM to display traffic both ways. But here's the first snag: the PokerStars client checks the server's certificate. However, this was an easy fix. I generated a keypair for the MITM (can also be used for the server emulator) and found that they stored the normal server cert in plaintext in the client binary. A quick patch script later and the MITM works like a charm, dumping raw data. A quick look over, and it became clear that there's a static initialization field (42 bytes) at the beginning of both the server and client transmissions, and then packets sent with a 16-bit (big endian) size in front of them.

Time to start looking at the client to figure out what it is. Breaking out my handy dandy IDA Pro and loading PokerStars into it, I started searching through the strings list for compression- and crypto-related strings. I stumbled upon a few interesting strings, the clearest of which was: "_CommCompressedCommunicator: LZHL frame is too long". Googling around a bit, I found that LZHL is a lightweight compression algorithm intended for high-performance network situations.

Next stumbling block: the original implementation of LZHL has fallen off the Internet, and the only remaining implementation I can find is buried in a big, unmaintained library. Cue 2 days of attempting to reimplement the algorithm in Python with no actual verification that it is the only thing in play. For those of you playing the home game, this is a Bad Idea. After a day of debugging, I gave up and simply beat the C code into submission, utilizing the ctypes library in Python to wrap it into a usable form.

Tying this into the MITM, I can now see the decompressed frames coming across the wire, complete with nice clean plaintext. However, there is a clear header on the packets, so that has to be reversed next.

Here are some choice packets from the beginning of a connection:

-> 106 bytes:
0000 00 6a c1 06 00 00 38 33 60 10 02 69 70 3a 36 36| .j....83 `..ip:66
0010 2e 32 31 32 2e 32 32 35 2e 31 37 3a 32 36 30 30| .212.225 .17:2600
0020 32 00 41 6c 74 31 4c 6f 62 62 79 53 65 72 76 65| 2.Alt1Lo bbyServe
0030 72 49 6e 73 74 61 6e 63 65 00 53 68 61 64 6f 77| rInstanc e.Shadow
0040 54 6f 75 72 6e 61 6d 65 6e 74 73 50 75 62 6c 69| Tourname ntsPubli
0050 73 68 65 72 30 00 00 00 00 00 00 00 00 00 00 00| sher0... ........
0060 00 00 00 00 00 00 00 00 00 00                  | ........ ..

<- 2048 bytes:
0000 08 00 81 00 0d 6a 54 06 00 00 39 33 60 10 02 d7| .....jT. ..93`...
0010 4e 7c e2 06 00 00 00 00 02 95 f3 10 30 00 00 1b| N|...... ....0... 
0020 18 4a 72 d0 48 eb 65 b3 2d 00 00 00 00 00 02 00| .Jr.H.e. -....... 
0030 00 00 00 01 02 95 f3 10 11 5b 00 00 e3 61 43 02| ........ .[...aC. 
0040 8f fd 1b 00 02 ff 00 e3 61 44 00 e3 61 44 ff 00| ........ aD..aD.. 
0050 00 00 85 0b 0b 60 df 69 70 3a 36 36 2e 32 31 32| .....`.i p:66.212 
0060 2e 32 32 35 2e 31 37 3a 32 36 30 30 32 00 00 00| .225.17: 26002... 
0070 00 00 00 00 00 00 00 00 00 00 06 e0 00 00 00 f0| ........ ........ 
0080 00 00 00 00 00 00 80 32 30 20 50 4c 20 42 61 64| .......2 0.PL.Bad 
0090 75 67 69 00 00 00 00 00 00 00 00 00 00 08 44 60| ugi..... ......D` 
00a0 02 d0 01 00 02 00 00 00 03 00 00 00 10 ff ff ff| ........ ........ 
00b0 ff 00 40 00 00 00 00 00 00 00 00 00 00 00 00 00| ..@..... ........ 
00c0 00 00 00 00 00 00 00 00 08 00 00 00 00 00 00 09| ........ ........ 
00d0 45 55 52 00 00 00 00 00 ff 00 e3 61 45 02 8f fd| EUR..... ...aE... 
00e0 1b ff 00 00 00 08 00 00 01 80 00 00 00 01 00 00| ........ ........ 
00f0 e3 61 46 02 71 f0 93 00 02 ff 00 e3 61 47 00 e3| .aF.q... ....aG.. 
0100 61 47 ff 00 00 00 85 0b 0b 60 e0 69 70 3a 36 36| aG...... .`.ip:66 
0110 2e 32 31 32 2e 32 32 35 2e 31 37 3a 32 36 30 30| .212.225 .17:2600 
0120 32 00 00 00 00 00 00 00 00 00 00 00 00 00 0a 6e| 2....... .......n 
0130 00 00 01 4a 00 00 00 00 00 00 80 33 30 20 50 4c| ...J.... ...30.PL 
0140 20 42 61 64 75 67 69 00 00 00 00 00 00 00 00 00| .Badugi. ........ 
0150 00 08 44 60 02 d0 01 00 02 00 00 00 03 00 00 00| ..D`.... ........ 
--snip--

<- 2048 bytes: 
0000 08 00 01 06 44 00 02 d7 02 00 03 00 00 00 09 00| ....D... ........ 
0010 00 00 10 ff ff ff ff 80 00 00 00 00 00 00 00 00| ........ ........ 
0020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 02 00| ........ ........ 
0030 00 00 00 00 00 09 45 55 52 00 00 00 00 04 ff 00| ......EU R....... 
0040 e3 61 9f 02 7e 31 23 ff 00 00 00 08 00 00 01 80| .a..~1#. ........ 
0050 00 00 00 00 00 00 e3 61 a3 02 8d 14 01 00 02 ff| .......a ........ 
0060 00 e3 61 a4 00 e3 61 a4 ff 00 00 00 b5 0b 0b 61| ..a...a. .......a 
0070 05 69 70 3a 36 36 2e 32 31 32 2e 32 32 35 2e 31| .ip:66.2 12.225.1 
0080 37 3a 32 36 30 30 32 00 00 00 00 00 00 00 00 00| 7:26002. ........ 
0090 00 00 00 00 24 40 00 00 02 d0 00 00 00 00 00 00| ....$@.. ........ 
00a0 80 31 30 30 20 4e 4c 20 48 6f 6c 64 27 65 6d 20| .100.NL. Hold'em. 
00b0 5b 48 65 61 64 73 2d 55 70 20 3c 62 3e 54 75 72| [Heads-U p.Tur...
00c0 62 6f 3c 2f 62 3e 20 31 36 20 50 6c 61 79 65 72| bo.1.... 6.Player 
00d0 73 5d 00 80 31 30 30 20 54 69 63 6b 65 74 00 00| s]..100. Ticket..
--snip--

I've truncated two of these, because the actual data doesn't much matter here. First and foremost, look at the first two bytes of each packet. It's a 16-bit value corresponding to the packet size. At first, I believed they had the compressed size (outside) and decompressed size (inside) for verification purposes, but a little later I discovered that they'd combine packets where possible. A single compressed packet can contain multiple packets, and they'll push it up to the 65536 bytes before compression (no logic for combining more intelligently).

Next up comes a flags byte. This I determined by guess and check largely. The breakdown is:

Flags & 0xC0 == 0x80: Beginning of a message chain. If the next byte is 0, there are 4 unknown bytes following.
Flags & 0x40 == 0x40: End of a message chain.

If the flags byte is anything else, it's a message in the chain (if it's placed in a chain) or a self-standing message. From here, we'll refer to 'message' as the raw data inside the transport stream. That is, everything from the flags byte+1 (or +5, in the case of it being the start of a message chain and the flags+1 byte being 0 as indicated above) onward, including additional messages in the chain.

Looking at the messages, we'll see some repetition in byte 2. I guessed that this was the opcode field indicating the type of message, and some quick verification shows this is most likely the case. One other interesting thing is that there's an incrementing ID that is shared by the client and server. With a little detective work, you can see that if byte 1 is 0, the ID is from 3-7 (exclusive), otherwise it's 1-5. By looking at the plaintext that's sent, you can see that the ID is incremented by the client when creating a new request chain, and is used by both sides for the duration of that chain. For instance, you can see that the ID in the packets above is 0x33601002. You can also see that the second packet is a new chain and the third packet is a continuation (it's not the final packet).

Now that we have all this, a clear picture of the transport itself is beginning to form, but here's something unusual: most packets have the same opcodes. Lots of 0x38s, 0x11s, etc. This baffled me for a little while, before I went back and looked at the initial packets, where it was requesting 'server instances' and such. After this, a light clicked on in my head, and I realized that the entire PokerStars protocol is a series of client-server connections funneled through one of a long line of frontend load-balancing servers.

A little bit more digging around told me the following about these connections: (note the --> and -<- conventions for client->server and vice versa)

-> 0x10: A sort of async method call, only used initially.
-> 0x38: Establish a connection to a service provider, from which streaming lobby, tournament, table, etc data is received.
<- 0x39: Response to a connection. Seems to have some sort of header with connection info before the data starts streaming in.

From this, I began to piece together the protocol and how the client/server could be implemented. I wrote a web-based protocol viewer and started pouring over logs.

In the Github repo you'll find the following:

LZHL: VC++ source tree, compiled dll, and Python wrapper.
PSHooker and runpstars.bat: This is an application using my Nethooker library to redirect the PokerStars connection to the MITM/emulator. You'll likely need to edit the binary path in runpstars.bat, and you need to add a HOSTS entry pointing 'pshack.com' to 127.0.0.1 to make it work.
pmitm.py: This is the meat of the codebase. It writes out log.txt containing the raw packet data for consumption by the web interface and gives hex dumps of the packets over stdout.
patchCerts.py: This patches the PokerStars binary to use the unofficial certs. Run with: python26 patchCerts.py path/to/PokerStars.exe path/to/newpokerstars.exe officialcert.pub mitmcert.pub
WebPoker: Pylons project to view the packets. Edit WebPoker/webpoker/lib/Protocol.py to point it at the location of logs.txt from pmitm.

I hope this was clear and that you got something out of it. There's a lot more to rip apart, and I hope to give more (and better) examples in the future.

Thanks for reading, and happy hacking,
- Cody Brocious (Daeken)

Renraku OS: The Way Up

Thu, 06 Aug 2009 00:00:00 GMT

It's now been just over a month since this iteration of the Renraku
project began. In that time, we've made monumental progress in
effectively every way. We're up to 4 developers and a designer (see
the pretty logo?), we've started to solidify the actual design of the
codebase and how it's going to function in the future, and we're
steadily pushing toward being a usable OS. We have a very long path
ahead of us, but we're iterating quickly.

Today, we're proud to announce the release of Renraku OS version 0.2
alpha, codenamed "The Way Up".

What's new:

Services – allow the interface to be decoupled from the implementation and provide a model for drivers, filesystems, network servers, etc.
Contexts – allow applications (Capsules, in Renraku terminology) to access services, and allows nesting such that you can run a Capsule in a sandbox where you provide filtered services, e.g. transparent proxying, nested windows, etc.
Mouse support – Renraku now supports PS/2 mice via a simple service interface.
VGA and general video interface – We can now use the latest in video technology (320x200 VGA, of course) in Renraku applications, even features double buffering. Progress is underway by one of our developers to build a basic GUI, but we have a few example apps.
BCL improvements – We've been gradually implementing various collections (ArrayList, Queue), text helpers (StringBuilder), and other pieces of the BCL, as we make the development process higher and higher level.
Compiler improvements – Inheritance, enums, interface implementation, method overloading, and many other things now work perfectly. There are now only a few key components (reflection and generics being the two big ones) missing before the compiler is effectively functionally complete.
Bootable ISO building – You can now build bootable ISOs along with the kernel image.
New apps to test out our new functionality:
- 'exclaim' – Example nested Context application, turning your '1's into '!'s by hooking the keyboard service. Screenshot
- 'draw' – Simple bouncing box VGA demo. Screenshot
- 'mouse' – Control a box cursor with your mouse, changing the colors with the click of a button. Screenshot 1 Screenshot 2
- 'logo' – Renders our logo in beautifully dithered 256 color VGA. Screenshot

You can get a bootable ISO or check out the tree at Github. We'd love to hear what you think; drop by our IRC
channel, #renraku on irc.freenode.net. If you run into
any bugs, feel free to throw them on our issue tracker.

You can also see the complete screenshot gallery as well as a few shots of Renraku running on real hardware.

We've had fun building it, I hope you have fun playing around with it.

Happy Hacking,
- Cody Brocious (Daeken)

Renraku OS: Initial Release

Fri, 17 Jul 2009 00:00:00 GMT

The Renraku OS team is proud to announce the first release, 0.1alpha.
Major strides have been made in the two weeks since the new codebase
was started, and we're very happy with where things are.

We now have:

Initial compiler with support for vtables, a good chunk of the CIL instruction set, string object constants, etc.
Initial memory manager (real memory manager is in progress).
Basic object manager.
The start of System.Array/String/Console.
Keyboard driver and US English keymap.
Basic shell that can execute applications.
Basic HAL and interfaces for timer, keyboard, and network drivers.
Initial PCI base class.
A few simple applications:
- 'echo' – Echo terminal
- 'reverse' – Reverses the arguments passed to it
- 'halstatus' – Prints the status of loaded drivers
- 'pci' – Scans for PCI devices and dumps the vendor and device IDs
- 'shell' – Creates a subshell

We've also created an IRC channel, #renraku on Freenode where
the developers are generally available. If you want to give it a
spin, grab the kernel image and boot it with GRUB. If you run into any problems, drop
us a line on the channel.

For the curious, you can check out this gallery of screenshots of the various commands. And as always, the
codebase is on Github.

Give it a shot and let us know what you think.

Happy hacking,
- Cody Brocious

Renraku OS: It Lives!

Thu, 09 Jul 2009 00:00:00 GMT

This is a follow-up to my previous post, Renraku: Future OS.

I got the latest incarnation of Renraku booting yesterday, based on the code base I started the same day I wrote the previous article. There's not much to it, but development is progressing very, very quickly since I have plenty of knowledge of what not to do from previous attempts. What's there (in the GitHub repo) now supports very basic functionality: multiple function compilation, a good number of "Base instructions" (from the ECMA CIL spec), static fields, pointer intrinsics, and string intrinsics (get_Chars for now).

The test kernel is available here (GitHub) and this boots as expected. Obviously, it doesn't do much, but that's coming. What needs to be done in the short term:

Struct support
Struct pointers
Basic object manager

That's it. Once those are done, there's a primitive object model which means actual functionality can be written. A few more days of work, and we should be able to see a simple shell come up.

Happy Hacking,
- Cody Brocious (Daeken)

Renraku: Future OS

Fri, 03 Jul 2009 00:00:00 GMT

Renraku is a family of managed research kernels I've been working on
for around 4 years. Though the goals and implementations have changed
over the years, they're all designed to be highly reliable, fast,
portable, and flexible enough to keep up with the changing needs of
today's technology.

Background

Initially, Renraku (known as Colorblind at the time) was simply an
experiment. Inspired by Unununium in its Python era, I saw the value
in kernels being very high level. Ease of development, in the hands
of the right hacker, could mean vast improvements in a short period of
time. With this in mind, I set out to build a prototype. I built a
number of revisions on top of existing OSes, but in the end I couldn't
get the design I wanted, so I shelved the project, but never forgot
about it.

Shortly thereafter, I started playing around with the development of
Talk, an L4-inspired microkernel. It was very simple (written in
straight x86 assembly) and was intended to see how small such a
microkernel could be without sacrificing critical functionality. It
didn't get far due to not having enough time, but while designing the
set of base services and their interactions, it made me think back to
Renraku.

The project sat on the back burner for about a year and a half, until I saw
MSR's Singularity project, some time around 2006. At this point, I
started seriously thinking about what I want in a future OS and how I
can go about implementing it. I've since written a few prototypes in
a mix of C# and Nemerle, and the goals have largely solidified.

Goals

High level:

Pure managed code. No C, no assembly not generated by the compiler, no pointer access outside of the object manager and certain parts of the kernel.
Everything is an object. Turtles all the way down.
Single address space and no userland. Everything is ring 0.
Completely network-transparent objects.
Object store instead of a traditional filesystem.
System-wide orthogonal persistence.
All components reloadable at runtime.

Implementation:

Pure .NET. Currently playing around with Boo for it, due to their new macro syntax and functionality.
Ahead of time compilation for the base services.
Object model implemented on top of itself. That is, the object model is implemented in terms of value types and pointers, and the compiler has intrinsics for object model instructions to auto-call the implementation.
While designing/implementing any feature, now is always better than later. Always, always, always go for the practical, simple route before considering a more complex one.

Roadmap

This is a rough roadmap of the implementation of Renraku. How far
this ends up going is really up in the air; it could go to the end,
it could be abandoned half way and rewritten again. Eventually, I'd
love to see these ideas come to fruition, though.

Initial CIL->x86 compiler. Uses the actual stack to execute code, as optimizations can be added later.
Initial memory/object manager. Nothing fancy, just enough to get things up and running.
Begin implementing the .NET BCL. Get basic string manipulation and collections working.
Implement keyboard support.
Implement a basic shell. Allow object inspection/manipulation.
Implement initial object store.
Implement basic networking and network transparent objects.
Build framebuffer support and a basic GUI library.
Implement the dependencies of the Boo, C#, and Renraku compilers.
Implement a basic terminal and REPL.

Once these things are implemented, development should be able to
progress very rapidly. There's just a lot of work to do to get there.

What would you want to see in a future OS? What would you like to see
go away? What new UI paradigms ought to be used? I'd love to hear
feedback on this, as there's simply a lot to think about.

Happy Hacking,
- Cody Brocious (Daeken)

Partially Destructive Garbage Collection

Thu, 02 Jul 2009 00:00:00 GMT

While lying in bed and trying to sleep last night, I had an idea I'd like some feedback on. The idea is partially destructive garbage collection, which allows the runtime to handle caching, instead of the programmer. With PDGC, the runtime is allowed to destroy an object and keep only the relevant data for recreation around. If you're running a photo management application, for instance, you may want to keep JPEGs and thumbnail bitmaps in memory, while not keeping the full bitmap around. In theory, this is a problem that PDGC can solve for you. At compile-time, the compiler can look at your class and determine whether or not your object's creation has side-effects, or depends on factors outside of its control (e.g. IO), and make a determination if it's partially destroyable. If it is, then the compiler embeds code into the constructor (and other relevant methods*) to save the data needed to recreate the object. At run-time, the frequency of access, object creation time, etc will be tracked, and the PDGC can make the decision to destroy the object or keep it around. If the object is destroyed, it doesn't entirely go away; rather, it's replaced with a stand-in object that stores the relevant data for recreation. Further ideas:

You could allow the PDGC to only destroy certain object properties, rather than the full object
A more general approach to language-level caching would allow the objects to be saved to disk and loaded lazily. This is something I plan to explore as part of my Renraku research kernel, the details of which I'll be posting on soon.

General open questions:

What research has been done on this previously? What were the findings?
How much of this belongs in the compiler and how much in the runtime? This obviously comes down to the implementation, but I believe there needs to be research into good, general approaches.

Implementation questions:

Should this be involved with the GC at all? In my mind, they're related, but in a specific implementation they may well be completely separated.
What factors are used in determining whether or not an object persists? Factors I can think of are: frequency of access, cost of object recreation, size, amount of free memory, amount of free swap space.
Should this be behind-the-scenes compiler magic, or should it be more user controllable? E.g. should the user be able to decide if an object is destroyable at a given point? Should the user be able to override the object recreation portion, rather than letting the constructor do its thing?

I'd love to hear your take on it.

Happy Hacking,
- Cody Brocious (Daeken)

New

Tue, 30 Jun 2009 00:00:00 GMT

I've finally decided to take the plunge and move to Posterous. If all goes according to plan, my posts from here should automagickly be crossposted to my Livejournal, but that's going to primarily become my personal blog; this blog will be used for all things tech/business.

To get things started, I'd like to introduce my new programming language, Ultilang. The name is horrid, so please, add suggestions for a new one here: .

Ultilang is a strange mix of Nemerle, Python, and Lisp. A curly brace-based syntax, heavy metaprogramming, and multiple backends (.NET, JVM, Python, and JavaScript planned) come together into what I intend to be the most flexible, powerful language out there. By leveraging features typically reserved for heavy-duty functional languages and blending them with the ease of use we've come to expect with languages like Python, I hope to create a language suitable for near every task.

The compiler, available at , is written in Python and builds/runs a Python AST. It's currently in the early stages, having just been started yesterday; the first test, a simple recursive fibonacci sequence printer, runs perfectly. There's a decent bit of work to do before the compiler can bootstrap up to Ultilang itself, but it's coming along nicely.

There's a list of planned/requested features at ; feel free to add your own. Even if they don't end up in the language itself, they may well become a macro in one of the libraries.

Stay tuned for more information as the language progresses.