I settled on these three commonly used elements:

*italic* or _italic_
**bold** or __bold__
`code`

I loves me some regular expressions and this is exactly the stuff regex was born to do! It doesn't look very tough. So I dusted off my copy of RegexBuddy and began.

I typed some test data in the test window, and whipped up a little regex in no time at all. This isn't my first time at the disco.

Bam! Yes! Done and done! By gum, I must be a genius programmer!

Despite my obvious genius, I began to have some small, nagging doubts. Is the test phrase...

I would like this to be *italic* please.

... really enough testing?

Sure it is! I can feel in my bones that this thing freakin' works! It's almost like I'm being pulled toward shipping this code by some inexorable, dark, testing ... force. It's so seductively easy!

But wait. I have this whole database of real world comments that people have entered on Stack Overflow. shouldn't I perhaps try my awesome regular expression on that corpus of data to see what happens? Oh, fine. If we must. Just to humor you, nagging doubt. Let's run a query and see.

select Text from PostComments
where dbo.RegexIsMatch(Text, '\*(.*?)\*') = 1

Which produced this list of matches, among others:

Interesting fact about math: x * 7 == x + (x * 2) + (x * 4), or x + x >> 1 + x >> 2. Integer addition is usually pretty cheap.

Thanks. What I needed was to turn on Singleline mode too, and use .*? instead of .*.

yeah, see my edit - change select * to select RESULT.* one row - are sure you have more than one row item with the same InstanceGUID?

Not your main problem, but you are mix and matching wchar_t and TCHAR. mbstowcs() converts from char * to wchar_t *.

aawwwww.... Brainf**k is not valid. :/

Thank goodness I listened to my midichlorians and let the light side of the testing force prevail here!

So how do we fix this regex? We use the light side of the force -- brute force, that is, against a ton of test cases! My job here is relatively easy because I have over 20,000 test cases sitting in a database. You may not have that luxury. Maybe you'll need to go out and find a bunch of test data on the internet somewhere. Or write a function that generates random strings to feed to the routine, also known as fuzz testing.

I wanted to leave the rest of this regular expression as an exercise for the reader, as I'm a sick guy who finds that sort of thing entertaining. If you don't -- well, what the heck is wrong with you, man? But I digress. I've been criticized for not providing, you know, "the answer" in my blog posts. Let's walk through some improvements to our italic regex pattern.

First, let's make sure we have at least one non-whitespace character inside the asterisks. And more than one character in total so we don't match the ** case. We'll use positive lookahead and lookbehind to do that.

\*(?=\S)(.+?)(?<=\S)\*

That helps a lot, but we can test against our data to discover some other problems. We get into trouble when there are unexpected characters in front of or behind the asterisks, like, say, p*q*r. So let's specify that we only want certain characters outside the asterisks.

(?<=[\s^,(])\*(?=\S)(.+?)(?<=\S)\*(?=[\s$,.?!])

Run this third version against the data corpus, and wow, that's starting to look pretty darn good! There are undoubtedly some edge conditions, particularly since we're unlucky enough to be talking about code in a lot of our comments, which has wacky asterisk use.

This regex doesn't have to be (and probably cannot be, given the huge possible number of human inputs) perfect, but running it against a large set of input test data gives me reasonable confidence that I'm not totally screwing up.

So by all means, test your code with the force -- brute force! It's good stuff! Just be careful not to get sloppy, and let the dark side of the testing force prevail. If you think one or two simple test cases covers it, that's taking the easy (and most likely, buggy and incorrect) way out.

[advertisement] Interested in agile? See how a world-leading software vendor is practicing agile.

]]> http://www.codinghorror.com/blog/archives/001285.html Tue, 07 Jul 2009 23:59:59 -0800 Jeff Atwood http://www.codinghorror.com/blog/archives/001284.html Remember that Stack Overflow thing we've been working on? Some commenters on a recent Hacker News article questioned the pricing of Stack Exchange -- essentially, a hosted Stack Overflow:

Seems really pricey for a relatively simple software like this. Someone write an open source alternative? it looks like something that can be thrown together in a weekend.

Ah, yes, the stereotypical programmer response to most projects: it's trivial! I could write that in a week!*

It's even easier than that. Open source alternatives to Stack Overflow already exist, so you've got a head start. Gentlemen, start your compilers! Er, I mean, interpreters!

No, I don't take this claim seriously. Not enough to write a response. And fortunately for me, now I don't need to, because Benjamin Pollack -- one of the few people outside our core team who has access to the Stack Overflow source code -- already wrote a response. Even if I had written a response, I doubt it would have been half as well written as Benjamin's.

Developers think cloning a site like StackOverflow is easy for the same reason that open-source software remains such a horrible pain in the ass to use. When you put a developer in front of StackOverflow, they don't really see StackOverflow. What they actually see is this:

create table QUESTION (ID identity primary key,
                       TITLE varchar(255),
                       BODY text,
                       UPVOTES integer not null default 0,
                       DOWNVOTES integer not null default 0,
                       USER integer references USER(ID));
create table RESPONSE (ID identity primary key,
                       BODY text,
                       UPVOTES integer not null default 0,
                       DOWNVOTES integer not null default 0,
                       QUESTION integer references QUESTION(ID))

If you then tell a developer to replicate StackOverflow, what goes into his head are the above two SQL tables and enough HTML to display them without formatting, and that really is completely doable in a weekend. The smarter ones will realize that they need to implement login and logout, and comments, and that the votes need to be tied to a user, but that's still totally doable in a weekend; it's just a couple more tables in a SQL back-end, and the HTML to show their contents. Use a framework like Django, and you even get basic users and comments for free.

But that's not what StackOverflow is about. Regardless of what your feelings may be on StackOverflow in general, most visitors seem to agree that the user experience is smooth, from start to finish. They feel that they're interacting with a polished product. Even if I didn't know better, I would guess that very little of what actually makes StackOverflow a continuing success has to do with the database schema--and having had a chance to read through StackOverflow's source code, I know how little really does. There is a tremendous amount of spit and polish that goes into making a major website highly usable. A developer, asked how hard something will be to clone, simply does not think about the polish, because the polish is incidental to the implementation.

I have zero doubt that given enough time, open source clones will begin to approximate what we've created with Stack Overflow. It's as inevitable as evolution itself. Well, depending on what time scale you're willing to look at. With a smart, motivated team of closed-source dinosaurs, it is indeed possible to outrun those teeny tiny open-source mammals. For now, anyway. Let's say we're those speedy, clever Velociraptor types of dinosaurs -- those are cool, right?

Despite Benjamin's well reasoned protests, the source code to Stack Overflow is, in fact, actually, kind of ... well, trivial. Although there is starting to be quite a lot of it, as we've been beating on this stuff for almost a year now. That doesn't mean our source code is good, by any means; as usual, we make crappy software, with bugs. But every day, our tiny little three person team of speedy-but-doomed Velociraptors starts out with the same goal. Not to write the best Stack Overflow code possible, but to create the best Stack Overflow experience possible. That's our mission: make Stack Overflow better, in some small way, than it was the day before. We don't always succeed, but we try very, very hard not to suck -- and more importantly, we keep plugging away at it, day after day.

Building a better Stack Overflow experience does involve writing code and building cool features. But more often, it's anything but:

1. synthesizing cleaner, saner HTML markup
2. optimizing our pages for speed and load time efficiency
3. simplifying or improving our site layout, CSS, and graphics
4. responding to support and feedback emails
5. writing a blog post explaining some aspect of the site engine or philosophy
6. being customers of our own sites, asking our own programming questions and sysadmin questions
7. interacting with the community on our dedicated meta-discussion site to help gauge what we should be working on, and where the rough edges are that need polishing
8. electing community moderators and building moderation tools so the community can police and regulate itself as it scales
9. producing Creative Commons dumps of our user-contributed questions and answers
10. coming up with schemes for responsible advertising so we can all make a living
11. producing the Stack Overflow podcast with Joel
12. helping set up logistics for the Stack Overflow DevDays conferences
13. setting up the next site in the trilogy, and figuring out where we go next

As programmers, as much as we might want to believe that

lots_of_awesome_code = success;

There's nothing particularly magical about the production of source code. In fact, writing code is a tiny proportion of what makes most businesses successful.

Code is meaningless if nobody knows about your product. Code is meaningless if the IRS comes and throws you in jail because you didn't do your taxes. Code is meaningless if you get sued because you didn't bother having a software license created by a lawyer.

Writing code is trivial. And fun. And something I continue to love doing. But if you really want your code to be successful, you'll stop coding long enough to do all that other, even more trivial stuff around the code that's necessary to make it successful.

* Although, to be fair, I really could write Twitter in a week. It's so ridiculously simple! Come on!

[advertisement] Interested in agile? See how a world-leading software vendor is practicing agile.

]]> http://www.codinghorror.com/blog/archives/001284.html Mon, 06 Jul 2009 23:59:59 -0800 Jeff Atwood http://www.codinghorror.com/blog/archives/001283.html We recently upgraded our database server to 48 GB of memory -- because hardware is cheap, and programmers are expensive.

Imagine our surprise, then, when we rebooted the server and saw only 32 GB of memory available in Windows Server 2008. Did we install the memory wrong? No, the BIOS screen reported the full 48 GB of memory. In fact, the system information applet even reports 48 GB of memory:

But there's only 32 GB of usable memory in the system, somehow.

Did you feel that? A great disturbance in the Force, as if 17 billion bytes simultaneously cried out in terror and were suddenly silenced. It's so profoundly sad.

That's when I began to suspect the real culprit: weasels.

No. Not the cute weasels. I'm referring to angry, evil marketing weasels.

That's more like it. Those marketing weasels are vicious.

We belatedly discovered post-upgrade that we are foolishly using Windows Server 2008 Standard edition. Which has been arbitrarily limited to 32 GB of memory. Why? So the marketing weasels can segment the market.

It's sort of like if you were all set to buy that new merino wool sweater, and you thought it was going to cost $70, which is well worth it, and when you got to Banana Republic it was on sale for only $50! Now you have an extra $20 in found money that you would have been perfectly happy to give to the Banana Republicans!

Yipes!

That bothers good capitalists. Gosh darn it, if you're willing to do without it, well, give it to me! I can put it to good use, buying a SUV or condo or Mooney or yacht one of those other things capitalists buy!

In economist jargon, capitalists want to capture the consumer surplus.

Let's do this. Instead of charging $220, let's ask each of our customers if they are rich or if they are poor. If they say they're rich, we'll charge them $349. If they say they're poor, we'll charge them $220.

Now how much do we make? Back to Excel. Notice the quantities: we're still selling the same 233 copies, but the richest 42 customers, who were all willing to spend $349 or more, are being asked to spend $349. And our profits just went up! from $43K to about $48K! NICE!

Capture me some more of that consumer surplus stuff!

How many versions of WIndows Server 2008 are there? I count at least six. They're capturing some serious consumer surplus, over there in Redmond.

• Datacenter Edition
• Enterprise Edition
• Standard Edition
• Foundation
• Web
• HPC

Already, I'm confused. Which one of these versions allows me to use all 48 GB of my server's memory? There are no less than six individual "compare" pages to slice and dice all the different features each version contains. Just try to make sense of it all. I dare you. No, I double dog dare you! Oh, and by the way, there's zero pricing information on any of these pages. So open another browser window and factor that into your decisionmaking, too.

I don't mean to single out Microsoft here; lots of companies use this segmented pricing trick. Even Web 2.0 darlings 37 Signals.

Heck, our very own product segments the market.

37signals just does it .. prettier, that's all. They're still asking you if you're poor or rich, and charging you more if you're rich.

Eric Sink also advocates the same "rich customer, poor customer" software pricing policy:

In an ideal world, the price would be different for every customer. The "perfect" pricing scheme would charge every customer a different amount, extracting from each one the maximum amount they are willing to pay.

• The IT guy at Podunk Lutheran College has no money: Gratis.
• The IT guy at a medium-sized real estate agency has some money: $500.
• The IT guy at a Fortune 100 company has tons of money: $50,000.

You can never make your pricing "perfect," but you can do much better than simply setting one constant price for all situations. By carefully tuning all these details, you can find ways to charge more money from the people who are willing to pay more.

This sort of pricing seems exploitative, but it can also be an act of public good -- remember that the poorest customers are paying less; with a one-size-fits-all pricing policy, they might not be able to afford the product at all. Drug companies often follow the same pricing model when selling life-saving drugs to third-world countries. First-world countries end up subsidizing the massive costs of drug development, but the whole world benefits.

What I object to isn't the money involved, but the mental overhead. The whole thing runs so contrary to the spirit of Don't Make Me Think. Sure, don't make us customers think. Unless you want us to think about how much we'd like to pay you, that is.

And what are we paying for? The privilege of flipping the magic bits in the software that say "I am blah edition!" It's all so.. anticlimactic. All that effort, all that poring over complex feature charts and stressing out about pricing plans, and for what? Just to get the one simple, stupid thing I care about -- using all the memory in my server.

Perhaps these complaints, then, point to one unsung advantage of open source software:

Open source software only comes in one edition: awesome.

The money is irrelevant; the expensive resource here is my brain. If I choose open source, I don't have to think about licensing, feature matrices, or recurring billing. I know, I know, we don't use software that costs money here, but I'd almost be willing to pay for the privilege of not having to think about that stuff ever again.

Now if you'll excuse me, I'm having trouble deciding between Windows 7 Smoky Bacon Edition and Windows 7 Kenny Loggins Edition. Bacon is delicious, but I also love that Footloose song..

[advertisement] Interested in agile? See how a world-leading software vendor is practicing agile.

]]> http://www.codinghorror.com/blog/archives/001283.html Wed, 01 Jul 2009 23:59:59 -0800 Jeff Atwood http://www.codinghorror.com/blog/archives/001281.html In programming, abstractions are powerful things:

Joel Spolsky has an article in which he states

All non-trivial abstractions, to some degree, are leaky.

This is overly dogmatic - for example, bignum classes are exactly the same regardless of the native integer multiplication. Ignoring that, this statement is essentially true, but rather inane and missing the point. Without abstractions, all our code would be completely interdependent and unmaintainable, and abstractions do a remarkable job of cleaning that up. It is a testament to the power of abstraction and how much we take it for granted that such a statement can be made at all, as if we always expected to be able to write large pieces of software in a maintainable manner.

But they can cause problems of their own. Let's consider a particular LINQ to SQL query, designed to retrieve the most recent 48 Stack Overflow questions.

var posts = 
  (from p in DB.Posts
  where 
  p.PostTypeId == PostTypeId.Question &&
  p.DeletionDate == null &&
  p.Score >= minscore
  orderby p.LastActivityDate descending
  select p).
  Take(maxposts);

The big hook here is that this is code the compiler actually understands. You get code completion, compiler errors if you rename a database field or mistype the syntax, and so forth. Perhaps best of all, you get an honest to goodness post object as output! So you can turn around and immediately do stuff like this:

foreach (var post in posts.ToList())
{
    Render(post.Body);
}

Pretty cool, right?

Well, that Linq to SQL query is functionally equivalent to this old-school SQL blob. More than functionally, it is literally identical, if you examine the SQL string that LINQ generates behind the scenes:

string query = 
  "select top 48 * from Posts
  where 
  PostTypeId = 1 and 
  DeletionDate is null and 
  Score >= -4
  order by LastActivityDate desc";

This text blob is of course totally opaque to the compiler. Fat-finger a syntax error in here, and you won't find out about it until runtime. Even if it does run without a runtime error, processing the output of the query is awkward. It takes row level references and a lot of tedious data conversion to get at the underlying data.

var posts = DB.ExecuteQuery(query);

foreach (var post in posts.ToList());
{
   Render(post["Body"].ToString());
}

So, LINQ to SQL is an abstraction -- we're abstracting away raw SQL and database access in favor of native language constructs and objects. I'd argue that Linq to SQL is a good abstraction. Heck, it's exactly what I asked for five years ago.

But even a good abstraction can break down in unexpected ways.

Consider this optimization, which is trivial in the old-school SQL blob code: instead of pulling down every single field in the post records, why not pull just the id number? Makes sense, if that's all I need. And it's faster -- much faster!

select top 48 * from Posts	827 ms
select top 48 Id from Posts	260 ms

Selecting all columns with the star (*) operator is expensive, and that's what LINQ to SQL always does by default. Yes, you can specify lazy loading, but not on a per-query basis. Normally, this is a non-issue, because selecting all columns for simple queries is not all that expensive. And you'd think pulling down 48 measly little post records would be squarely in the "not expensive" category!

So let's compare apples to apples. What if we got just the id numbers, then retrieved the full data for each row?

select top 48 Id from Posts	260 ms
select * from Posts where Id = 12345	3 ms

Now, retrieving 48 individual records one by one is sort of silly, becase you could easily construct a single where Id in (1,2,3..,47,48) query that would grab all 48 posts in one go. But even if we did it in this naive way, the total execution time is still a very reasonable (48 * 3 ms) + 260 ms = 404 ms. That is half the time of the standard select-star SQL emitted by LINQ to SQL!

An extra 400 milliseconds doesn't sound like much, but slow pages lose users. And why in the world would you perform a slow database query on every single page of your website when you don't have to?

It's tempting to blame Linq, but is Linq really at fault here? These seem like identical database operations to me:

1. Give me all columns of data for the top 48 posts.

or

1. Give me just the ids for the top 48 posts.
2. Retrieve all columns of data for each of those 48 ids.

So why in the wide, wide world of sports would one of these seemingly identical operations be twice as slow as the other?

The problem isn't Linq to SQL. The problem is that we're attempting to spackle a nice, clean abstraction over a database that is full of highly irregular and unusual real world behaviors. Databases that:

• may not have the right indexes
• may misinterpret your query and generate an inefficient query plan
• are trying to perform an operation that doesn't fit well in available memory
• are paging data from disks which might be busy at that particular moment
• might contain irregularly sized column datatypes

That's what's so frustrating. We can't just pretend all our data is formatted into neat, orderly data structures sitting there in memory, lined up in convenient little queues for us to reach out and casually scoop them up. As I've demonstrated, even trivial queries can have bizarre behavior and performance characteristics that are not at all clear.

To its credit, Linq to SQL is quite flexible: we can use strongly typed queries, or we can use SQL blob queries that we cast to the right object type. That flexibility is critical, because so much of our performance depends on these quirks of the database. We default to the built-in Linq language constructs, and drop down to hand-tuning ye olde SQL blobs where the performance traces tell us we need to.

Either way, it's clear that you've got to know what's happening in the database every step of the way to even begin understanding the performance of your application, much less troubleshoot it.

I think you could make a fairly solid case that Linq to SQL is, in fact, a leaky and failed abstraction. Exactly the kind of thing Joel was complaining about. But I'd also argue that virtually all good programming abstractions are failed abstractions. I don't think I've ever used one that didn't leak like a sieve. But I think that's an awfully architecture astronaut way of looking at things. Instead, let's ask ourselves a more pragmatic question:

Does this abstraction make our code at least a little easier to write? To understand? To troubleshoot? Are we better off with this abstraction than we were without it?

It's our job as modern programmers not to abandon abstractions due to these deficiencies, but to embrace the useful elements of them, to adapt the working parts and construct ever so slightly less leaky and broken abstractions over time. Like desperate citizens manning a dike in a category 5 storm, we programmers keep piling up these leaky abstractions, shoring up as best we can, desperately attempting to stay ahead of the endlessly rising waters of complexity.

As much as I may curse Linq to SQL as yet another failed abstraction, I'll continue to use it. Yes, I may end up soggy and irritable at times. But it sure as heck beats drowning.

[advertisement] Interested in agile? See how a world-leading software vendor is practicing agile.

]]> http://www.codinghorror.com/blog/archives/001281.html Tue, 30 Jun 2009 05:01:29 -0800 Jeff Atwood http://www.codinghorror.com/blog/archives/001280.html The original iPhone was for suckers hard-core gadget enthusiasts only. But as I predicted, 12 months later, the iPhone 3G rectified all the shortcomings of the first version. And now, with the iPhone 3GS, we've reached the mythical third version:

A computer industry adage is that Microsoft does not make a successful product until version 3. Its Windows operating system was not a big success until the third version was introduced in 1990 and, similarly, its Internet Explorer browsing software was lackluster until the third version.

The platform is now so compelling and polished that even I took the plunge. For context, this is the first Apple product I've owned since 1984. Literally.

I am largely ambivalent towards Apple, but it's impossible to be ambivalent about the iPhone -- and in particular, the latest and greatest iPhone 3GS. It is the Pentium to the 486 of the iPhone 3G. A landmark, genre-defining product, no longer a mere smartphone but an honest to God fully capable, no-compromises computer in the palm of your hand.

Here's how far I am willing to go: I believe the iPhone will ultimately be judged a more important product than the original Apple Macintosh.

Yes, I am dead serious. Just check back here in fifteen to twenty years to see if I was right. (Hint: I will be.)

There's always been a weird tension in Apple's computer designs, because they attempt to control every nuance of the entire experience from end to end. For the best Apple^tm experience, you run custom Apple^tm applications on artfully designed Apple^tm hardware dongles. That's fundamentally at odds with the classic hacker mentality that birthed the general purpose computer. You can see it in the wild west, anything goes Linux ecosystem. You can even see it in the Wintel axis of evil, where a million motley mixtures of hardware, software, and operating system variants are allowed to bloom, like little beige stickered flowers, for a price.

But a cell phone? It's a closed ecosystem, by definition, running on a proprietary network. By a status quo of incompetent megacorporations who wouldn't know user friendliness or good design if it ran up behind them and bit them in the rear end of their expensive, tailored suits. All those things that bugged me about Apple's computers are utter non-issues in the phone market. Proprietary handset? So is every other handset. Locked in to a single vendor? Everyone signs a multi-year contract. One company controlling your entire experience? That's how it's always been done. Nokia, Sony/Ericsson, Microsoft, RIM -- these guys clearly had no idea what they were in for when Apple set their sights on the cell phone market -- a market that is a nearly perfect match to Apple's strengths.

Apple was born to make a kick-ass phone. And with the lead they have, I predict they will dominate the market for years to come.

Consider all the myriad devices that the iPhone 3GS can sub for, and in some cases, outright replace:

• GPS
• Netbook (for casual web browsing and email)
• Gameboy
• Watch
• Camera
• MP4 Video Recorder
• MP3 player
• DVD player
• eBook reader

Oh yeah, and I heard you can make phone calls with it, too. Like any general purpose computer, it's a jack of all trades.

As impressive as the new hardware is, the software story is even bigger. If you're a software developer, the iPhone can become a career changing device, all thanks to one little teeny-tiny icon on the iPhone home screen:

The App Store makes it brainlessly easy to install, upgrade, and purchase new applications. But more importantly, any software developer -- at the mild entry cost of owning a Mac, and signing up for the $99 iPhone Developer Program -- can build an app and sell it to the worldwide audience of iPhone users. Apple makes this stuff look easy, when historically it has been anything but. How many successful garage developers do you know for Nintendo DS? For the Motorola Razr? For Palm? For Windows Mobile?

Apple has never been particularly great at supporting software developers, but I have to give them their due: with the iPhone developer program, they've changed the game. Nowhere is this more evident than in software pricing. I went on a software buying spree when I picked up my iPhone 3GS, ending up with almost three pages of new applications from the App Store. I was a little worried that I might rack up a substantial bill, but how can I resist when cool stuff like ports of the classic Amiga Pinball Dreams are available, or the historic Guru Meditation? The list of useful (and useless) apps is almost endless, and growing every day.

My total bill for 3 screens worth of great iPhone software applications? About fifty bucks. I've paid more than that for Xbox 360 games I ended up playing for a total of maybe three hours! About half of the apps were free, and the rest were a few bucks. I think the most I paid was $9.99, and that was for an entire library. What's revolutionary here isn't just the development ecosystem, but the economics that support it, too. At these crazy low prices, why not fill your phone with cool and useful apps? You might wonder if developers can really make a living selling apps that only cost 99 cents. Sure you can, if you sell hundreds of thousands of copies:

Freeverse, one of the leading developers and publishers of iPhone games, sold the millionth copy of its Flick Fishing game over the weekend, making Flick Fishing the first paid application to reach the one million download milestone. Flick Fishing, which costs 99 cents, allows iPhone and iPod touch users to take a virtual fishing trip with the flick of a wrist. The game uses the iPhone's accelerometer to recreate a casting motion, then a combination of bait choice and fishing skill helps players land the big fish.

Preliminary weekly reports for the period from 23 March to 19 April indicate that Flight Control sold a total of 587,485 units during this time. We estimate total sales are now over 700,000 units, with the bulk of sales occurring in a 3 week period. Flight Control

That's an honorable way to get rich programming, and a nice business alternative to the dog-eat-dog world of advertising subsidized apps.

I love nothing more than supporting my fellow software developers by voting with my wallet. it does my heart good to see so many indie and garage developers making it big on the iPhone. (Also, I'm a sucker for physics games, and there are a bunch of great ones available in the App Store). I'm more than happy to pitch in a few bucks every month for a great new iPhone app.

If this has all come across as too rah-rah, too uncritical a view of the iPhone, I apologize. There are certainly things to be critical about, such as the App Store's weird enforcement policies, the lack of support for emulators, or Flash, or anything else that might somehow undermine the platform as decided in some paranoid, secretive Apple back room. Not that we'd ever hear about it.

I didn't write this to kiss Apple's ass. I wrote this because I truly feel that the iPhone is a key inflection point in software development. We will look back on this as the time when "software" stopped being something that geeks buy (or worse, bootleg), and started being something that everyone buys, every day. You'd have to be a jaded developer indeed not to find something magical and transformative in this formula, and although others will clearly follow, the iPhone is leading the way.

"There's an app for that." Kudos, Apple. From the bottom of my hoary old software developer heart.

[advertisement] Interested in agile? See how a world-leading software vendor is practicing agile.

]]> http://www.codinghorror.com/blog/archives/001280.html Wed, 24 Jun 2009 23:59:59 -0800 Jeff Atwood http://www.codinghorror.com/blog/archives/001279.html In My Scaling Hero, I described the amazing scaling story of plentyoffish.com. It's impressive by any measure, but also particularly relevant to us because we're on the Microsoft stack, too. I was intrigued when Markus posted this recent update:

Last monday we upgraded our core database server after a power outage knocked the site offline. I haven't touched this machine since 2005 so it was a major undertaking to do it last minute. We upgraded from a machine with 64 GB of ram and 8 CPUs to a HP ProLiant DL785 with 512 GB of ram and 32 CPUs ...

The HP ProLiant DL785 G5 starts at $16,999 -- and that's barebones, with nothing inside. Fully configured, as Markus describes, it's kind of a monster:

• 7U size (a typical server is 2U, and mainstream servers are often 1U)
• 8 CPU sockets
• 64 memory sockets
• 16 drive bays
• 11 expansion slots
• 6 power supplies

It's unclear if they bought it pre-configured, or added the disks, CPUs, and memory themselves. The most expensive configuration shown on the HP website is $37,398 and that includes only 4 processors, no drives, and a paltry 32 GB memory. When topped out with ultra-expensive 8 GB memory DIMMs, 8 high end Opterons, 10,000 RPM hard drives, and everything else -- by my estimates, it probably cost closer to $100,000. That might even be a lowball number, considering that the DL785 submitted to the TPC benchmark website (pdf) had a "system cost" of $186,700. And that machine only had 256 GB of RAM. (But, to be fair, that total included another major storage array, and a bunch of software.)

At any rate, let's assume $100,000 is a reasonable ballpark for the monster server Markus purchased. It is the very definition of scaling up -- a seriously big iron single server.

But what if you scaled out, instead -- Hadoop or MapReduce style, across lots and lots of inexpensive servers? After some initial configuration bumps, I've been happy with the inexpensive Lenovo ThinkServer RS110 servers we use. They're no match for that DL785 -- but they aren't exactly chopped liver, either:

Lenovo ThinkServer RS110 barebones	$600
8 GB RAM	$100
2 x eBay drive brackets	$50
2 x 500 GB SATA hard drives, mirrored	$100
Intel Xeon X3360 2.83 GHz quad-core CPU	$300

Grand total of $1,150 per server. Plus another 10 percent for tax, shipping, and so forth. I replace the bundled CPU and memory that the server ships with, and then resell the salvaged parts on eBay for about $100 -- so let's call the total price per server $1,200.

Now, assuming a fixed spend of $100,000, we could build 83 of those 1U servers. Let's compare what we end up with for our money:

	Scaling Up	Scaling Out
CPUs	32	332
RAM	512 GB	664 GB
Disk	4 TB	40.5 TB

Now which approach makes more sense?

(These numbers are a bit skewed because that DL785 is at the absolute extreme end of the big iron spectrum. You pay a hefty premium for fully maxxing out. It is possible to build a slightly less powerful server with far better bang for the buck.)

But there's something else to consider: software licensing.

	Scaling Up	Scaling Out
OS	$2,310	$33,200*
SQL	$8,318	$49,800*

(If you're using all open source software, then of course these costs will be very close to zero. We're assuming a Microsoft shop here, with the necessary licenses for Windows Server 2008 and SQL Server 2008.)

Now which approach makes more sense?

What about the power costs? Electricity and rack space isn't free.

	Scaling Up	Scaling Out
Peak Watts	1,200w	16,600w
Power Cost / Year	$1,577	$21,815

Now which approach makes more sense?

I'm not picking favorites. This is presented as food for thought. There are at least a dozen other factors you'd want to consider depending on the particulars of your situation. Scaling up and scaling out are both viable solutions, depending on what problem you're trying to solve, and what resources (financial, software, and otherwise) you have at hand.

That said, I think it's fair to conclude that scaling out is only frictionless when you use open source software. Otherwise, you're in a bit of a conundrum: scaling up means paying less for licenses and a lot more for hardware, while scaling out means paying less for the hardware, and a whole lot more for licenses.

* I have no idea if these are the right prices for Windows Server 2008 and SQL Server 2008, because reading about the licensing models makes my brain hurt. If anything, it could be substantially more.

[advertisement] Interested in agile? See how a world-leading software vendor is practicing agile.

]]> http://www.codinghorror.com/blog/archives/001279.html Tue, 23 Jun 2009 23:59:59 -0800 Jeff Atwood http://www.codinghorror.com/blog/archives/001278.html Remember The Problem of the Unfinished Game? And the almost 2,500 comments those two posts generated? I know, I like to pretend it didn't happen, either. Some objected to the way I asked the question, but it was a simple question asked in simple language. I think what they're really objecting to is how unintuitive the answer is.

Which reminds me of another question that you've probably heard of:

Suppose the contestants on a game show are given the choice of three doors: behind one door is a car; behind the others, goats. After a contestant picks a door, the host, who knows what's behind all the doors, opens one of the unchosen doors, which reveals a goat. He then asks the contestant, "Do you want to switch doors?"

Should the contestant switch doors?

This is, of course, the Monty Hall problem. It's been covered to death, and quite well I might add, by dozens of writers who are far more talented than I.

What's interesting about this problem, to me at least, is not the solution, but the vehemence with which people react to the solution -- as described in The Drunkard's Walk: How Randomness Rules Our Lives.

It appears to be a pretty silly question. Two doors are available -- open one and you win; open the other and you lose -- so it seems self-evident that whether you change your choice or not, your chances of winning are 50/50. What could be simpler? The thing is, Marilyn said in her column that it is better to switch.

Despite the public's much-heralded lethargy when it comes to mathematical issues, Marilyn's readers reacted as if she'd advocated ceding California back to Mexico. Her denial of the obvious brought her an avalanche of mail, 10,000 letters by her estimate. If you ask the American people whether they agree that plants create the oxygen in the air, light travels faster than sound, or you cannot make radioactive milk by boiling it, you will get double-digit disagreement in each case (13 percent, 24 percent, and 35 percent, respectively). But on this issue, Americans were united: Ninety-two percent agreed Marilyn was wrong.

Perhaps the public can be forgiven their ignorance, but what of the experts? Surprisingly, the mathematicians fare little better.

Almost 1,000 Ph.D.s wrote in, many of them math professors, who seemed especially irate. "You blew it," wrote a mathematician from George Mason University. From Dickinson State University came this: "I am in shock that after being corrected by at least three mathematicians, you still do not see your mistake." From Georgetown: "How many irate mathematicians are needed to change your mind?" And someone from the U.S. Army Research Institute remarked, "If all those Ph.D.s are wrong the country would be in serious trouble." Responses continued in such great numbers and for such a long time that after devoting quite a bit of column space to the issue, Marilyn decided she whould no longer address it.

The army PhD who wrote in may have been correct that if all those PhDs were wrong, it would be a sign of trouble. But Marilyn was correct. When told of this, Paul Erdos, one of the leading mathematicians of the 20th century, said, "That's impossible." Then, when presented with a formal mathematical proof of the correct answer, he still didn't believe it and grew angry. Only after a colleague arranged for a computer simulation in which Erdos watched hundreds of trials that came out 2-to-1 in favor of switching did Erdos concede that he was wrong.

You may recognize Paul Erdos from a particularly obscure XKCD cartoon last week. So if you feel like an idiot because you couldn't figure out the Monty Hall problem, take heart. The problem is so unintuitive one of the most notable mathematicians of the last century couldn't wrap his head around it. That's ... well, that's amazing.

How can something that seems so obvious be so wrong? Apparently our brains are not wired to do these sorts of probability problems very well. Personally, I found the text of Jeffrey Rosenthal's Monty Hall, Monty Fall, Monty Crawl (pdf) to be the most illuminating, because it asks us to consider some related possibilities, and how they might affect the outcome:

Monty Fall Problem: In this variant, once you have selected one of the three doors, the host slips on a banana peel and accidentally pushes open another door, which just happens not to contain the car. Now what are the probabilities that you will win, either by sticking with your original door, or switching doors?

Monty Crawl Problem: Once you have selected one of the three doors, the host then reveals one non-selected door which does not contain the car. However, the host is very tired, and crawls from his position (near Door #1) to the door he is to open. In particular, if he has a choice of doors to open, then he opens the smallest number available door. (For example, if you selected Door #1 and the car was indeed behind Door #1, then the host would always open Door #2, never Door #3.) Now what are the probabilities that you will win the car if you stick versus if you switch?

Paul Erdos was brilliant, but even he realized his own limits when presented with the highly unintuitive Monty Hall problem. For his epitaph, he suggested, in his native Hungarian, "Végre nem butulok tovább". This translates into English as "I've finally stopped getting dumber."

If only the rest of us could be so lucky.

[advertisement] Interested in agile? See how a world-leading software vendor is practicing agile.

]]> http://www.codinghorror.com/blog/archives/001278.html Sun, 21 Jun 2009 23:59:59 -0800 Jeff Atwood http://www.codinghorror.com/blog/archives/001277.html In my previous post, Url Shorteners: Destroying the Web Since 2002, I mentioned that one of the "features" of the new generation of URL shortening services is to frame the target content.

Digg is one of the most popular sites to implement this strategy. Here's how it works. If you're logged in to Digg, every target link you click from Digg is a shortened URL of their own creation. If I click through to a Stack Overflow article someone else has "Dugg", I'm sent to this link.

http://digg.com/d1tBya

For logged in users, every outgoing Digg link is framed inside the "DiggBar". It's a way of dragging the Digg experience with you wherever you go -- while you're reading the target article, you can vote it up, see related articles, share, and so forth. And if you share this shortened URL with other users, they'll get the same behavior, provided they also hold a Digg login cookie.

At this point you're probably expecting me to rant about how evil the DiggBar is, and how it, too, is destroying the web, etcetera, etcetera, so on, and so forth. But I can't muster the indignant rage. I can give you, at best, ambivalence. Here's why:

1. The DiggBar is not served to the vast majority of anonymous users, but only to users who have opted in to the Digg experience by signing up.
2. The new rel="canonical" directive is used on target links so search engines can tell which links are the "real", authoritative links to the content. They won't be confused or have search engine juice diluted by Digg's shortened URLs. At least that's the theory, anyway.
3. No Digg ads are served via the DiggBar, so the framed content is not "wrapped" in ads.
4. I believe Digg users themselves can opt out of DiggBar via a preferences setting.

In short, Digg isn't the problem. But even if they were -- if you don't want to be framed by the DiggBar, or any other website for that matter, you could put so-called "frame-busting" JavaScript in your pages.

if (parent.frames.length > 0) {
    top.location.replace(document.location);
}

Problem solved! This code (or the many frame-busting variants thereof) does work on the DiggBar. But not every framing site is as reputable as Digg. What happens when we put on our hypothetical black hats and start designing for evil?

I'll tell you what happens. This happens.

   var prevent_bust = 0  
   window.onbeforeunload = function() { prevent_bust++ }  
   setInterval(function() {  
     if (prevent_bust > 0) {  
       prevent_bust -= 2  
       window.top.location = 'http://server-which-responds-with-204.com'  
     }  
   }, 1)  

On most browsers a 204 (No Content) HTTP response will do nothing, meaning it will leave you on the current page. But the request attempt will override the previous frame busting attempt, rendering it useless. If the server responds quickly this will be almost invisible to the user.

When life serves you lemons, make a lemon cannon. Produce frame-busting-busting JavaScript. This code does the following:

• increments a counter every time the browser attempts to navigate away from the current page, via the window.onbeforeonload event handler
• sets up a timer that fires every millisecond via setInterval(), and if it sees the counter incremented, changes the current location to an URL of the attacker's control
• that URL serves up a page with HTTP status code 204, which does not cause the browser to navigate anywhere

Net effect: frame-busting busted. Which might naturally lead you to wonder -- hey buster, can you bust the frame-busting buster? And, if so, where does it end?

In the 1998 movie, The Big Hit, the protagonists kidnap the daughter of an extremely wealthy Japanese businessman. When they call to deliver the ransom notice, they turn to Gump who employs a brand name Trace Buster to prevent police from tracing the call.

Unbeknownst to Gump, the father has a Trace-Buster-Buster at his disposal. This in turn triggers Gump to use his Trace-Buster-Buster-Buster in an ever escalating battle to evade detection.

What's really scary is that near as I can tell, there is no solution. Due to cross-domain JavaScript security restrictions, it is almost impossible for the framed site to block or interfere with the parent page's evil JavaScript that is intentionally and aggressively blocking the framebusting.

If an evil website decides it's going to frame your website, you will be framed. Period. Frame-busting is nothing more than a false sense of security; it doesn't work. This was a disturbing revelation to me, because framing is the first step on the road to clickjacking:

A clickjacked page tricks a user into performing undesired actions by clicking on a concealed link. On a clickjacked page, the attackers show a set of dummy buttons, then load another page over it in a transparent layer. The users think that they are clicking the visible buttons, while they are actually performing actions on the hidden page. The hidden page may be an authentic page, and therefore the attackers can trick users into performing actions which the users never intended to do and there is no way of tracing such actions later, as the user was genuinely authenticated on the other page.

For example, a user might play a game in which they have to click on some buttons, but another authentic page like a web mail site from a popular service is loaded in a hidden iframe on top of the game. The iframe will load only if the user has saved the password for its respective site. The buttons in the game are placed such that their positions coincide exactly with the select all mail button and then the delete mail button. The consequence is that the user unknowingly deleted all the mail in their folder while playing a simple game. Other known exploits have been tricking users to enable their webcam and microphone through flash (which has since been corrected by Adobe), tricking users to make their social networking profile information public, making users follow someone on Twitter, etc.

I've fallen prey to a mild clickjacking exploit on Twitter myself! It really does happen -- and it's not hard to do.

Yes, Digg frames ethically, so your frame-busting of the DiggBar will appear to work. But if the framing site is evil, good luck. When faced with a determined, skilled adversary that wants to frame your contnet, all bets are off. I don't think it's possible to escape. So consider this a wakeup call: you should build clickjacking countermeasures as if your website could be framed at any time.

I was a skeptic. I didn't want to believe it either. But once shown the exploits on our own site -- fortunately, by a white hat security expert -- I lived to regret that. Don't let frame-busting code lull you into a false sense of security, too.

[advertisement] Interested in agile? See how a world-leading software vendor is practicing agile.

]]> http://www.codinghorror.com/blog/archives/001277.html Wed, 17 Jun 2009 23:59:59 -0800 Jeff Atwood http://www.codinghorror.com/blog/archives/001276.html Is anyone else as sick as I am of all the mainstream news coverage on Twitter? Don't get me wrong, I'm a Twitter fan, and I've been a user since 2006. To me, it's a form of public instant messaging -- yet another way to maximize the value of my keystrokes. Still, I'm a little perplexed as to the media's near-obsession with the service. If a day goes by now without the New York Times or CNN mentioning Twitter in some way, I become concerned. Am I really getting all the news? Or just the stupid, too long, non-140-character version of the news?

I guess I should be pleased that I was a (relatively) early adopter and advocate of software that has achieved the rarest of feats in the software industry -- critical mass. Adoption by the proverbial "average user". Whatever you may think of Twitter, consider this: as a software developer, you'll be fortunate to build one project that achieves critical mass in your entire life. And even then, only if you are a very, very lucky programmer: in the right place, at the right time, with the right idea, working with the right people. Most of us never get there. I don't think I will.

There is one side effect of this unprecedented popularity, though, that I definitely wouldn't have predicted: the mainstreaming of URL shortening services. You can barely use Twitter without being forced into near-mastery of URL shortening. For example, this is the super-secret, patented formula I often use when composing my Twitter messages:

"brief summary or opinion" [link for more detail]

Twitter only allows 140 characters in each status update. Some might view this as a limitation, but I consider it Twitter's best feature. I am all for enforced brevity. Maybe that's due to the pain of living through a lifetime of emfail. But depending on the size of the comment and the URL (and some URLs can be ridiculously long), I can't quite fit everything in there without sounding like an SMS-addled teenage girl. This is where URL shortening comes in.

Now, I know what you're thinking. You're a clever programmer. You could implement some kind of fancy jQuery callback to shorten the URL, and replace the longer URL with the shorter URL right there in the text as the user pauses in typing. But you don't even have to be that clever; most of the URL shortening services (that aren't in their infancy) deliver a rather predictable size for the URLs they return. You could simply estimate the size of the URL post-shortening -- maybe adding 1 character as a fudge factor for safety -- and allow the update.

Twitter, I can assure you, is far more brain damaged than you can possibly imagine. It will indeed shorten URLs that fit in the 140 character limit (whoopee!), but it does nothing for URLs that don't fit -- it will not allow you to submit the message. All part of its endearing charm.

Lame, yes, but it means that the typical, mainstream browser-based Twitter user is forced to become proficient with URL shortening services. Due to the increased exposure they've enjoyed through Twitter's meteoric rise to fame, the number of URL shortening services has exploded, and rapidly evolved -- they're no longer viewed as utility services to make URLs more convenient, but a way to subjugate, control, and perhaps worst of all, "monetize" the web experience.

This is dangerous territory we're veering into now, as Joshua Schachter explains.

So there are clear benefits for both the service (low cost of entry, potentially easy profit) and the linker (the quick rush of popularity). But URL shorteners are bad for the rest of us.

The worst problem is that shortening services add another layer of indirection to an already creaky system. A regular hyperlink implicates a browser, its DNS resolver, the publisher's DNS server, and the publisher's website. With a shortening service, you're adding something that acts like a third DNS resolver, except one that is assembled out of unvetted PHP and MySQL, without the benevolent oversight of luminaries like Dan Kaminsky and St. Postel.

The web is little more than a maze of hyperlinks, and if you can insert yourself as an intermediary in that maze, you can transform or undermine the experience in fundamental ways. Consider the disturbing directions newer URL shortening services are taking:

• NotifyURL sends an email when the link is first visited.
• SnipURL introduces social bookmarking features such as usernames and RSS feeds.
• DwarfURL generates statistics.
• Adjix, XR.com and Linkbee are ad-supported models of URL shorteners that share the revenue with their users.
• bit.ly offers gratis click-through statistics and charts.
• Digg offers a shortened URL which includes not just the target URL, but an iframed version that includes a set of Digg-related controls called the Digg bar.
• Doiop allows the shortening to be selected by the user, and Unicode can be used to achieve really short URLs.

Believe it: the humble hyperlink, thanks to pervasive URL shortening, can now be wielded as a weapon. The internet is the house that PageRank built, and it's all predicated on hyperlinks. Once you start making every link your special flavor of "shortened" link, framing the target content -- heck, maybe wrapping it in a few ads for good measure -- you've completely turned that system on its head.

What's aggravating to me is that the current situation is completely accidental. If Twitter had provided a sane way to link a single word, none of these weaselly URL shortening clones would have reared their ugly heads at all. Consider how simple it is to decouple the hyperlink from the display text in, say, phpBB, or Markdown, or even good old HTML markup itself:

<a href="http://example.com">foo</a>
[url=http://example.com]foo[/url]
[foo](http://example.com)

Every tiny URL is another baby step towards destroying the web as we know it. Which is exactly what you'd want to do if you're attempting to build a business on top of the ruins. Personally, I'd prefer to see the big, objective search engines who naturally sit at the center of the web offer their own URL shortening services. Who better to generate short hashes of every possible URL than the companies who already have cached copies of every URL on the internet, anyway?

[advertisement] Interested in agile? See how a world-leading software vendor is practicing agile.

]]> http://www.codinghorror.com/blog/archives/001276.html Tue, 16 Jun 2009 03:08:15 -0800 Jeff Atwood http://www.codinghorror.com/blog/archives/001275.html In Why Isn't My Encryption.. Encrypting? we learned that your encryption is only as good as your understanding of the encryption code. And that the best encryption of all is no encryption, because you kept everything on the server, away from the prying eyes of the client.

In The Bathroom Wall of Code we learned the potential danger of copy-pasting code from the internet, and the continued importance of regular peer review for every line of code that enters your codebase, from whatever source.

I didn't anticipate this series becoming a trilogy, but apparently it has, because Thomas Ptacek of Matsano Security wrote a long blog entry about it. A blog entry masquerading as an overly dramatic college screenplay, but still. These guys, unlike us, are real security experts, so it's worth reading.

But you don't have to read that screenplay, because I'm going to reveal the twist in the final act right here.

1. The root problem wasn't failing to understand the encryption.
2. The root problem wasn't copy and pasting code from the internet.
3. The root problem wasn't failing to peer review the code.

Mr. Ptacek is absolutely right. The root problem was that we were working at the wrong layer of abstraction.

Rather than construct code from the low-level cryptography primitives provided in .NET, we should have used a library to handle our encryption needs. I'm reminded of a common Stack Overflow joke:

Q: How do I write this in JavaScript?

A: You don't. You use JQuery.

You can save a tremendous amount of time and effort by using the browser-independent framework that JQuery has spent untold man-hours testing, debugging, and proving in the field. While there's nothing wrong with writing JavaScript, why not speed your development time by writing to the library instead? As I've always said, don't reinvent the wheel, unless you plan on learning more about wheels.

Abstractions are important. You could view most of computer programming history as slowly, painfully clawing our way up the evolutionary tree of abstraction -- from assembly language, to C, to Java, to JavaScript, all the way up to JQuery, where the air starts to get pretty darn thin. We've already layered an operating system, web browser, and interpreted scripting language on top of each other to get to this point. It's a testament to the power of abstraction that any of it works at all.

Getting back to specifics: how can you stop programmers from working at the wrong layer of abstraction? One solution would be to disallow the .NET encryption primitives entirely. This is akin to Steve Gibson's holy crusade against raw socket programming in Windows XP. That's one way to do it, I suppose. But putting roadblocks in front of programmers is tantamount to a challenge; why not offer them more attractive alternatives, instead?

Hiding the low-level encryption primitives feels like a temporary solution. That said, I'd strongly recommend marking some of the older encryption methods as deprecated, so programmers who do stumble down some dusty old code path at least have some warning sign that they're using an algorithm with a lot of known vulnerabilities. I'm envisioning a Clippy that pops up with something like:

"Hey! It looks like you're using a method of encryption that's widely regarded as insecure by security experts! Would you like to see alternatives?"

One of those alternatives would be a full-blown library, perhaps something like Bouncy Castle, or Keyczar, or cryptlib. What could be easier than a EncryptStringForBrowser() method which has security and tamper-resistance built in, that's part of a proven, domain-expert-tested set of code that thousands if not millions of developers already rely on?

Using encryption libraries doesn't mean that crucial encryption mistakes will magically disappear overnight. But these libraries, because they force developers to work at a higher level of abstraction, do make it harder to misuse cryptography. And perhaps more importantly, usability improvements to the library can be better handled by the specialists who created the library, rather than the generalists working on the .NET framework itself.

So the next time you set out to write code -- not just encryption code, any code -- ask yourself: am I working at the right level of abstraction?

[advertisement] Interested in agile? See how a world-leading software vendor is practicing agile.

]]> http://www.codinghorror.com/blog/archives/001275.html Thu, 11 Jun 2009 23:59:59 -0800 Jeff Atwood http://www.codinghorror.com/blog/archives/001274.html If you've followed my blog for any length of time, you know that I am a total regular expression fanboy. It's almost embarrassing how much I love the damn things. I'm pretty sure my teammates roll their eyes every time they see yet another class I've touched that has using System.Text.RegularExpressions at the top. You might as well rename it to JeffHasBeenHere.

I say that because I end up writing a lot of string handling code, even when people tell me I shouldn't. Now, I only advocate responsible and judicious use of regular expressions when you happen to be dealing with strings. In the wrong hands, regular expressions can be dangerous. You might end up wondering if Q*Bert just vomited all over your source code. Or you might be programming in Perl. Is there any difference? (instant rimshot)

But I digress. Although I love regex, I've never been a fan of the classic regular expression reference book, Friedl's Mastering Regular Expressions. I found it dry, a bit academic, and lacking in practical real world examples. It just didn't speak to me as a working programmer in the way that regular expressions themselves did, and I found that disappointing.

That's why I was so excited to discover that two of the gnarliest regex gurus I knew -- Jan Goyvaerts (author of RegexBuddy and regular-expressions.info) and Steven Levithan (author of XRegExp and RegexPal) -- were putting their heads together to create a regular expression reference for the rest of us. I immediately pre-ordered it sight unseen.

That book is Regular Expressions Cookbook. It arrived a few days ago, and although my expectations were high, I think this book has exceeded even the loftiest expectations I had. It is outstanding.

What I love about this book is two things:

1. It's filled with practical, real world examples of RegEx use. At every step of the way, from beginner to master level, you're building regular expressions that are actually useful in the wild, and not just abstract, obtuse academic exercises in solving string matching puzzles.
   
   
2. It covers all the common gotchas that you inevitably run into when you start building non-trivial regular expressions. Things like the sometimes massive (and painful) differences between regex libraries in various languages, subtle regex flavor quirks, catastrophic backtracking, unicode support, and so forth. These are all presented in context of the solutions, exactly as you'd encounter them in real programming. I know because I have the scars to prove it.

Regular Expressions Cookbook manages to be simultaneously accessible and almost ridiculously comprehensive. I consider myself a fairly advanced regex user and about 50 pages in I've already had three big "oh, wow, I didn't realize that" moments. In my mind, at least, this completely replaces the Friedl book as the go-to reference for programmers of any skill level or background who seek regular expression enlightenment.

Needless to say, recommended.

[advertisement] Interested in agile? See how a world-leading software vendor is practicing agile.

]]> http://www.codinghorror.com/blog/archives/001274.html Mon, 08 Jun 2009 23:59:59 -0800 Jeff Atwood http://www.codinghorror.com/blog/archives/001273.html Unix turns 40: The past, present and future of a revolutionary OS is fascinating reading.

Forty years ago this summer, a programmer sat down and knocked out in one month what would become one of the most important pieces of software ever created.

In August 1969, Ken Thompson (pictured at left), a programmer at AT&T subsidiary Bell Laboratories, saw the month-long departure of his wife and young son as an opportunity to put his ideas for a new operating system into practice. He wrote the first version of Unix in assembly language for a wimpy Digital Equipment Corp. (DEC) PDP-7 minicomputer, spending one week each on the operating system, a shell, an editor and an assembler.

The article is accompanied by a graph from wikipedia, illustrating the lineage of the Unix family.

To me, Unix has become synonymous with Linux, and the open source movement in general. The last *nixes standing shake out as follows:

Open Source	Mixed / Shared Source	Closed Source
Minix Linux FreeBSD NetBSD OpenBSD OpenSolaris	Mac OS X	AIX OpenServer HP/UX

I didn't realize there were that many closed source Unix variants still surviving in the wild. It's also odd how OS X brings us full circle with the original Unics and BSD licensing. If it's lonely in the "Closed" column, imagine the existential angst of being the only vendor in the "Mixed / Shared Source" column. (NB: I think the currently tiny category Apple occupies represents the future of commercial software, but that's a topic for another blog post.)

I've been primarily a Windows developer since the early 90s, but over time, I've developed a grudging respect for Unix. I think Michael Feathers summarized it best:

There's something deep in software development that not everyone gets but the people at Bell Labs did. It's the undercurrent of "the New Jersey Style", "Worse is Better", and "the Unix philosophy" - and it's not just a feature of Bell Labs software either. You see it in the original Ethernet specification where packet collision was considered normal.. and the same sort of idea is deep in the internet protocol. It's deep awareness of design ramification - a willingness to live with a little less to avoid the bigger mess and a willingness to see elegance in the real rather than the vision.

I find this to be deeply and profoundly true in everything I've ever worked on as a programmer, and to the extent that Unix reflects these philosophies, it is undeniably on the right path. Unlike Rich Skrenta, I didn't grow up as a Unix developer, so I have come late in life to this appreciation. Joel Spolsky's take on the Unix / Windows divide, after reading The Art of UNIX Programming, is this:

What are the cultural differences between Unix and Windows programmers? There are many details and subtleties, but for the most part it comes down to one thing: Unix culture values code which is useful to other programmers, while Windows culture values code which is useful to non-programmers. This is, of course, a major simplification, but really, that's the big difference: are we programming for programmers or end users? Everything else is commentary.

So on one side, you have hundreds of command line applications, built in wildly different styles, with thousands of arcane command line parameters, all of which can be flexibly combined together to accomplish almost anything. And on the other side, you have the windows registry and MFC.

Sometimes, you just can't win.

So, yes, I'm a fan of Unix. And I'm also a fan of Windows. I think it's worth studying what both are getting right and wrong, because as a programmer, I'm a fan of whatever the heck works.

[advertisement] Interested in agile? See how a world-leading software vendor is practicing agile.

]]> http://www.codinghorror.com/blog/archives/001273.html Sun, 07 Jun 2009 23:59:59 -0800 Jeff Atwood http://www.codinghorror.com/blog/archives/001272.html Everybody loves BitTorrent. And rightfully so.

With BitTorrent, you also start by placing your large file on a central server. But once the downloading begins, something magical happens: as clients download the file, they share whatever parts of the file they have with each other. Clients can opportunistically connect with any other client to obtain multiple parts of the file at once. And it scales perfectly: as file size and audience size increases, the bandwidth of the BitTorrent distribution network also increases. Your server does less and less work with each connected client. It's an elegant, egalitarian way of sharing large files with large audiences.

BitTorrent radically shifts the economics of distribution. It's one of the most miraculous ideas ever conceived on the internet. As far as I'm concerned, there should be a Nobel prize for computing, and the inventor of BitTorrent should be its first recipient.

I've been a happy consumer of files distributed via BitTorrent for years; it was only natural that I would turn to BitTorrent to distribute our cc-wiki licensed Stack Overflow data. I figured serving a several-hundred megabyte file with BitTorrent wouldn't be much harder than downloading one. Boy, was I ever wrong. Sharing files with BitTorrent is way more complicated than downloading them! After two frustrating hours, I finally came up with a relatively straightforward way to share a file via BitTorrent, and in the interests of saving future readers a little time, I'm documenting it here.

Now, I'm going to show you an easy way, but it isn't technically the easiest way. The easiest way is to let someone else do the sharing for you. If you own content that you want to share, LegalTorrents is the obvious choice:

LegalTorrents^tm is an online digital media community.

We discover and distribute high quality open-license (Creative Commons) digital media and art, and provide support to Content Creators. We host creative content in its entirety, ensure fast, reliable downloads, and enable users to directly sponsor Content Creators and their work.

We distribute content with the full permission of the rights holders and use the peer-2-peer file-sharing technology called Bittorrent.

The site is still in beta, but signup is a snap, because they support OpenID! I encourage anyone interested to check it out. If nothing else, get the furtive thrill of actually downloading legal content through BitTorrent for once! Yes, it can happen. Shocking, I know. Don't worry, you crazy kids can get right back to your regular non-copyright-respecting torrenting ways immediately afterwards.

Anyway, you can't start sharing files on LegalTorrents without some kind of special email-us-please permission, and I was in a hurry. I wanted to share files via BitTorrent right now. I did, and you can too! But you'll need a few things first:

1. A copy of uTorrent (it's free!)
   
   
2. Your external IP address; if you don't know what it is, use http://www.whatismyipaddress.com to find out.
   
   
3. The uTorrent listen port. This is under Options | Preferences | Connection. This is typically set randomly every time uTorrent starts, so you may want to specify a more memorable value here.
   
   
4. You must have port forwarding properly configured so the outside world can get to your IP address and the port specified above. A full discussion of how to do this is outside the scope of this post, but it usually starts with your firewall settings and/or router configuration. uTorrent has a fairly nice help page at Options | Speed Guide that's a good start; just click the Test if port is forwarded properly button on that dialog to begin.

Here's where I hit a major roadblock: to share files via BitTorrent, you need a tracker.

A BitTorrent tracker is a server that assists in the communication between peers using the BitTorrent protocol. It is also, in the absence of extensions to the original protocol, the only major critical point, as clients are required to communicate with the tracker to initiate downloads. Clients that have already begun downloading also communicate with the tracker periodically to negotiate with newer peers and provide statistics; however, after the initial reception of peer data, peer communication can continue without a tracker.

Without a tracker, you're sort of hosed, as clients will never be able to find your file, much less each other. Unfortunately, most of the freely open, public trackers out there are sort of.. disreputable. And the LegalTorrents tracker won't track files unless they are on its creator whitelist, which involves that manual sign-up process. You've got precious few legit options for tracking, unless you're willing to take a trip to the wrong side of town, and associate yourself and your files with that kind of .. neighborhood. I wasn't.

Fortunately, uTorrent has a solution: you can become your own tracker!

1. in uTorrent, go to Options | Preferences | Advanced.
2. Scroll down to bt.enable_tracker and set it to True
3. Restart uTorrent.

Now, let's create the torrent for the file we want to host, which will point to our newly created tracker.

1. In uTorrent, click the Create New Torrent button.
2. Select the file or directory you want to share.
3. Enter your tracker in this format: http://my-ip-address:my-port/announce
4. That's it! Click Create and save the new *.torrent file you've created.

Now go forth and share your *.torrent file with the world. Share it with anyone and everyone! The more the merrier! Any client that opens your *.torrent file will attempt to connect to your tracker, download your file, and share it with other downloading clients in classic BitTorrent stylee. Pat yourself on the back; you just shared a file with the world using the transformative distribution power of BitTorrent!

But you do have to keep uTorrent running as a desktop application all the time, which is sort of a bummer. What if you wanted to share your file on a server, or via a silent background process? No problem. It's just a few more steps:

1. Enable the uTorrent web interface under Preferences, Web UI. Note that the URL for it is, by default, http://my-ip-address:my-port/gui/, and it requires a username and password to be set here.
   
   
2. Obtain a copy of the user-defined service utilities, srvany.exe and instsrv.exe. Copy them to the same folder as uTorrent.exe.
   
   
3. Issue this command to make uTorrent run as a service:
   
   

   instsrv uTorrent "C:\uTorrent\srvany.exe"

   
   
4. Enter this registry file to set the path for the service named "uTorrent" you just created in the previous step:
   
   

   Windows Registry Editor Version 5.00
   
   [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\uTorrent\Parameters]
   "Application"="C:\\uTorrent\\uTorrent.exe"
   

   
   
5. In Control Panel, Services, set the account that the uTorrent service will run under. Note that you must use the same account that you set uTorrent options with if you want the service to respect those settings, so plan accordingly.
   
   
   
   
6. Start the uTorrent service.

(Obviously, replace the above paths with the actual paths that you installed uTorrent to.)

Bam -- you're sharing files with the world using BitTorrent, even when you're not logged in. You can control everything remotely, too, by navigating your browser to the WebUI URL.

Like so many things in Windows, it ain't pretty, but it gets the job done. It's ironic that BitTorrent, which is justly famous for equalizing the highly asymmetric nature of most people's internet connections, is itself so asymmetric when it comes to sharing: trivially easy to consume, but awkward and confusing to share. That's too bad, because BitTorrent is such a powerful tool for sharing. Hopefully this post demystifies the process a bit!

[advertisement] Interested in agile? See how a world-leading software vendor is practicing agile.

]]> http://www.codinghorror.com/blog/archives/001272.html Wed, 03 Jun 2009 23:59:59 -0800 Jeff Atwood http://www.codinghorror.com/blog/archives/001271.html Emrah Diril recently asked me this via email:

Steve Yegge mentioned in the comments of his last post that he gets quite a bit of hate directed his way.

Fake51: you underestimate the ability of people to get mad. Some people start mad and just take it out on you. The hating has gradually become a little too much for me.

I read the guy's blog too, but don't understand where this is coming from. Some people just have this tendency I suppose.

Do you have a similar experience? I don't see you wanting to quit blogging, so how do you deal with this? Is it just a matter of personality? Are you better able to ignore this stuff?

I answered with one of my favorite quotes from Randy Pausch's Last Lecture:

And when it was all over, one of the other assistant coaches came over and said, yeah, Coach Graham rode you pretty hard, didn't he? I said, yeah. He said, that's a good thing. He said, when you're screwing up and nobody's saying anything to you anymore, that means they gave up. And that's a lesson that stuck with me my whole life: when you see yourself doing something badly and nobody's bothering to tell you anymore, that's a very bad place to be. Your critics are your ones telling you they still love you and care.

Welcoming and appreciating reasonable criticism is the right attitude to have, but it's not the full story. Do I love criticism? Do I seek it out? No. I have many personality deficiencies, but masochism isn't one of them. I don't have fantasies of waking up every day to an R. Lee Ermey browbeating from commenters. Or, maybe I do. I should blog about that.

Criticism, painful though it may be, is still a conversation. It means your readers and listeners are engaging with you, and there's something to learn from following that conversation. Those messages you're broadcasting out into the world are being received, in some form, by someone on the planet. Even if that person is, well .. this guy:

I stopped reading the blog a while a go. Joel explains my reasoning nicely in his latest post.

The mystery of the non-reading Coding Horror reader. Another NP-complete problem, I guess.

If you think something sucks to the extent that it's actively harming the world and you want it to go away, leaving comments to that effect is not the way. I know, because I bear the psychic scars of a million online flamewars, dating all the way back to 300 baud dualup modems and BBSes. I've been doing this a very long time. I've seen what works, and what doesn't.

I'm here to tell you that there is something much more powerful than criticism that you can bring to bear in these situations. Something almost unimaginably powerful in its ability to shape human behavior.

The "just don't look" strategy [is] effective in any situation where someone or something runs on attention. On the web attention comes in the form of links and pageviews so "just don't look" translates roughly into "just don't link or read". If you don't like who's on the cover of Wired, just don't look. If no one talks about her, she'll go away. Think media gossip sites are ruining the web? Don't read them. Leggy blonde conservative got your knickers in a knot? Just don't look. Commenters ruining the internet? Moderate your comments or close them up. If some Web 2.0 blowhard says something stupid, just don't look. Hate blonde socialites? Just. Don't. Look.

I am absolutely sick to death of hearing about Susan Boyle, both in the traditional media and online. Nothing personal, you understand, I'm sure she's a perfectly lovely person. But I don't talk about Susan Boyle, because talking about her gives Susan Boyle power and currency. I just ignore Susan Boyle. I wish I had two brains so I could ignore her twice as hard. I. Just. Don't. Look. And if we could convince enough people to ignore her, she .. disappears. Poof. Like magic.

One of my favorite books as a child was the Great Brain series, the story of a family in rural Utah, set in the late 1800s. In these books, there was a strange punishment the parents doled out to their children when they seriously misbehaved. For a period of a week, or longer -- depending on the severity of the misbehavior -- nobody in the family would talk to, acknowledge, or address in any way, that particular boy. It was called "The Silent Treatment". This didn't seem like much of a punishment to me. In fact, as an introverted kid who loved solitary activities like computers and reading more than anything, it seemed kind of like a .. reward. I couldn't reconcile this feeling with the semi-biographical reality depicted in the books. To the Fitzgerald boys, the silent treatment was the worst possible punishment, far worse than a physical beating. They would go to incredible lengths to avoid getting the silent treatment. As punishments go, it must have been a doozy, though I couldn't quite wrap my geeky, socially maladjusted young head around exactly why.

The silent treatment was a punishment I didn't fully understand until years later in life. That's how you change the world. Not by arguing with people. Certainly not by screaming at them. You do it by ignoring them.

And if you feel strongly enough about me and what I do here, you can begin by ignoring this.

[advertisement] Improve Your Source Code Management using Atlassian Fisheye - Monitor. Search. Share. Analyze. Try it for free!

]]> http://www.codinghorror.com/blog/archives/001271.html Tue, 02 Jun 2009 05:33:18 -0800 Jeff Atwood http://www.codinghorror.com/blog/archives/001270.html One of my all time favorite blog entries is a truly epic tale of dating gone wrong that culminates in the strangest reference to P=NP you'll probably ever encounter.

Joey: So you really did graduate from computer engineering?

New Girl: Yes I did, from UBC!

Joey: And you took the "Algorithms" course?

New Girl: Of course!

Joey: And you have all the papers you wrote?

New Girl: Yes! I kept them all, and I'll show them to you tomorrow!

Joey: I want to see the one we always called the "Hell Paper" at Queen's -- the mandatory fourth-year paper. You know the one, where we prove P = NP?

New Girl: I did that! I proved P = NP! I placed near the top of the class, and the professor used my paper as an example!

Joey: You proved P = NP?

New Girl: Yes!

Joey: Gotcha.

Poor Joey. Dating crazy people is one thing, but dating crazy people who claim to have proved P=NP is another matter entirely. I know, I know, my track record with P=NP is hardly any better. But at least you're not dating me, right?

NP completeness is one of the great unsolved mysteries in computer science; perhaps the best way to illustrate is through this xkcd cartoon:

The defining characteristic of an NP-complete problem is that optimal solutions, using math and logic as we currently understand them, are effectively impossible. Sure, you can approximate a solution, but an optimal solution requires so many calculations as to be infeasible, even with computers that operated at, say .. the speed of light.

In fact, one of the outstanding problems in computer science is determining whether questions exist whose answer can be quickly checked, but which require an impossibly long time to solve by any direct procedure. Problems like the one listed above certainly seem to be of this kind, but so far no one has managed to prove that any of them really are so hard as they appear, i.e., that there really is no feasible way to generate an answer with the help of a computer.

It's doubtful whether anyone will ever prove that P=NP (pdf), but in the meantime it's useful to recognize problems that are NP complete:

Unfortunately, proving inherent intractibility can be just as hard as finding efficient algorithms.

The theory of NP-completeness provides many straightforward techniques for proving that a given problem is "just as hard" as a large number of other problems that are widely recognize as being difficult and that have been confounding the experts for years. Armed with these techniques, you might be able to prove that the bandersnatch problem is NP-complete and march into your boss's office and announce:

I can't find an efficient algorithm, but neither can all these famous people.

At the very least, this would inform your boss that it would do no good to fire you and hire another expert on algorithms.

Now you can spend your time looking for efficient algorithms that solve various special cases of the general problem. You might look for algorithms that, though not guaranteed to run quickly, seem likely to do so most of the time. Or you might even relax the problem somewhat, looking for a fast algorithm that merely finds designs that meet most of the component specifications. Thus, the primary application of the theory of NP-completeness is to assist algorithm designers in directing their problem-solving efforts toward those approaches that have the greatest likelihood of leading to useful algorithms.

As with so many things in programming, the first step is learning enough to know when you're really screwed.

Unfortunately for poor Joey, this sad corollary to NP-completeness apparently applies to dating, too.

[advertisement] Improve Your Source Code Management using Atlassian Fisheye - Monitor. Search. Share. Analyze. Try it for free!

]]> http://www.codinghorror.com/blog/archives/001270.html Mon, 01 Jun 2009 04:26:30 -0800