blog.borud

Correct, Understandable, Fast

2012-01-02T10:57:00.002+01:00

Alex Reid recently wrote a blog posting about there being no shame in code that is "good enough". I suspect most people are only going to read the first few paragraphs and use this as some sort of justification for producing crappy code. So, as a service, I am going to quote the important part at the end of the blog posting.

Good enough doesn't imply half-arsed or lashed together. It should concisely meet the requirements at hand, not what you think the requirements might be next week. It doesn't mean you are naive and haven't considered the big picture, nor are you lazy or stupid. It doesn't mean you are a moron if you don't use wildcard generics and don't have a fetish for multiple inheritance.
I believe all developers should have a geek valve that prevents them from introducing overly-generic, indecipherable black magic to a codebase. In conversation you would look a bit unusual if you insisted on using flowery language to express a point that could be adequately conveyed in more standard terms. Some people may miss your point. The fact that their grasp of English isn't as advanced as yours doesn't make them stupid. It means you aren't communicating efficiently. Why can't the same logic apply to code? Favour explicit and clear over clever.

The challenge is in defining what "good enough" means. I'm not going to attempt that here, but I am going to say something about what I minimally expect from code that I am supposed to take seriously.

When I write code I have the following priorities:

Correct
Understandable
Fast

Correct.

If the code is not correct it has no reason to exist in the first place. The most convenient way to verify that the code is correct is to write proper tests. If I stumble across a piece of code that lacks tests, I am going to assume that the author either doesn't know or doesn't care if his or her code actually works.

It is not okay to skip writing tests. Yes, I know all the excuses for not writing tests so you can spare me the tedium of repeating them.

Understandable.

The code you write is for communicating with other people. Not for communicating with the compiler. The compiler will accept a lot of unreadable nonsense -- your coworkers will not. For your code to be understandable its intent has to be clear. Once the intent is clear you should aim for the simplest solution that will do -- the simpler a solution is the easier it will be to verify.

It is not okay to skip documenting your code. It is not okay to add layers of abstraction where none are needed.

Fast.

If you have managed to demonstrate that your code is correct and understandable, you should ask yourself if it will perform adequately. In a lot of cases there will be no need to optimize your code. The first thing you need to do is to understand what your performance goal is. Then you measure. Then you optimize.

Most programmers are bad at writing correct and understandable code -- which sort of makes it irrelevant whether it is fast or not.

Receipts

2011-12-27T13:47:00.001+01:00

While I'm sorting through the receipts from this year's travels, one thing really annoys me: the terrible formatting of receipts.

A receipt essentially has only three interesting pieces of information:

What
When
How much

If I can't find the "when" and "how much" in under a second, the person doing the receipt layout has failed.

Usually, the hardest piece of information to locate is the when. Our inability to express times and dates has always amazed me. I can't think of any single piece of common data that incurs more processing cost and errors than dates and times.

Why does anything related to accounting have to be so badly designed? Is this stupidity or laziness?

Democracy and the press.

2011-12-21T17:58:00.000+01:00

According to the "Democracy Index 2010" from The Economist, democracy is steadily retreating. The world is gradually becoming less democratic.

A few days ago I heard that CNN is letting something like 50 of their reporters go. The reasoning was that CNN would rather not have that many editors, reporters and photographers on staff, but want to source the material from "citizen reporting".

Whatever that means.

The biggest problem with this is that when a private person starts taking pictures of something that isn't quite right, there is a risk that law-enforcement will intervene and harass the individual attempting to record events in any way. People have ended up getting harsh sentences for recording the actions of law enforcement in public spaces. Pointing a camera at a police officer is extremely high risk. Even in many western countries.

The threshold for throwing an accredited journalist in jail is considerably higher. First off, law-enforcement usually think twice about molesting journalists in nominally democratic countries. Second, journalists are usually more aware of their rights and more prepared to challenge law enforcement. Third, if they do end up in jail, they have the backing of organizations that know a thing or two about making life miserable for corrupt officials.

We need proper journalists to poke their lenses in the faces of those tasked with upholding the law. "Citizen reporting" is all well and good, but it is inadequate to prevent our democracies from crumbling.

Sound, dynamics, and such

2011-11-29T11:41:00.000+01:00

This morning I came across a newspaper article trying to describe difference between current and former mastering practices for records. And of course with generous helpings of confusion. Most records today are mastered in a way that leaves very little dynamics. For those of you not familiar with what "dynamics" means in this context; high dynamics means that there is a bigger difference between the loud and the silent parts of a recording.

There are lots of tricks to boost silent parts and dampen the loud parts, but the most important tool in the arsenal to accomplish this is various forms of compressors. For more information on dynamic compressors the article on Wikipedia might be helpful.

While over-zealous use of compressors makes the sound somewhat dull and uninteresting it might be useful to point out why this is done. It is an adaptation to make recordings sound a bit better in challenging listening conditions. Most of us do the majority of our listening (to music and other content) on crappy earbuds in noisy environments. By compressing the dynamic range you don't have to ride the volume controls on your device when you are listening to music in your car, for instance.

This is nothing new, but it is, and has always been, a balancing act. There have always been devices and listening conditions where dynamics is a challenge. For instance, a lot of mastering engineers will have multiple sets of loudspeakers, some of which are rubbish, to test their settings. In an interview I read years ago, one mastering engineer pointed out that he used the crappy stereo in his car to test his work. If it worked there, and on his high end studio monitors, it was probably a good final master.

As a consumer I've long wished for compressors to become part of the player -- so that you can choose to what degree content should have its dynamic range compressed. For instance, if I am listening to a record at home, on a decent'ish stereo, I want the full dynamic range and as little compression as possible. However, if I am in my car, I might want the dynamic range to be as narrow as possible so the faint parts are not drowned out by noise and the loud parts do not burst my eardrums.

Some movies have awful sound mixing. The silent parts are too silent and the loud parts are too loud. If you are watching a movie in a noisy environment you end up riding the volume buttons and being annoyed.

Some devices have very simple compressors. For instance I had a CD player years ago that had a built-in compressor. (Generally you want to perform any compression on the digital signal, not the analog signal). However, this is not a very wide-spread feature.

Learning how to use a mastering compressor takes time. In fact, they can be quite complex beasts, featuring different compression parameters for different frequency bands etc. It is going to take a while for fully automated detection of adequate compression parameters is going to sound good. This is why I would love if the sound industry could come up with a reference model for compressors and encode the parameters into an automation track delivered with the content. This way the consumer could turn compression on when desirable -- or even be given a control that lets the consumer decide the severity of compression.

The lost art of mending.

2011-11-10T20:22:00.001+01:00

A couple of years ago one of my Alesis M1 speakers stopped working. These are powered studio monitors, which means they have a PSU in each speaker and an embedded amplifier. This is a known issue for these speakers. The problem is a 2 watt resistor that is mounted too close to an electrolytic capacitor, so the resistor heats up and over time ends up frying the capacitor.

At the time when this happened I didn't bother googling it, so I just noted the problem and dropped the speaker off at a company that my local dealer of sound equipment recommended.

It took them a couple of weeks to fix and the bill was for about $500-600. The guy had just thrown out the PSU, ordered a brand new one and put it in.

Of course, the other PSU failed shortly after.

Not wanting to spend another chunk of cash, this time I researched the issue a bit. All in all it took me 3 minutes of googling to figure out what was wrong. At the time I didn't have a selection of components lying around, so I had to spend 3-4 minutes online finding and ordering the components. They arrived 2 days later and then it took me all of 10 minutes to get set up, replace the components (with somewhat uprated components) and verify that the repair worked.

Total component cost was something in the area of $2-3 and then I had nine spare sets of components, so if the problem returns I can fix it easily. Any idiot can fix it.

What depresses me is that people don't know how to fix things. The dude in the shop apparently had no useful skills. Because if he did he could have saved himself the hassle of ordering (and waiting for) a new part, and he could have saved me a huge bill and perhaps seen some return business.

I see the same thing with my car. Every spring and every fall I switch to or from winter wheels and I stored the other set at this company. The last time I picked up my car, one of the wheel nuts was missing and about 4-5 of the remaining wheel nuts had been over-torqued thus destroying them. You would think that when you pay someone to do this, and they do this for a living, they would be able to do a better job. Even I know that you can't apply arbitrary amounts of torque to the wheel nuts, and I am not a mechanic -- I don't do wheel changes for a living.

About once per year I go to the local landfill to get rid of things I don't need. Thankfully, they have various recycling stations there so I can at least hope that some of the things I get rid of there actually are recycled responsibly. But I am still depressed when I drive home. So much STUFF. And most of it still usable -- and a lot of the stuff that is broken should be easily fixable.

If you care about the environment you should care about the fixability of your gadgets. You can buy carbon offsets or take part in all manner of feelgood environmental nonsense, but the fact is that the device you are using to read this blog posting will end up in a landfill when it breaks or when you get a new gadget. Because you are not going to fix it, and I am pretty sure you do not know of anyone capable, willing or qualified to fix it either. It might end up in some third world country where some kid is going to set fire to it to get the metals out of it -- releasing noxious gases.

If you have kids or you want a hobby: lots of gadgets have cool components that you can scavenge. Rather than throwing away that useless printer, rip it apart and have a look inside. There's lots of fun stuff inside that you can build interesting toys from. And it isn't hard. You don't have to be a genius to play with these things. Get an Arduino and learn.

I think we need to make it cool to know how to fix stuff again.

Lightbulbs, why isn't this a solved problem?

2011-11-02T12:37:00.000+01:00

With all the new regulations that require lightbulbs to be of some energy-saving design, buying lightbulbs has become a major pain in the neck -- as well as an overly expensive affair. Here are some suggestions for both manufacturers and groups that advise consumers.

Many of the new types of light bulbs have physical dimensions that are not within the envelope of the traditional tungsten counterparts. Lightbulbs that are outside this envelope should be clearly marked as such since it is very hard to judge while you are standing there in the store without ripping open boxes and comparing light bulbs.
The type of socket should be clearly marked on the box. One would think that this piece of information would be prominently displayed on the packaging, but quite often it is tucked away somewhere in the fine print. I've even come across packaging that doesn't print the socket type at all. The result is that to be sure you have to rip open the packaging to inspect the socket.
The lifetime figures printed on the box are nonsense. Exactly NONE of the power-saving lightbulbs I've purchased have lasted anywhere near as long as it says on the box. Not even after I moved into a brand new house. Quite obviously the manufacturers either lie outright or the standards for testing are inadequate. Many of the energy-saving lightbulbs boast of longer life (to justify their high price), but in reality they have only somewhat longer life-span. If it is longer at all.

Points 1 and 2 should be easily fixable for manufacturers. The fact that many major manufacturers don't do this just means they have figure out what on earth the people who design their packaging are doing.

Point 3 is probably never going to happen. Lightbulbs have limited life-span by design so manufacturers are incentivized to make sure their bulbs stop working as soon as possible -- but not so soon that consumers catch on. If they wanted to they could make nearly unbreakable bulbs that would keep functioning for decades, but they won't. Fine, we know this, but I think some manufacturers have become a bit too greedy and it is time to introduce some balance. Perhaps some regulatory action is needed.

IPv6 has to be made sexy.

2011-10-25T18:57:00.001+02:00

Switching to IPv6 is like taking out the trash: everyone hopes someone else is going to do it so they don't have to.

I've known a lot of people who have enthusiastically set out to IPv6ify the world only to realize that their mission is met by as much enthusiasm as experienced by those who turn up at your doorstep with pamphlets to guide your soul away from some inevitable doom.

If I were an economist (a proper one) I would probably ask: where are the incentives?

Because there are no incentives. In fact, there are only disincentives. It is hard enough to teach people how to set up a network using IPv4. If you start talking about IPv6 their eyes just glaze over.

For IPv6 to become a mainstream occurrence companies like Cisco and Intel need to hire people that understand the need to make products that are significantly easier to configure. The products have to be so easy to configure that they are sexy. And this has to go beyond just slapping some My First Pony GUIs on top. The sexiness has to be deeper. To appeal to morons and tinkerers alike.

Ever tried to configure a router using its command line interface? Exactly. They are rubbish almost no matter what the price range. It seems no matter how many times I try, I can't get the broadband router in my home to behave sensibly. It has been some years since I had to set up a Cisco router, and while those are made by somewhat brighter people, you are still going to need to study the manual to figure the thing out.

When networking gear becomes more sensible, and is designed by people who have an ambition to simplify their use adequately, IPv6 can succeed. Until then it will remain the domain of people with pony tails, suspect body odors and an undying belief that the rest of the world even gives a shit about the diminishing supply of public IP addresses.

Android as a platform for launching a new language?

2011-10-11T09:51:00.002+02:00

I'm not sure if I would say that there is a lot of focus on designing languages at Google, but with the recent announcement of Dart, I couldn't help but think that Google actually has what it takes to make a language successful quickly. If they develop a new language for the largest smart-phone platform on the planet, they can reach a lot of programmers in a very short time.

(Just look at how quickly the world learned Objective-C, which was all but an oddity, to make apps for iOS).

I think Oracle are mismanaging Java and that they have proved to be extremely short-sighted. Trying to extort Google for cash rather than seeing Android as an opportunity for becoming relevant on mobile platforms is a very weak response and one that tells me that Oracle does not have the long term in mind. And if you make an investment in a language you want a responsible "owner" that has the long term in mind.

Oracle is not that responsible owner, and as long as the attitudes of Larry Ellison persist, there will be a dark cloud hanging over Java.

I'm not sure if either Go or Dart are suitable for writing applications for a mobile platform. But it if Google should decide to create a replacement for Java on Android I think it could have a profound impact on the industry. They can reach a lot of developers very fast.

There will be no "later".

2011-09-28T01:30:00.001+02:00

One of the more peculiar things you often see in source code is when programmers write messages to their future selves. In the shape of TODOs or even excuses for why the code is not done properly and how they are going to come and clean it up later.

I'll let you in on a secret: there will be no "later". Most unfinished code is going to remain unfinished to the day it is replaced or becomes irrelevant.
If you don't do it properly now, you are not going to come back later and put things right. You have just one shot at doing it right.

Because it is never a good time to get things done properly. That green field on the other side of the release isn't there. It is a mirage. Because when you get there you are going to waste the first part of it on distractions, and you are going to spend the second part of it running around like a headless chicken with your ass on fire producing even more unfinished code in time for the next release.

Steve

2011-08-27T16:29:00.011+02:00

My admiration for Steve Jobs started the first time I read about the NeXT computer back in 1988. The NeXT computer was just a stunning piece of work. Absolutely every aspect of this computer was forward-looking in a way that I had never seen before. In fact, the least interesting thing about this computer was the hardware spec -- even though it was pretty spectacular.

What made the NeXT so amazing was that it represented Steve Jobs' ability to look ahead. He understood the importance of having a solid operating system at the bottom. He made great strides in bringing graphical user interfaces to the forefront. He understood the importance of networking. He understood the importance of industrial design. And he was able to keep all of these things in his head at the same time.

The NeXT was the first beautiful computer. And it wasn't just beautiful on the surface. Anyone who has taken apart a NeXT cube and looked inside can attest to the fact that this was the first personal computer that was properly engineered. Right down to the spring-loaded back-panel screws and the only cooling fan I have ever seen that not only keeps the device cool, but also filters the air and is easy to service because even the power connector was designed with care and attention to detail. Not to mention the single cable running from the main unit to the screen. One cable with video signal, audio in and out, keyboard, mouse and power, yet it was flexible and not overly thick. It took some hard work to design that thing. Amazing attention to detail.

It was also the first proper personal computer to have a UNIX flavor operating system. This was a wise choice. In evolutionary terms, UNIX may be old, but it has stuck around because it was an unusually optimal design philosophy. The core idea behind UNIX is simplicity, smallness, and abstraction. Which means it is uniquely suitable as the foundation for layering operating environments and user interfaces on top. UNIX today is the dominant operating system in mobile telephones as well as the vast server farms of companies such as Google.

UNIX, as I use the term, refers to a family of operating systems rather than any particular operating system. NeXTStep was, strictly speaking a "Mach kernel with a BSD personality". Explaining what UNIX really means may be the topic of a whole other blog post.

The NeXT computer was, of course, the direct precursor to the computers we use today. And in evolutionary terms, the Mac practically is the NeXT computer.

Steve Jobs is often thought of as a harsh, direct and difficult person. And I have no doubt that he is all of those things. Over the past days I've read a few articles on how Steve Jobs is not a good role model for aspiring leaders.

I both agree and disagree.

I disagree because this seems to ignore the fact that it takes a strong personality and a strong will to realize a vision without compromising until mediocrity is inevitable. When a product is designed it goes through several stages. You start with the concept. The the concept is watered down during engineering. Then the product is further watered down when it has to be manufactured. This is why concept cars look so exciting -- and why the finished production model is inevitably a disappointment. The production model is always a compromise.

This is also why personal computers, for the most part, are rubbish. Their realization is the result of giving in to every limitation that has been encountered from vision to production model. One of the key things Jobs did was to be unaccepting of this and to turn it upside down. Rather than having current limitations dictate the product he would push to address these limiting factors. In design, in manufacturing and in attitude. This, of course, causes friction, (and occasionally disaster) but in his case, the end results were hard to argue with.

(Example: Jobs wanted the original iMac to be silent. In other words it could not have a cooling fan. The engineers said it couldn't be done. Jobs insisted and the engineers had to find a way. And they did. The iMac had no cooling fan. It may be hard to understand why Jobs insisted that this detail be delivered -- until you consider that most computer professionals suffer from hearing loss due to cooling fan noise).

If Jobs had been a nice, accepting, more "democratic" leader, Apple would still be making the low quality grey boxes that were made while Jobs was away from Apple -- if they had been in business at all. The workplace might have been "nicer", but Apple would have descended into mediocrity and eventually death.

However I do agree that he is not a suitable role model for most leaders. The reason is that Steve has a strong sense of direction, product, vision, and quality -- and most leaders do not. Before you can lead you have to know where you are going. And even if you know where you are going, you have to get up and lead every day. And you have got to deal with setbacks without becoming too timid.

Compare this to the management at companies like Sun and Yahoo! -- companies that have had a distinct lack of vision and direction. I hear Carol Ann Bartz is a tough CEO, but she has been the CEO of Yahoo! since 2009 and I have yet to see any clear direction or philosophy. What is it that Yahoo! is supposed to be good at? I suspect that no amount of "being tough" is going to make Carol Bartz a better leader for Yahoo!. Because she just doesn't have any credible vision. She is treading water. Just like the last 3 CEOs.

McNealy was the CEO of Sun for 22 years. Perhaps one of the most promising computer companies ever -- yet it never seemed to realize its potential. What on earth was McNealy doing? Sun should have been the queen of the prom in the cloud computing era. Instead the company went out with a whimper and was sold to perhaps one of the least innovative tyrants in the industry where what little it had going for it is being torn to shreds. I can not remember having seen McNealy say anything even remotely inspiring or insightful. Can you?

If you are to lead forcefully, then you must have clear direction. Steve Jobs had clear direction. The majority of leaders do not. The majority of leaders aren't even involved in the products their companies make and thus it would be silly to adopt a style that requires you to be involved. Deeply involved.

I think the aspects of Jobs style that leaders should focus on is the almost fanatical urge to evolve and to invest in the future. It is easy to forget that Steve Jobs spent a lot of time failing. That it took him a long time to find the right balance between short term needs and long term strategy. But he always had a long term strategy. And he always had the guts to be unaccepting of shortcuts that betray bigger goals.

Don't look at where Apple is now and what they are doing now. Look at how Apple got there. Hate and fear them if you must, but at least understand why they succeeded. Or more precisely, how Steve Jobs made Apple a successful company.

Near Field Communications Hype

2011-08-19T13:07:00.001+02:00

Occasionally people ask me what I think about NFC. Or rather, whether the hype around NFC is warranted. In the interest of not having to repeat myself I thought I'd just jot down a couple of thoughts on my blog.

For the most part I tend to be relatively immune to at least some types of hype. This is not an attempt to seem clever or cool, but this is just the way I am. And the reason for this is that I find it very hard to become excited about technologies that are outside our practical reach.

I'm excited about 3D printers because I know that I could afford buying one, or if I had the time, it is well within my abilities to build one. I also know that some 3D printing technologies work amazingly well and that it is inevitable that prices will come down.
I am not at all excited about going to Mars -- because we are still very far away from knowing how to do that (despite what the popular science crowd would have you believe). That of course doesn't mean that I don't think we should aim for Mars -- I just fail to be excited about it as long as I don't see how we would get there. At most I might be excited about sub-problems that need to be solved, but the end goal is too far away for me to feel any excitement.

The hype around NFC never excited me. In fact it annoyed me because there was too much talk and too little action. Yes, there are NFC-enabled gadgets and the technology is now trivially available to manufacturers. However, NFC-enabled devices are not widespread. It is only when the vast majority of new phones coming to the market are NFC enabled that NFC will have the potential to make any impact whatsoever.

This means that the most important thing right now is not what we might be able to do with NFC. Forget about that for now. The most important thing right now is to get manufacturers to take the risk of including NFC in all mobile phones without any clear idea of what it is going to be used for. This is a necessary leap of faith. And it is not without risk. But unless manufacturers are willing to make this leap, NFC has absolutely no future.

Once NFC availability crosses some hitherto unknown tipping point innovation, excitement, and explosive development will happen. And we will have absolutely no way of saying where it may take us. And I am absolutely positive that at that point I will be excited.

Just ship the damn devices already.

Flickr is dead

2011-08-13T17:17:00.000+02:00

In a few days my pro account on Flickr expires. Although the cost of a Flickr Pro account is relatively trivial, I have been giving renewal of my Pro account some thought. The conclusion I have reached is that I am going to renew it, but it isn't because I belive Flickr has much of a future.

Because Flickr really has no future.

Since Yahoo! acquired Flickr in march 2005 nothing much has happened. While some effort was put into making the site scale better initially, Yahoo! hasn't really done anything to improve any observable aspects of Flickr for years. Which is a shame. Because Flickr used to be the site for photographers. It even launched the careers of a handful of photographers who used to be mere hobbyists.

Today the site, and the brand, is in decline. What used to be a thriving community has increasingly taken on the appearance of a ghost town.

Even from a technical perspective Flickr is failing. When linking images from twitter or from social news sites such as Reddit, people have started noting how there is a reluctance to click on image links hosted by Flickr -- simply because the service is too slow. Users respond to this and instinctively avoid Flickr.

I will be renewing my pro account for another year. But the main reason I am doing that is because I want to buy a bit more time to figure out where I am moving my pictures. I hope that Google+ or whomever I migrate my image galleries to, will provide some tools for migrating my content off Flickr.

As for why Yahoo! are so consistently running their properties into the ground, I don't think there are any simple explanations. But I used to work for Yahoo! and I was never very impressed with the upper echelons of management at Yahoo!. There's a distinctive lack of passion for products and technology. Also I was never able to observe any clear and strong direction. Yahoo! is the master of half-hearted efforts.

Here's an interesting experiment: ask people what Yahoo!, as a company, is really good at?

I can't think of anything. And I used to work there.

Please do not take my renewal of my Flickr Pro account as an endorsement. Because it isn't. If you are shopping around for a photo site I'd keep an eye on Google+ or Facebook. They aren't there yet, but they have more momentum.

Reforming Congress

2011-08-10T14:44:00.001+02:00

Having watched american politics from the sidelines, and through the highly polarized lens of dysfunctional media, one has to wonder what the CIA World Factbook means when it says that the US has a "strong democratic tradition". I don't see it.

I don't see a press that is capable of objectively reporting the facts. And I don't understand how you can have a democracy when politicians get elected mainly on the basis of how successful their heavily funded PR campaigns were.

The only obvious way to start fixing the severely dysfunctional political process would be to outright ban campaigns as they exist today. No more political commercials or advertising. No more donations from corporations and private parties. No more special interests keeping politicians as pets. Strict anti-corruption laws and strict enforcement of same laws.

The only campaigning that should be allowed is unscripted debate on public television. With a tight regimen of fact-checking and public exposure of politicians who lie or distort.

If politicians are allowed to lie and misinform the voters and to leverage billions of dollars in elaborate propaganda efforts to gather votes you do not have an election as much as you have an auction -- where the power goes to the highest bidder.

Free investment tip.

2011-05-23T02:40:00.002+02:00

Do you want an investment tip? If I had a few million dollars lying around I'd start looking for a startup aiming to write software to support the upcoming design, fabrication and manufacturing revolution. The sort of software you need to prototype, test, and yes, machine or 3D print physical objects.

Sure, there is already lots of software for this on the market, but it is extremely complex, expensive and most of it, if not all, is targeted at Windows. Makers don't use Windows. Makers use OSX or Linux. And it is the Makers you have to keep an eye on because they are the people defining what is right now a relatively small niche. Even if you ultimately will sell larger volumes on Windows (though I am not so sure you will).

The reason I think this is going to be a good area to invest is because the software houses in this industry are going to miss this market entirely. They are, after all, professionals with decades of experience and wouldn't even think of licensing even scaled down versions of their ultra expensive CAM software for calculating optimal toolpaths at prices affordable by mere hobbyists. Much less engage with them to externalize innovation and have mere users extend the ecosystem with their own extensions and tools.

As manufacturing becomes available to the masses (both in the shape of cheaper machines, but also in the shape of companies selling access to equipment through the net), in much the same way computers slowly became available to the masses in the late 70s, there will be a need for better, more user-friendly, more portable, and more extensible and adaptable software. Because that is the big hole in the ecosystem right now: the software. There are some cheap packages for doing rudimentary 3D design. There are also some packages to drive mills and 3D printers. But there is a very noticable absence of decent CAM packages to bridge the gap between those two worlds. That is: for the sort of audiences that are just now starting to enter the stage.

I predict that within the next 2 years a startup will emerge from the Bay Area that addresses this gap and makes an affordable CAM package that runs on something other than Windows. In 5-6 years the industry giants are going to be scratching their heads and asking themselves how on earth they could not have seen that coming -- and people will point out to them that history just keeps repeating itself. They will miss the market precisely because they think they know everything there is to know about their particular domain and they will be too heavily invested in the status quo to adapt. This happens in almost every industry from time to time.

There are also going to be a lot of innovation in the type of fabrication processes that are possible and practical. From advanced voxel machines to manufacturing living tissue. This is going to require flexibility and hackability.

So if you have a few million dollars lying around and you want to invest in something that may become a really exciting future market, I suggest you pack your bags, go to the Bay Area and start looking for startups that want to build affordable CAM-software. Of course, you'll have to be in it for the long haul, but I think there are fortunes to be made here.

Through sites like http://cloudfab.com/ you can already fabricate one-offs at relatively low costs. These types of companies will also be very important in helping you figure out what fabrication processes will be appropriate for you since this area is experiencing very rapid innovation. But you still have to provide them with decent designs and that requires proper tools.
A package like SolidWorks is going to cost you something in the neighborhood of $4.000 to $5.000. It is a niche product for industrial use, though a bit overkill for the hobbyist. Of course, it is also quite obviously too expensive for the $50-$500 price range that a hobbyist might be able to pony up. Judging by the number of hobbyists I've encountered that seem to know SolidWorks and similar packages without having a job where they'd use it, I'd suspect that there are a lot of pirated installations of SolidWorks out there.

Oh and of course, SolidWorks is neither available on MacOS X, nor on Linux. Which is a big handicap even if Dassault Systèmes were to create a "Maker version".

If you want to go into the business of building these tools I wouldn't worry too much about being undercut by traditional players. By the time they catch on and slash prices they'll have nothing worthwhile to offer to the "amateur" segment anyway. They'll sit on the fence until it is too late.

There is no Web 3.0

2011-05-14T12:43:00.014+02:00

People keep talking about Web 3.0 expecting the sort of significant shift that Web 2.0 represented. I don't expect there to be one. I view Web 2.0 as the transition where certain aspects of the web "went hockey-stick". Where certain potentials already inherent in the web were realized -- mainly that it went from being a one-way medium to a many-to-many medium. Also we saw the emergence of businesses that "get" the web. But also small things like Firefox challenging Microsoft and bringing actual progress back into the stagnant world of web browsers.

The Next Big Thing.

So what is The Next Big Thing? The next significant shift? I think it is about the physical world. About machine to machine communication, about "dumb" devices becoming "smart", or lacking that, connected. About sensor networks, remote control and about everyday devices interacting with each other in useful ways across vendors and types of gadgets. About devices and gadgets becoming addressable in sensible ways. About connectivity coming to devices that we previously thought of as too cheap or too small for it to be cost effective to put communication and processing capabilities into them.

Very little of this is new. Machine to machine communication is already an established art, but it has lacked openness and the right sort of mindset that is needed to fuel explosive growth in the consumer market. It is still a specialty field. And connected devices are still somewhat of a novelty.

If you consider trivial examples such as your pulse monitor. You have a sensor you place on your body and a watch that can accept data for storage and processing. If you work out at a gym they might have a treadmill that can accept the signal from your sensor, but that's about it. Nothing else talks to it. If you want your mobile phone to interact with the sensor you need to do a bit of hacking. And it isn't for everyone to design their own electronics, figure out the protocols and write their own software to do this.

Various makers of home entertainment systems have had systems for allowing their devices to interact for years. Sony, Bang & Olufsen etc. The TV, DVD player and amplifier can talk to each other, albeit in crude and limited ways.   We've had this for at least 25 years now. But it hasn't really taken off. Because every manufacturer only offers interoperability between their own devices.   Which isn't terribly useful as you are likely to have a mish-mash of different devices from different manufacturers bought at different points in time.

Most everyday devices capable of interacting wirelessly exist within closed ecosystems. Closed proprietary protocols, and when the protocols are indeed "open" they are usually special purpose or they are designed by committees that produce bum-numbingly boring specs that most people would rather not play with.

As far as the hardware is concerned, you would be amazed how cheap processing power has become. Texas Instruments have some microcontrollers that cost $0.25 apiece. Low power communication hardware for wireless networking are getting ridiculously cheap as well.

Form matters.

If I may go off on a tangent here for a while; the qualities of a standard or a spec matter greatly. For it to be useful to a big audience it has to be brief, concise and precise. What it describes also has to be simple. Any excess fat must be trimmed. It is extremely important that the spec is written from the point of view of someone who has actually implemented what is described.   SMTP, HTTP and even TCP/IP didn't win because they were the best possible protocols -- they won because it was practical to implement them. Because they were simple enough. X.500 and the OSI stack were not, and quite deservedly starved to death from lack of enthusiasm.

If a spec is not usable as a practical blueprint for someone building or implementing the spec, it is not a useful document.

Also, the format matters. Again, look at the standards that make up the bulk of basic Internet technologies. They are text files. Now compare to some of the more unwieldy standards and specs of today that are maintained as big, fat Word documents. Often horrendously badly formatted Word documents at that.   There is a number of reasons why word processing files are not suitable for authoring specs and standards. However the minutiae of this is a topic is beyond the scope of this blog posting.

If you want a good example: compare the SAML specs (http://saml.xml.org/saml-specifications) with the OAuth specs (http://tools.ietf.org/html/rfc5849). No, really, have a look and see which you would rather work on.

Clarification: I think SAML is a good idea. But it is a pain in the butt to work with because of the massively big, badly written specs.

Hardware.

Still, the main problem seems to have been in the hardware domain. Anyone and their grandmother tinkers with software. Not a lot of people play with hardware. And for good reason. You need to have a fair bit of basic knowledge to play with hardware. And to be honest, the hardware mostly isn't all that great.

For instance, I have been playing a bit with mesh-networking and the XBee devices. They are interesting devices, but to be quite honest, not very easy to work with. The manufacturer does not really understand the potential of their product and have not paid much attention to making it easy to configure them. If you want to configure an XBee device you'll have to set aside a few evenings to study the docs. Then you will need to figure out how to work around their worthless, Windows only configuration program. It takes a lot of trial and error to figure out how to work around the dysfunctional and confusing configuration software and I would assume that it is probably better to write your own software for this if you plan to spend much time working with it.

Arduino.

However, there is hope. As advanced microcontrollers have become cheap there are some brilliant solutions available. Chief among them is the unassuming, but all-important Arduino. The Arduino is the first wide-spread prototyping platform that makes hardware truly accessible to the masses. It has three things going for it. First the hardware: it is cheap, offers adequate processing power, lots of I/O and the physical interface is exactly what a tinkerer needs.

Second is the software. While the main IDE used with Arduino is aimed at non-programmers, you can still elect to take a more "traditional" route if you prefer that -- using avr-gcc and traditional C programming tools. To help beginners (and more experienced hardware hackers) along there is also a large library available for the Arduino. For doing anything from driving displays to driving servos.

Third, the Arduino has a vibrant community around it. There is an abundance of forums, howtos and people who are willing to share their knowledge. If you want to hack hardware using an Arduino you can accomplish a lot of fun and interesting things by participating.

The Arduino isn't important for the consumer in much the same manner as Java, PHP, Linux or MySQL is unimportant to the users of the web. It is important for the people who innovate, tinker, build and invent. It significantly lowers the barriers for prototyping hardware solutions and it has adequate I/O and processing capabilities to accomplish interesting things.

Interestingly Google has recognized this and are now using the Arduino as a platform for their Android Open Accessory Development Kit.

Right now the Arduino is one of the most important gadgets on the market. Because it significantly lowers some important barriers.

Era of talking hardware.

As I said earlier, I do not believe there will be a Web 3.0 in the sense that we will see a marked shift. There is perhaps a Web 2.5, which is "more better" and the combined fruits of Big Data and machine learning, but I don't see any distinct shifts for the web as such up ahead.

I think the Next Big Things is having gadgets talk to each other and that this is done using technology that is far more available to the masses than is currently the case. The key to this is simplicity, and in the electronics industry, the winners will be those who realize that to fully exploit this opportunity they have to re-think who their target audience is.

Why talk of "segmentation" frightens me.

2011-05-08T20:32:00.001+02:00

When I wish to provoke thought I often state that "there is no such thing as segments - only successful or unsuccessful products" or that "I do not believe in segmentation".

The first statement is, at best, a simplification about how I think about customers and users in relation to products, but it is a simplification that is closer to what I believe than just stating that any market can, or should, be thought of in terms of segments. One could say that I am rounding off to the nearest significant decimal.

The second statement is largely true. I think segmentation is counterproductive because it tends to act as a handy excuse for not succeeding in appealing to the user. But more importantly, I think segmentation is counterproductive because the partitioning of the customer base is often not sufficiently knowledge-based. Ask for hard numbers and more often than not, you will get the results from a poll or a focus group. (This is so wrong I cringe at the lack of scientific validity whenever I see the results from polling). And even when segmenting is informed by at least some data, people don't always understand what to do with that knowledge.

By observation, not by decree.

Now, I have already admitted that "there are no segments" is not something I truthfully believe in. But this statement is a useful stand-in for what I really believe. Because what I really believe may be a bit trickier to understand on an intuitive level. What I really believe is that segments can only be derived from recent observation. There are two reasons for this.

The first reason is that merely guessing what segments there are is...well, just guessing. Of course, there are many forms of guessing. Some of them are cleverly disguised as science. You can ask people what segment they are in. You can ask questions that you think will determine what segment they are in (which might work, rarely does) or you can do what a lot of people do: recycle random factoids from books or from the web.

Litmus test: If you have names for the segments even before you have data supporting their existence or describing their size or importance (not always the same), you are not being scientific.

(A quick comparison. Automated news aggregation sites, like Google News, are all about automatic segmenting of news reporting. Or "clustering" as it is called in search or machine learning nomenclature. I forget which. The software is not really pre-configured to know anything about any topic. It works by ingesting a fair share of the world's newspapers and looks at significant similarities between articles to determine when new themes appear, breaking stories etc. and then clusters these articles together. It knows nothing about, for instance, ice skating -- but if a significant event were to occur in the ice skating world, it would most likely be able to tell that "something" happened and that these articles belong together. Even though the system had no way of knowing or anticipating anything about ice skating. These clusters come and go. Some are short-lived, some are more persistent -- perhaps even permanent. I see this as the same phenomenon. Only: I doubt that most segmentation in market research is done as dilligently)

The second reason is that segmentation changes. While a given partitioning criterion may remain somewhat stable over shorter periods of time, there may arise new partition criteria that are suddenly more important. For instance, Internet access used to be rare and relevant to a small number of people. In the west today, not having Internet access is more rare than having it used to be. This happened relatively slowly over a decade or so, yet the advertising business and the content industry failed to catch on early enough and as a result, were late in upping their game. There was no segment for "Internet users". Then there was one but it was small. Then it got big. Then it disappeared because everyone was an Internet user and those who were not...well, marketers usually don't care about cave-dwelling hippies off the grid, now do they.

To sum it up: if segmentation is to be useful it must be based on scientifically sound analysis of reality and one has to understand that it is a dynamic phenomenon and that the rate of change is usually accelerating. Also it is important to understand that both quantitative and qualitative changes in partitioning occur. (Segmentation is, when we get down to it, a mathematical concept that has been sufficiently dumbed down to be taught in business schools. But don't tell anyone I said that).

Which brings me to why I willfully mislead people by saying "there are no segments": if saying so leads to people focusing on the product they want to build rather than flimsy data and naíve analysis of same, that outcome preferable.

Besides: build a sufficiently brilliant product and it will cause observable change in segmentation.

Product first.

The last statement in the previous section also sums up why I don't think paying too much attention to segments is important. It limits you in how you think about products and it does so in big and important ways. Segmentation can be a tool for tweaking existing products, to eke out small incremental gains, but it isn't a tool that is usable to innovate. It isn't like they ran the numbers and figured out in the 1920s that "our market research says there should be a market for watches worn on our wrists". Also there's this old chestnut:

"If I’d asked people what they wanted, they would have asked for a better horse"
(attributed to Henry Ford)

If you pre-constrain a product to fit within a given mold you may miss big opportunities. You may constrain your creative forces to focus their energy on filling a spec rather than going back and asking fundamental questions. (This is probably why I have yet to find a garlic press that actually works: they are all just bad copies of each other and the fundamental problem has yet to be solved).

The reason Apple revolutionized mobile phones is because they asked fundamental questions. Not because they accepted the mold into which the incumbents hammered their products.

In retrospect most people will giggle when you show them the almost perverse degree of segmentation some mobile handset manufacturers wasted their energy on. But just 5 years ago, the same people would probably think quietly to themselves "wow, they've really covered all the bases here" and be impressed at the diligence with which manufacturers managed to fill every niche and tweak every last bit out of their product.

There are more people thinking about products in more radical ways and with the means to actually act on their ideas at relatively low cost than before. It would be naive to think that this has no impact on how we model and predict. The consumer is exposed to more new ideas every year than ever before.

So when I say I don't believe in segmentation, it is because it evokes thoughts of slow, ill informed models that become irrelevant so quickly that they have limited predictive power. But more importantly, I think they are not a good tool for ensuring nimble strategy.

The when quantitative factors go all hockey-stick it leads to qualitative changes.

(I would have liked to say something about segment size vs segment importance as well, but this blog entry is already too long and rambling so I'll save it for some other time).

Speak, friend, and enter

2011-05-08T13:45:00.002+02:00

Ubiquitous and universal single-sign on is never going to happen. Everyone wants to own the user, or at the very least not relinquish control of the user to possible competitors if they can help it. This just seems to be a fact of life; whether informed by reason or fear. A bit of both I suppose.

This of course means that we need to keep track of gazillions of passwords. Which is a problem. A balancing act between convenience and security. Every so often you are faced with another registration screen that requires you to type in a password. And what on earth should you type in?

You may have a system for choosing passwords -- in itself a security risk, but still a very common way to cope with the massive number of accounts you have without having to write anything down. The risk being that someone will figure out your scheme. The gamble is that a) the scheme won't be self-evident by cursory study of one or a few samples, b) you are not that interesting so why would anyone invest time in cracking your password scheme.

So, you are looking at a form and you need to choose a password. You type in something and the validation scheme of the site says you can't use that as a password. This is annoying. Because it means you have to come up with something that might be harder to remember. Perhaps you'll even have to write it down to ensure you remember it.

I particularly dislike password validation schemes that require you to enter mixed case characters and digits with a minimum length of N characters. I have a theory that this does not enlarge the search space for possible passwords: it is going to severely limit the space. Why? Because we're human and it is very likely that we are going to pick a memorable password string that has these properties.

Ask yourself: what information has mixed case and a number and will be easy to remember? What are the first 5 password schemes you can think of that uses information you would remember that has these properties? Next, how much of this information exists in some form in the public space?

I believe the correct response to this question contains at least one expletive.

Unlike in the movies, where the hacker manually types in passwords until he or she succeeds, in the real world you have web crawlers, you have frequency dictionaries, you have oodles of neat software to look for patterns and you have ways of automatically assembling a dossier on your target. A dossier that can be used to generate password candidates. Password candidates that can be used to mount a brute force attack.

(Okay, so sometimes it happens like in the movies. Sarah Palin's mail account was broken into because her security question asked for information that was readily available on her wiki page. This is why security questions to enable password reset is a Really Bad Idea).

We have to assume that the people who write validation code for websites are not going to be experts in information theory, psychology or cryptography. This is why I wish people who actually know something about this subject would put together a sort of manual of sound practices in designing password validation and helping people choose sensible passwords.

I know nobody wants to stick their neck out for fear of criticism. Especially not people with computer security backgrounds since they have the most to lose if they were to write something that just makes things worse (which is a very real possibility). But I would wish that there at least existed some design guidelines or a brief discussion that programmers could have a look at before adding counterproductive password validation schemes to their websites. A go-to resource that "everyone" knows about.

Before we part I thought I'd point out that possibly the biggest security risk is right in front of you. You're looking at it. Your browser. It most likely has a cache of a considerable number of passwords that you use to access your most important web sites.

How well guarded do you think that password database is?

Failed library design

2011-05-07T08:15:00.002+02:00

In the past week I've spent a considerable amount of time being angry and frustrated with what I view as inept API and library design.

The point of having libraries is to make a given problem domain more accessible, by creating meaningful abstractions that insulate the programmer from some of the underlying complexity, to help the programmer produce correct code and to reduce the workload on the programmer.

It is important to note that the latter is more about time and effort than the number of lines of code necessary. It isn't about the amount of typing that has to be done. This also extends to the documentation and whether it is necessary to spend days reading up on something just to use parts of a library. If you have to spend a considerable amount of time to understand the design and implementation of a library just to use it in a simple context, the library designer has failed.

Failed library design is counterproductive and uncool.

In the past couple of weeks I have tried to make use of various SAML-related libraries and I have to say that I am thoroughly unimpressed. In fact, I am angry. I haven't seen such massive amounts of inept library design and sloppy programming since the last time I had to deal with ... well, certain XML technologies.

It is for good reason I become wary when people tell me they have written infrastructure code that deals with XML because it would seem that this particular niche in programming attracts people who simply are no good at what they do. Before hiring people with extensive XML-experience on their CV I need to see two things: I need to see code they have written which they think represents their best effort, and I need to talk to them to understand if they are prone to pointlessly complex library design.

If what I see is along the lines of what you can often find in certain XML libraries, then they can't expect to be hired. I'm sorry, but I firmly believe that the job of a programmer is to shun complexity and create simplicity wherever possible and I have no desire to work with people who are either too lazy or too dumb to at least aspire to usefulness.

An example.

So let's have a look at just a single line of code (admittedly with a line break inserted by me to make it fit into the blog) from the sort of library I despise. This is the recommended, idiomatic way you are supposed to create an Assertion. Or rather, create a builder that can eventually be used to create an assertion:

SAMLObjectBuilder<Assertion> builder
= (SAMLObjectBuilder) builderFactory.getBuilder(Assertion.DEFAULT_ELEMENT_NAME);

The above code tells you a lot about what's in store for you if you should be so foolish to waste time trying to use the OpenSAML library.

So, the programmer wishes to create an Assertion.

In order to do this he first has to get hold of a builderFactory (not shown in the above code). This in itself is sloppy. Why do we need a builder factory? Under what circumstances would an ordinary user need to even be aware of there existing multiple sets of builders for any given type? This is plumbing sticking out of the walls for no good reason.

Next the programmer has to get a builder for the assertion. The first thing I find interesting is that the designer fails to abstract away the underlying complexity. The programmer has to know the element name of the Assertion. Why? Isn't the point of having a library to help the user deal with the problem domain in a more abstract manner? The fact that the Assertion class defines a handy constant for this does not excuse this. This is a level of detail that the user should not have to worry about. Besides, you will note that the constant holding the DEFAULT_ELEMENT_NAME has "default" in its name so it isn't like the designer didn't have an intuitive sense that there was an expected default. A good designer would have thought about this and concluded that if the default element name is what will be used in 99% of the circumstances, then the programmer should not have to type this in. Ever.

The second interesting thing is this whole SAMLObjectBuilder affair. To get one you have to hand-crank the creation process with casting and the assigning it to a genericized SAMLObjectBuilder type. Why? Would it not have been better to just have an Assertion.Builder type that you simply instantiate and then use? Why would you first want to have a factory, and then design it so badly you have to cast the results it produces? In fact there is a AssertionBuilder that extends an AbstractSAMLObjectBuilder that extends an AbstractXMLObjectBuilder which is defined god knows where, but instantiating it directly isn't the intended way to do it, and if you start to mess with those types then it is going to take you a while, and a few screenfuls of windows with various Java source files, to make any sense of the whole giant mess. There's a lot of structure for structure's sake.

Also, why would you call something a builder when it is in fact a factory? While some literature may be overly vague about what a builder is, good practice is to use builders as an instantiation helper that ensures the produced object is in the desired state. Often producing an immutable object -- sometimes to insulate the programmer from ardous initialization. Assertion has both getters and setters and the builder, near as I can tell, is just used for object creation with the inane buildObject() method. In other words, the whole builder thing is just another bit of plumbing sticking out of the walls.

If you want a good example of builders you can have a look at the API of code generated by Google Protobuffer compiler. Here's a typical example of what object construction looks when you use protobuffers:

Timber.LogEvent logEvent = Timber.LogEvent.newBuilder()
.setTimestamp(rec.getMillis())
.setLevel(rec.getLevel().intValue())
.setHost(hostName)
.setServiceName(serviceName)
.setTid(rec.getThreadID())
.setType("T")
.addPayload(
  Timber.Payload.newBuilder()
  .setContentType("text/plain")
  .setPayload(ByteString.copyFromUtf8(rec.getMessage())))
.build();

In the above code snippet we create a log event builder, set some values, add a nested payload and then construct the needed objects. Understanding how to use this is dead simple. All generated types have a newBuilder() that produces a correct builder without any silly fluff. All builders have setters for their values and adders for nested collections. Setters are chainable. The build() method validates and produces a finished instance. Done. All this takes you about 5 minutes to learn and is so simple you won't have to read the source or the javadoc again for months or years.

It should be noted that we have still only investigated a SINGLE line of idiomatic OpenSAML use, and already we have trodden in a miserable swamp of shoddy design and had to tour the source for answers (which is like peeling an onion: layer upon layer with little or no substance in any one layer and a lot of crying) -- even before actually looking at how you would accomplish anything of use with this library.

Believe me, it gets a lot worse when you do.

You may ask why I am singling out OpenSAML for criticism. Well, why not? I had to pick an example and it might as well be OpenSAML because I have been trying to use it lately -- and have come to the conclusion that it is actually easier, faster and safer to just read the SAML specs and write things from scratch. It is a pain in the ass, but considerably less risky than trusting heaps of code too messy to read. Granted, the SAML specs are a dull read and I still have to deal with heaps of terribly designed XML infrastructure, but it beats wasting even more time on an unhelpful library that fails to hide the details anyway.

Besides, it will mean that I do not force those who come after me to take on the design debt of OpenSAML as well.

I am also using OpenSAML to make a point. It must be okay to point out bad design. It may not be nice, but then again, wasting my time is also not nice. If we avoid naming and shaming bad design, people will not learn.

Now, if you have worked on OpenSAML I would imagine that you are feeling defensive right now. Perhaps a bit angry. You may think that I am unfair because you just did what everyone else did when designing and writing the code. (The idioms found in OpenSAML sure are plentiful in other XML code, though that doesn't make them "good").

If you feel like responding (please don't. I am not interested in hearing a defense for OpenSAML. There really is no point in even trying to defend it): before you respond, I think you should sit down and ask yourself if you are really sure that OpenSAML is helpful to anyone but those who are prepared to a) spend a lot of time understanding the library, or b) are utterly comfortable with cutting and pasting code they barely understand to put something together that apparently works.

Then you should ask yourself if what OpenSAML does is really so complicated that the complex, unfriendly, messy API is warranted. (I can give you a hint: it isn't).

If you think that people should have to spend a lot of time understanding OpenSAML you are wrong and you should probably never write library code ever again.

Hard problems and unreasonable expectations

2011-04-24T16:30:00.005+02:00

After spending a decade in the search engine industry you start to take for granted that people of normal intelligence and above are able to reach their own conclusions on what it means when certain numbers become large enough. This is an occupational hazard; becoming blind to the fact that most people have no way of thinking about large numbers or how hard a particular problem is.

A very common theme in litigation against search engines, or other parties that deal with large corpora of content, is that some party demands or expects that the search engine or service provider curates the content. For instance, that potentially offensive material is filtered out.

There are two main ways to approach this:

Manual curation
Algorithmic curation

Of course, there are many hybrid solutions between these two extremes, but let's explore the extremes to gain some understanding of the bounds of the solution space.

Manual curation.

Manual curation of a large corpus is not impossible per se, but it is unfeasible for very large amounts of data. This seems to be extremely hard for many people to grasp so I am going to provide a handy example in the shape of a very rough estimate.

Imagine that the web indexed by a search engine is 1 billion documents. Now, imagine that it takes, on average, about 10 seconds for a human being to determine if a web page contains offensive content or not (some pages merely need a glance while others require you to actually read the page).
That means that to evaluate the whole corpus indexed you would need 10 billion seconds of work being performed. That's roughly 2.78 million hours. Assuming a 8 hour work day, that is 347,222 work-days. Assuming you work 5 days per week that's 17,361 months. Again assuming you work 11 months per year, that gives us 1578 years of work.

So if you have 1578 ideal workers who can keep it up 8 hours per day without breaks, are never sick and only take 4 weeks of vacation every year, it will take you one year to chew through 1 billion web pages.

So what would this cost? Well, get out your calculator and run the numbers. Fixed wage? Per page? Whichever way you slice it, it is going to cost a lot. Now, is it reasonable to impose this cost on, say, search engines? What does this do to the competitive landscape? (As if building a web scale search engine wasn't hard enough and expensive enough as it is).

And these are extremely conservative numbers. You see, a large scale web index was 1 billion documents 10 years ago. Today the size of a web index is roughly two orders of a magnitude higher and the web never stands still. Not only are new pages added, but older pages are updated. The problem size grows exponentially.

If you do not understand, instinctively, that problems that grow exponentially cannot be solved efficiently using manual labor you are intellectually unfit to reason about these sort of problems.

Any 10 year old knows all the math needed to reason about these things.

Algorithmic curation.

An algorithm is a precise recipe for how you solve a problem. In order to write computer programs that solve a given problem you need a precise recipe -- that can be realized as a program.

Quick, can you come up with a precise recipe for figuring out whether a page contains offensive material or not?

For the vast majority of you the answer is "no" even if you dedicated the rest of your life to this problem. In fact, for the vast majority of the worlds smartest people working in this field the answer will still be "no" if you expect the algorithm to be right every single time. You can get some high percentage of the cases right, but it is unlikely that we will ever discover a method that will always produce the correct answer every time. (If you have a liberal arts background or some other non-scientific degree you may think of "unlikely" as "won't happen ever").

Not least because there is no correct answer to this problem. What is offensive is in the eye of the beholder as well as subject to social and cultural norms. What you consider offensive may even change over time. It can also change depending on context. We are faced with many problems that have this nature when it comes to large scale information processing -- problems that have no definitive answers.

So to sum it up: it is possible to achieve high success rates for these sorts of problems using an algorithmic approach -- but perfection will not be achievable. So we will just have to live with imperfect results.

That, or just decide that not being offended is so important to us that we do not want search engines at all. Or the web.

Postscript.

What is striking when you reason about these things is that you do not need an education in information retrieval, computer science or mathematics to understand that these problems are very hard and that some types of approaches will be completely unfeasible. Yet on a daily basis, ignorant law-makers, judges, lawyers and opinion leaders make statements that seem uninformed by even the most basic understanding of the problem domains they talk about.

And there are a lot of problems that exhibit the same properties as curating a web index in that they have vague problem definitions, involve ultra-large scale, and expectations that can never be met fully. It is going to be a real challenge to educate law-makers, judges, bureaucrats and the public on these issues.

But it is important that it is done. It is important that these things are explained over and over, and in simple terms so that lack of knowledge or a lack in intellectual fortitude can not be used as an excuse for making dimwitted decisions.

More pictures

2011-04-11T18:21:00.000+02:00

Lars Magne thinks my blog has too few pictures. Here's a random picture:

Free advice for Skype

2011-04-11T16:45:00.000+02:00

Lets make this short and sweet: the user interface of Skype sucks.

It is too dominating on the screen and it does not take into account how people actually use similar products. If you look at the closest relatives of Skype, the aggregating chat clients, you will see what I mean. Small, focused on being unobtrusive and usually centered around how people actually use communication software. By comparison, Skype wastes screen real-estate and doesn't really show you that much of what you are interested in.

In fact, I was going to include an illustrative screen-capture, but I decided against it. When thinking about why I realized it was because I didn't want a honking big graphic in the middle of my blog posting. That is how badly designed the Skype UI is.

No, I have no use for a cover-flow-like UI to leaf through my contacts. Copying UI design mistakes by Apple and then, to make matters worse, using them in the wrong context is just stupid. Nor do I have any use for seeing a list of whom I talked to two days ago. No really, I don't.

What I have some use for is to see which one of my contacts are online. Without having to click around and then reshuffle windows. This should be the default view. Not some oddball, tucked-on auxiliary UI wart like now.

One more thing...

What really puzzles me is why nobody at Skype cared enough about the product to ensure that people would have a reason to run Skype all the time when logged in. In case you hadn't noticed: most people don't. They fire up Skype when they "need it". Often initiating the conversation out of band via chat, email or (gasp) phone.

The easiest way to accomplish this would be to make sure Skype is also an IM aggregation client. A good IM client. One that is on par with, say, Adium. With support for all the major chat networks. (If you work for Skype and are not familiar with Adium I suggest you install it now and start using it. It is important you understand why Adium is so popular)

You see, a lot of people would like to have Skype support in their IM client. This is no accident. Because they run their IM clients all the time. They do not run Skype all the time. Because it is an annoying piece of single-use software.

By ensuring that Skype is also a great IM aggregation client you would give people real reason to run Skype all the time. Which would boost the usefulness of the application more than any amount of advertising could ever do.

Attention to detail.

2011-03-18T14:38:00.000+01:00

We recently moved into a new house, which means that all our kitchen appliances are brand new. We opted for an induction based cooker. Cooking with induction is rather neat since it provides very quick and precise temperature control. There is a lot less lag between varying the power and getting observable result.

However, I am not very pleased with the industrial designer's work when it comes to the user interface. The UI for the cooker is a touch interface. They layout is fairly simple, a master on/off and +/- buttons for each surface. Increasing or decreasing power is done by repeatedly pushing the plus or minus buttons respectively. You can also hold down these buttons to turn the power up or down several steps, but this is a rather slow way of doing it.

I don't like that interface. It is slow and fiddly and lacks tactile feedback. But what is far worse is that the designers were shortsighted and incredibly sloppy in designing this. You see, the interface gets utterly confused if water is splashed onto the surface. If it gets bad enough, you simply have to shut down the cooker and restart it. This happens quite often so it is a fairly tedious design flaw.

I am not sure how this got through testing. Any sensible engineer or designer would conclude that this is a serious design flaw, go back to the drawing board and figure out how to attack the problem.

And in all seriousness, it doesn't really take a lot of effort to come up with alternative designs that would work significantly better. Both technically and with regard to user interface. If your industrial designers do not obsess over such details they have no business designing consumer goods.

(A cursory glance at the products of various manufacturers of magnetic rotary encoders suggests that it would be neither complicated nor expensive to design and implement a user interface based on removable, surface-mounted rotary encoders. Which makes you wonder whether these manufacturers take any pride whatsoever in their products).

Quo Vadis?

2011-03-06T18:18:00.002+01:00

Someone recently posed me the following question in an email:

Consider a leading communcation provider in 2014. (Including what is now known as "telcos" if such a thing still exists.) What are the communication functions such a provider will provide? How would you as a user expect the provider to help you with your
daily communication needs?

Rather than attempt to answer it directly, which may or may not be possible, I thought that it might be helpful to understand where we are and what the context of this question really is. The following blog post is a somewhat edited version of the response I gave.

The fundamental problems of communication technologies today is fragmentation and low quality.

Fragmentation

Fragmentation arises from the fact that we have a gazillion services for communicating and there is a rather large space of modalities for communicating (public/private, synchronous/asynchronous, voice/video/text, etc.) Many of these communication facilities are tied to a particular service provider (Skype, Facebook, Google, Twitter) while some are interoperable (telephony, email, Jabber etc).

If you look closely at your social graph you will notice that in order to reach everyone you know effectively, you will most likely have to use half a dozen different communication systems. Some people respond quickly to email, some prefer Twitter, others live on Facebook, some still have land-lines to their house and know nothing of this newfangled intertube stuff.

This is a challenge. Not only because it is inconvenient for the user, but because it is very hard to identify how one can build a one-size-fits-all communication product that reduces the amount of friction and chaos.

Low quality

The other serious issue today is low quality. When we moved from land-lines to first generation mobile networks, we accepted more sketchy availability and a noticable drop in audio quality. GSM was a bit more consistent in areas with good coverage, but still not great. Skype and other VoiP-offerings can sometimes offer superior audio quality, but at the price of even worse availability: you never know if it'll work. Facebook and Twitter offer no meaningful SLA so your message may or may not be delivered.

Standards

A way to address fragmentation is standardization. But standardization is slow, and sometimes contrary to commercial interests. In particular in later years actors in the market are never content with a strategy that makes them a big part of an even bigger ecosystem: they want to own the ecosystem. and since everything is in flux these days, there are a lot of companies jockeying for dominant positions. (Perhaps the most promising may be Facebook since they have 600M users by now (or nearly 1/10 of the world's population). With that kind of reach they have the potential to have an enormous impact in whatever they choose to do, standards be damned: when you are this large you can unilaterally set the standards and force everyone else to follow).

Also, there is standardization and then there is standardization.

The telco way of standardizing is dead: a very formally driven process that takes many years to complete, accumulates vast complexity and then leads to different players adopting different interpretations of the "standard" may have worked when we only had to deal with a market partitioned neatly into countries. But that world no longer exists. The market is global.

There is also the "Internet way" of standardization: RFC's ratified by the IETF. In order to create an internet standard you need at least two implementations of which at least one must be openly available. in other words: it is an implementation-driven approach. this cuts down on complexity considerably and tends to favor simpler, more compact and more implementation friendly standard. (It is actually easier to show why this is so by looking at the antithesis of these standards. Things like X.400, X.500 and SGML for which no fully compliant implementations exist because these standards are hopelessly complex and immensely huge).

Global market

Any communications player not thinking of everything they do in terms of a global strategy is wasting their money and time and will eventually disappear. This applies to all aspects of what telcos do today. (I start every meeting I have with people by asking them if they have a global strategy for their project. Almost everyone admits that they either have none, or they are "perhaps thinking about expanding to neighboring markets". This is frightening and essentially means that money is being wasted on doing things over and over again with very little impact beyond a single market).

Pace

Technology and trends are moving faster every year. The iPhone is about 4 years old now, and we are on the FOURTH generation of iPhone and the third or fourth generation of Android phones (how do count generations on Android?). More has happened in the handset world in the past 4 years than in the entire history leading up to 2007.

Traditional handset vendors are struggling to keep up. Some do a little better than others, but the big winners are not the same companies that were dominant just 4 years ago. In particular Nokia provides a frightening example of how quickly a success can turn into a sob-story.

As for telcos: they were caught out. Telcos have had the privilege of operating in a maturing, then a mature, business. This means that they have been able to optimize away many of their traditional functions. Such as engineering, research and development. This would have served them well if the world had had the decency to stay constant. Unfortunately, this is not the case, and they desperately need engineering muscle.

Telcos need a strong ability to do research and development and to have the ability to make stuff. This is the antithesis to the core values held in high regard in many telco operations.

The reason for this is quite simple: if you look at the sort of companies that pose the greatest threat to telcos, they are all technology companies. Facebook is a technology company. Google is a technology company. Amazon is a technology company. Companies that have recognized that it is neither fast nor cost-efficient to be entirely dependent on vendors. Companies that have recognized that they have to own the key enablers of their services. Companies that have strong internal engineering cultures and where engineering talents go to create and build stuff.

In the time it takes a typical telco to decide that they want to build a particular service, the more nimble and capable among their "new" competitors will already have done it and rolled it out. Globally. And by the time telcos are able to squeeze out a mere baseline product, the competition is already on their third or fourth iteration and the goalposts will have moved further afield.

Postulates

Reducing percieved (not necessarily actual) fragmentation is an obvious way to approach the communication challenges of users today. However, this is very, very hard to do well. It can't really be done effectively in back-end systems alone, but must primarily be addressed "at the edges". In other words: applications and apps that aggregate -- possibly with some back-end support systems. The problem is: the services you want to aggregate do not necessarily want you to aggregate them. They want to own the user.

Telcos have a diminishing role in providing infrastructure. In particular service infrastructure. The service infrastructure that is offered by most large Internet players is not based on off-the-shelf systems. Telcos have optimized away their technology capabilities. This made sense in the 90s and early 00s -- but today it puts telcos at a perilous disadvantage. While telcos still provide considerable amounts of low-level connectivity (ie. bit-pipes), this connectivity can easily be replaced by other players. Things are tending towards higher speeds at lower cost so good luck making lots of money there in the future. In my opinion telcos must:

Cultivate a strong engineering capability
Accept that it will take years to (re)build engineering capability
Be willing to accept failures, learn, get up, and try again
Take ownership of key enablers
Understand Internet-scale engineering.

The game is "global or nothing". If you are in the communication business and your product, or its underlying infrastructure, is based on the notion of a local or national market, you are wasting your time and shareholder money. This applies to everything. From the user-facing services themselves to everything that need to be in place under the hood to make it happen (user management, billing, payment etc) to how the organization operates. Everything.

In essence, "everyone" knows that the world the traditional telcos address is slowly disappearing. Everyone knows that at the end of an industry maturity cycle, engineering capability becomes vital. The business-school literature exists and there is a plethora of examples of how once dominant players either failed spectacularly or faded away. (I'm not an MBA so it really shouldn't be my job to educate people on these things ;-)).

This piece doesn't offer that many answers, but it provides some pointers to how I perceive the current context.

The loss of a very special place?

2011-02-20T19:25:00.007+01:00

I've been on quite a few trips to the Nürburgring and to me, the Ring is a very special place. Not only is it perhaps the most exciting race track I have ever visited (or driven on), but it is a historical site. All the greats have done battle on the Ring.

You don't so much go on a trip there as go on a pilgrimage of sorts.

In addition to the track, there's the people. The guys at Pinocchio, who always greet us when we visit and seem genuinely happy to see us again. There's Christa's where we always eat schnitzel on the evening when we arrive. Die Pistenklause where we stop by to lie and brag about our driving accomplishments and drink beer. Sonnenhof, where we usually have our briefings or dinner the night before we venture out on the track. There's Ron Simons' company RSR Nürburg, where we rent cars. And of course, there's Haus Marvin and Pension Diana where we usually stay. You know people by name, you stay at small places run by families, and you enjoy a personal level of service, and you come back year after year to the same places because it is good to see these people again.

But of course, we would never be there if it wasn't for the Ring.

It would appear that some people have recognized the pull of the Ring and that naive politicians have bought into the idea that the best way to benefit the tax payer is to commercialize the Ring in wichever way possible. To hand it over to private interests to package it and sell it.

Well, I am to a large degree the demographic they want to attract, and I can only say that I do not like what I am seeing. What I have read over the past week about what is going on at the Nürburgring has made me wonder if I will ever go back again.

I am not interested in a sanitized, packaged, watered down experience. I am interested in two things when I am at the Ring:

Driving as many laps as I feel like driving around the Nordschleife
Enjoying myself in various cosy establishments run by people from the area.

I am not interested in staying at some large hotel. I am not interested in visiting shops. I am not interested in watching static exhibitions of dead objects when just outside the door there is one of the greatest racing tracks of all time. I am not interested in paying lots of money to be ferried around the track one or two laps. I am interested in what is genuine. I'm interested in experiencing the track. And there is only one way to experience it: you have to drive on it.

If they want to turn the whole thing into a theme park: I'm sorry, but I'll just as well stay at home and not visit the Ring. Because watching what was a really nice place, with lots of nice people running small businesses being torn to pieces by greedy businesspeople who have absolutely no understanding of the cultural significance of the Ring, the people who live and work in the area and what it represents to people like me is just too painful. I'd rather not witness that.

As an aside, just look at F1. I've visited F1 races. Perhaps my most exciting live F1 experience was at the Hungaroring. Not the greatest of race tracks and the facilities are...of a somewhat limited standard. But you know what? It was real! The wooden bleachers without any sort of roof may not have been the most comfortable, but the closeness to the track and the spirit of the people around us elevated the sense of genuine enthusiasm and fun. After the race I was wet, muddy and immensely pleased. We stayed a few days extra to walk around Budapest and spent our evenings in various restaurants. I often think about it and when I get the time: I'll return and hope to have much of the same experience.

In comparison, I have been to one of the more modern tracks. Large, modern multi-billion dollar facilities designed to cater to ... I don't know who, but definitely not people like me. It sure as hell wasn't racing enthusiasts they were catering to. The whole experience was numb, dull, impersonal, and utterly uninteresting. It felt fake. I wasn't so much taking part in an event as just paying to be in the presence of lots of people who didn't even seem to be paying any attention to the race. I will never go back. I have zero cherished memories from that trip. None.

If the german politicians want to extract the maximum possible value from the Ring, they must recognize that they can only do so by not screwing with the formula: the accessibility of the Ring is the key. Anything that limits access to the Ring makes it less interesting to prospective tourists. And if people do not come, nobody wins. Customers don't win, the people of the surrounding villages don't win, the german taxpayer definitively doesn't win and the whole region is worse off.

Because we'll take our money and go elsewhere.

As I mentioned before, I don't know if I will return to the Ring if what I used to come for is no longer there. I was going to visit the Ring this spring, but I will have to talk to some people to find out if we really want to.

We've been talking about possibly taking a trip to Spa Francorchamps instead. After 7 years of faithfully visiting the Ring and leaving behind a ton of money at various local establishments, we are considering our options. Spa has a great race track, but I don't know the area well. Perhaps, if we find a nice place to stay and a company to rent track-prepared cars from, we will go there instead.

But if we go there it will not be to prance about the place like a bunch of tourists. It will be to study the track, drive, and to drive as much as we can possibly afford.

I hope that the Nordschleife is not lost. I hope that the politicians will take charge of the situation and not allow misguided commercial interests ruin the livelihood of so many locals and ruin a cherished travel destination for even more people all around the world.

Do not forget why people come to the Nordschleife.

Mythbusters and the lunar X-Prize

2011-02-19T13:59:00.000+01:00

While reading about the Google sponsored Lunar X-Prize I got an idea: It would be great fun if the Mythbusters dedicated a whole season (or three) to having a go at sending a rover to the moon?

As many education professionals have pointed out, Mythbusters is probably the most important TV show right now. Although much of the show is a lot of fun and games and things being blown to bits, at the core, the show is firmly anchored in the application of scientific method and problem solving. While there have been several TV series that attempt to teach its viewers something about science, this show goes a bit further in that it focuses on actually performing experiments rather than talk about them.

There is also a sort of renaissance these days of people making and building stuff. Lots of people build exciting things in their basement -- from autonomous vehicles to 3D printers to clever electronics gadgets.

The efforts towards private space exploration over the past couple of decades are sort of exciting, but not very engaging. While the results that have been achieved are vaguely interesting to those of us not directly involved in the industry, these efforts do not really engage us as much as they could. If Branson and Rutan manage to perform a sub-orbital flights for rich people, well, that is of course nice for them, but it isn't exactly engaging and inspiring. The really exciting stuff of making it happen takes place behind cosed doors.

I think that if Discovery Channel and M5 Industries were to head up an effort to send a rover to the moon, that would make for absolutely fantastic TV. Of course, I don't expect the producers of Mythbusters to have the sort of cash needed to make it happen, so they would probably need to secure funding for the project from companies and well-off individuals.

Given their high profile, the Mythbusters team might also be able to enlist the help of volunteers with special skills from various engineering disciplines and conduct a large part of the effort as a sort of open source project.

Of course, I don't expect Discovery Channel to pick up the challenge. But it would make for really great TV.