The Netflix Prize has a winning entry. There are i's to dot and t's to cross, but there is now an entry that has achieved the 10% improvement over Netflix's existing system that the prize demanded.
Judging from the team name, the winning entry is a joint effort between three leading teams: BellKor, Pragmatic Theory, and BigChaos.
Market Rebels: How Activists Make or Break Radical Innovations, Hayagreeva Rao, Princeton University Press 2009
For decades, economists have extended their intellectual reach beyond mere money in an attempt to encompass all the social sciences in their analytical framework. But now the boot is on the other foot and it looks like even core economic observations may be better explained by other social sciences. Robert Solow apparently said that attempts to explain differences in economic growth across countries typically end in "a blaze of amateur sociology". The focus on psychology in explanations of the banking crash shows that growth is not the only area of economics where the discipline runs out of steam before reaching its destination. The rise of behavioural economics, surely a last-gasp attempt by economists to match their models to the real world without changing departments, suggests that the condition goes deep.
Despite its title, Hayagreeva Rao's Market Rebels (Open Library link, publisher's page) challenges the economic analysis of innovations. At 180 pages and full of case studies it's easy to read quickly, but I was so taken by it that I went through it a second time and found much that I had missed. Rao does not hammer the reader over the head with the implications of his case studies, but for me as a non-sociologist and non-economist the implications are huge and I'll be thinking about the book for a long time.
The case studies are diverse, but are centered around a single claim: the "joined hands of activists" play an important part in the creation, diffusion, and blocking of innovations. Collective action matters. Rao describes how hobbyists were key to the cultural acceptance of the car and the development of the personal computer; how microbrewers brought diversity back to beer; how nouvelle cuisine grew from the rebellious student movements of Paris 1968; how shareholder activism has pushed large companies to change behaviours; how community activists attempted to stall the spread of chain stores and then of big-box stores; how the green movement blocked the development of biotechnology in Europe. These studies, many based on his own research, help to bring activist groups and their campaigns out from the wings and into the spotlight as we think about innovation and social change, and by doing so Rao is performing a valuable service.
The book is not strong on systematic analysis. The closest he gets to describing what determines whether movements succeed or fail is that successful movements must create "hot causes" and "cool mobilizations". Both concepts are tied in to the concept of "identity", which for Rao is the underlying motive that causes people to join with or against social movements. A "hot cause" is the spark that successful activists use to light a fire. It's a lightning-rod incident or issue that arouses strong emotions such as pride or anger. Examples include the frustrated demonization of "big beer" by real ale enthusiasts; the outrageous bonus paid to Home Depot CEO Robert Nardelli which crystallized the shareholder rights movement. But "hot causes" by themselves are not enough for a prolonged campaign. "Cool mobilizations" are actions that keep a movement going forward by "engaging audiences in new behaviors and new experiences that are improvisational and insurgent". Examples include setting up a microbrewery or formulating shareholder resolutions. The concepts are useful, but it's a shame Rao doesn't have a stronger turn of phrase. Naming concepts can be key to owning them, and "hot cause" and "cool mobilization" are too literal and clumsy to take on the weight they need. But this is a detail (and I don't have better ideas).
For someone who has spent most of their non-fiction reading time reading economics and economics-inspired books in recent years, Rao's is a welcome and refreshing change. Economic analysis too-often reduces the political left-right split to the false dichotomy of market vs state, but this reduction maps badly on to the real experience of political activism. Those who protest Monsanto's private-sector use of genetic engineering are often the same as those who protest state-driven wars. Many of those who oppose new Wal-Mart stores also oppose the extension of surveillance powers by the state. Where do such activists see themselves in a market vs state debate? For many, they don't: market vs state is not what it's about. So it's not surprising that economists have a blind spot when it comes to social movements, and that the discipline systematically minimizes their impact. By putting social movements at the centre of his stories, Rao shows that they can and do have an influence, and that they deserve a place in any serious look at institutions that shape social change.
Although he says almost nothing about the Internet and digital collaboration, Market Rebels' focus on innovations makes the book obviously relevant. Rao's analysis is a welcome alternative to the usual focus of widely-read writers like Yochai Benkler and Clay Shirky. These writers take the economics point of view and focus on issues such information as a public good, lowering transaction costs for online exchanges, and the vanishingly small marginal cost of reproduction of digital information. Rao's unspoken counterargument, which convinces me, is that group formation is not a problem of information, it's a problem of identity. If he is right then although we can expect to see many examples of successful groups in the online world, we won't see not a huge flowering of groupiness compared to the information-starved analogue world.
What's more, if Rao is right and initiatives such as Wikipedia, blogging and the Open Source movement really are social movements, then they may have a limited lifespan. Digital activist identity is a rebellious and anti-establishement stance, but such a stance can only be maintained while the movement is oppositional. Open Source, with a longer history behind it than Wikipedia or blogging, has changed its identity to being more professionalized, more establishment. Its participants are less likely to be the radicals of yesteryear. Once Wikipedia is established as a success for a few years, rebellion becomes irrelevant as a motive and one may wonder whether the activists will find the "cool mobilizations" to maintain involvement and participation.
If there is an economics tie-in with Rao's analysis, it's with the analysis of identity pioneered by Rachel Kranton and Robert Akerlof (and which I sketched in the final chapter of No One Makes You Shop at Wal-Mart). Rao does little to pick apart the concept of identity and it looks to me like the K&A analysis would have been helpful to him. For Kranton and Akerlof, identity is a set of social categories (car enthusiast, green activist), a set of prescriptions that go along with those categories, and a set of costs and benefits associated with following or not following these prescriptions. We each choose an identity from the range that society provides ("environmentalist", "conservative", etc). Forcing this choice is the object behind Rao's "hot cause": the lightning-rod issue that polarizes participants into supporters and opponents ("which side are you on, boys, which side are you on?", as the greatest living American sings). Once you have chosen an identity, you must affirm it by following the prescriptions associated with that identity (shopping at independent stores, eating nouvelle cuisine, etc) or you pay the price of dissonance if you take actions that go against those prescriptions (shopping at Wal-Mart, eating classical cuisine). Creating a strong set of such prescriptions is the essence of Rao's "cool mobilizations", which serve to maintain a sense of solidarity and identity among movement members.
One of the more common criticisms of No One Makes You Shop at Wal-Mart was that, although I argued in favour of collective action as a corrective force to free markets, I had little to say about what forms that action should take. It's a fair knock, and I'm happy that I can now point such readers to Rao's book. Not only does he take on several of the issues that I cover (Wal-Mart and big-box stores, biotechnology, real ale) but he takes them much further than I could ever had done, and in a wonderfully specific and constructive way that provides concrete guidance to activists. I take my hat off to him.
I've learned two lessons in the last couple of days.
First, if you want to get some attention for a blog post, call it something eschatological like "Online Monoculture and the End of the Niche". If I had called it "Simulation of a 48-product market under simplistic assumptions" somehow I don't think I would be writing a follow up. I don't like this lesson much. But I don't feel too guilty: if I was really trolling for traffic I could have called it "Learning from the Big Penis Book" [see Music Machinery for why].
Second, no matter how hard you try to be clear, many people don't get what you are trying to say. So maybe it's not their fault. For examples, see some of the comments here and here and even a bit here and on the original. The main complaint is that picking two example runs from a simplistic simulation of a small system with a small and fixed number of customers and products doesn't simulate the entire Internet. Where is the statistical sampling, the exploration of the sensitivity to parameters, the validating of the recommendation model? And on and on.
These folks don't get why people do simple models of complex things.
The goal of simulations is not always to reproduce reality as closely as possible. In fact, building a finely-tuned, elaborate model of a particular phenomenon actually gets in the way of finding generalizations, commonalities, and trends, because with an accurate model you cannot find commonalities.
For example (and I'm not comparing my little blog post to any of these people's work), in chemistry, Roald Hoffmann got a Nobel Prize and may be the most influential theorist of his generation because he chose to use a highly simplified model of electronic structure (the extended Huckel model). It is well known that the extended Huckel model fails to include the most elementary features needed to reproduce a chemical bond. Yet Hoffman was able to use this simple model to identify and explain huge numbers of trends among chemical structures precisely because it leaves out so many complicating factors. Later work using more sophisticated models like ab initio computations and density functional methods let you do much more accurate studies of individual molecules, but it's a lot harder to extract a comprehensible model of the broad factors at work.
Or in economics, think of Paul Krugman's description of an economy with two products (hot dogs and buns). Silly, but justifiably so. In fact, read that piece for a lovely explanation of why such a thought experiment is worthwhile.
Or elsewhere in social sciences, think of Thomas Schelling's explorations of selection and sorting in Micromotives and Macrobehaviour, or of Robert Axelrod's brilliantly overreaching The Evolution of Cooperation, which built a whole set of theories on a single two-choice game and influenced a generation of political scientists in the process. All these efforts work precisely because they look at simple and even unrealistic models. That's the only way you can capture mechanisms: general causes that lead to particular outcomes. More precise models would not improve these works - they would just obscure the insights.
That said, there are valid questions. Under some circumstances, aggregating large numbers of opinions into a single recommendation can give this odd combination of broader individual horizons and a narrower overall culture. Are there demonstrable cases of the monopoly populism model out there in the wild (aside from the big penis book)? Is this a common phenomenon or an uninteresting curiosity? Well I don't know. I do think so, obviously, otherwise I would not have written the post. But it's a hunch, a hypothesis, a suggestion, that I find intriguing and which I may or may not try to follow up. Hey, it's a blog post, not an academic paper.
[This would be the thirteenth episode of Mr. Amazon's Bookshop, if it were an episode. The previous episode is here. A list of all episodes is here. In the previous episode, Kylie and Edmund vanished from Whimsley Hall after finding out that Mr. Amazon's recommendations were doing nothing to help sell Kylie's story The Adventures of Wazzock. Whimsley sunk into a brief depression before being roused by Jennie the one-legged housekeeper, who sent him into the village just in time to see Kylie and a pack of teenagers heading towards Mr. Amazon's Bookshop with trouble on their minds.]
I wrote the first ten episodes of this story in a couple of weeks around Christmas, when I had some time off work and the story seemed fresh and interesting. The two most recent episodes have been written on weekends in and around cooking, cleaning and so on. And now we have all the principals with the exception of Jennie the one-legged housekeeper - Whimsley, Google, Kylie, Edmund (who is in the crowd, near the back, though you might not have noticed him), Mr. Amazon - in one place, and something dramatic is obviously about to happen. But what?
Now I don't want to go all John Fowles here (see the book in the title) and pontificate about the nature of the plot as artifice and the role of the author. That's so last century. Plus, when Fowles wrote that he could send his heroine this way or that, and when he wrote his alternative endings, he had actually finished his book, so the questioning of his role as author was a little precious.
My problem is different. I don't have two ways to finish the story, I have none. In fact, there is no story from here on.
I have some ideas about things that might happen next, including what seems to me a rather neat twist about the obviously impending assault on Mr. Amazon's Bookshop, but they are vague and they have problems. Not least among them is my growing tiredness with Whimsley himself. His one-dimensional nature was a benefit in Mr. Google's Guidebook (written just over a year ago! how time flies), but after stretching him over 15,000 words rather than 2,000 I am not sure there is much more to be said about the internet from the point of view of a delusional alcoholic gothic anachronism.
I did think Kylie may provide an alternative focus for the story, but she has fallen somewhere between a youthful village mob-boss, a smart and witty kid, and a yokel, without one aspect of her character really crystallizing in my imagination.
And then there is the technology issue. Clearly the Kindle has to come in to the plot somehow. Having Google already on scene means that a last-minute twist driven by the Google Book Settlement might be a nice way to finish off the tale. Perhaps Kylie drives Amazon out of town only to hand Google her stories for him to distribute? And I am sure that after all this it will turn out that there are indeed mole-people, but that they are working for Google, not Amazon, slavishly copying all those books for his ever-expanding guidebook.
So we will see. I realize this raises questions about the responsibility of the author to finish a story once started. I would like to finish it, and I expect that I will do so. But there will be no episode this week.
This is not a fish for attention and reinforcement, by the way. I have appreciated the comments on earlier episodes. And to prove that it is not, I will close comments on this posting. It's just an observation that I have other things going on, like work and family and friends and books and spring, and I have no episodes stocked away ready to be brought out and posted. If anyone wishes to continue the story on their own blog, feel free: I will be happy to link to it.
So with that, see you next week, with luck.
[This is the twelfth episode of Mr. Amazon's Bookshop. The previous episode is here. A list of all episodes is here. In the previous episode, Kylie and Edmund spent a morning running Whimsley's differ and made some disturbing findings about the Mr. Amazon's recommendations. Then Kylie left without warning to go and steal some lunch, leaving Whimsley even more bemused than usual.]
After the excitement and confusion of Kylie and Edmund's investigation, evening found me tired and emotional. My head was spinning with all the numbers they had collected and what Kylie seemed to make of them. You may not believe this, but I was feeling a little out of my depth. At sea even. We gentry typically have little use for numbers bigger than a dozen or so; anything more and we have hired help to deal with them. And though my interest in the differ had got me more familiar with numerology than most of my class, I had always been more enthusiastic about building the device than actually using it.
I had written Kylie off as riff-raff when I first met her, but I was beginning to realize that some of these youngsters may actually know a thing or two. Her outburst about Mr. Amazon's unfairness carried the mark of immaturity and self-dramatization so characteristic of her class and sex, but as I pored over the notebook I really did not know what to make of the reams of Edmund's writing in my notebook. Yesterday Kylie had showed me one page of summary and claimed that is showed Mr. Amazon's populist claims to be fraudulent, but as I turned page after page of my notebook I could make neither head nor tail of the admirably neat tables and lists that I saw in front of me. My first thought was that it would be embarrassing to have to ask for more detailed explanations, but then I realized that numerology is really a matter for the trades, not for men of standing, so there is no shame in being a little foggy when it comes to details. My neighbour Mr. Belloc was correct when he told me of his friend's fate:
Lord Finchley tried to mend the electric light
It struck him dead, and serve him right.
It is the duty of the wealthy man
To give employment to the artisan.
I had not heard of this amazing test until seeing a mention in the morning paper, but it has been around since 1985. That was when Liz Wallace told it to her friend Alison Bechdel who put it in the comic strip Dykes to Watch Out For. See here for the comic and here for discussion.