A Review of Wrong: Why Experts Keep Failing Us – And How to Know When Not to Trust Them, by David H. Freedman, New York: Little, Brown and Company, 2010. 304 pps. $25.99. ISBN-13: 978-0316023788

The market for books about how ordinary people make thinking mistakes being fairly saturated (Predictably Irrational, Sway, Nudge), it makes sense that someone would turn the spotlight on a group that’s supposed to mess up less than the rest: experts. Journalist David Freedman walks us through an impressive list of false and conflicting claims made by experts in a variety of fields that really drives home the dubiousness of much – if not most – of what passes for expert wisdom. The book is worth this carefully assembled and annotated collection of dueling truth claims alone.

There’s plenty of blame for our expert misinformation to go around, says Freedman. From respected scientists to financial wizards to self-proclaimed relationship gurus, people whom we credit with specialized knowledge conduct sloppy research, suppress disconfirming data and leap to unwarranted conclusions. Journalists oversimplify and misrepresent study findings. Bad advice thrives in part because the public demands easy fixes that are “resonant, provocative and colorful.”

We would expect Wrong to cover famous cases of expert fraud like the South Korean human embryo cloning scandal, and it does. Most of the expert errors documented here, however, are not intentional, but originate in the cognitive biases to which everyone is prone. Like the rest of us, experts have sharper eyes for data that supports their hypotheses, claim to have started out looking for what they eventually found, and play to their employers’ metrics (research funding agencies). Nor are peer review and other forms of self-regulation much of a remedy: Thomas Kuhn showed some fifty years ago how the practices of scientific communities reinforce and perpetuate prevailing paradigms.

It would be unremarkable to learn that experts sometimes make mistakes, but if Wrong is right, the magnitude of the problem is much greater than most people suspect. The findings of two out of every three published medical studies fail to hold up. When you consider that the truth claims of less highly-educated and credentialed experts like the inventor of the latest diet or management fad are on average even less reliable, you realize we’re awash in untrustworthy advice.

Books written to hammer a single point are vulnerable to overstating their case, and Freedman’s expert targets are sometimes damned if they do and damned if they don’t. For example, he criticizes mainstream scientific research for careless procedures and small, unrepresentative samples but later defends junk science because, even though its procedural rigor and signal to noise ratio are typically even lower, it occasionally stumbles upon a nugget missed by the pros. That’s having it both ways. Similarly, he fails to take his own advice against retrospective sense-making when he says the warning signs in a famous case of scientific fraud were “glaringly obvious” when looked back upon.

Freedman knows his task leaves him open to the charge of begging the question: the same lack of certainty he says accompanies expert judgments must also apply to his own assumptions that particular expert claims are wrong. I agree with his defense that, though any individual claim may be mistaken, accumulating and pooling evidence allows us to converge on the truth. A historical progression of geographic maps provides an example: early maps of the world are wildly discrepant, but the shapes of the continents and details in their coastlines gradually converge and stabilize across cartographers over time.

So common are the serious errors catalogued in Wrong by even the most eminent researchers and institutions, though, and so influential are false claims in directing the flow of dollars and in propping up whole industries and reputations, that by the time you reach the simple guidelines at the end of the book for knowing when to suspect that an expert opinion might be wrong, it feels the equivalent of being advised to move a foot inland from the beach to protect against tsunamis.

Wrong’s abundant examples of how experts fail us demonstrate how complex the world is. The contingencies of cause-and-effect relationships can be many and difficult to trace, and good advice in one situation can lead to disastrous results in another.

Unfortunately, directing experts (or the middlemen who report their work) to qualify their truth claims with all of the ways they could be wrong, and publicizing negative findings as well as positive ones, are no solution. That just adds to the pile of stuff we have to sift through, and makes it harder for our brains to fulfill what is perhaps their most important function, and is the main reason we rely on experts in the first place: information reduction; sorting through the noise for the signal. The best we can hope to do (whether expert or layman) is narrow the confidence intervals of our predictions a little, and be wrong a little less often.

On the surface, Wrong is about the untrustworthiness of expert advice, but it has much deeper implications. As William James observed a century ago:

Truth lives, in fact, for the most part on a credit system. Our thoughts and beliefs “pass,” so long as nothing challenges them, just as bank-notes pass so long as nobody refuses them.

Even though experts might eventually converge on the truth about specific clearly defined causal relationships, across the multitude of main effects and complex interactions that each of us experiences in a typical lifetime, the saying, “the truth will out,” is false. At any given time, a substantial number of our individual and shared beliefs about the causes of those effects are simply wrong. We’re guaranteed to take to our graves false conclusions about why this diet did or didn’t work for us, why that relationship went sour, or whether our grown child wouldn’t have committed spousal abuse if only we’d spared the rod.

The degree to which the truth doesn’t “out” is the latitude that experts – and anyone, for that matter – have to construct social realities with impunity. For the objectively false claims described in Wrong and others we hear every day to have the power they do to launch movements, sell products, determine government policy and distribute social rewards, being right doesn’t matter as much as being accepted.

That, I think, is the real lesson of Wrong.

This review appears in the July 14, 2010 eSkeptic newsletter, and will be published in Skeptic magazine Vol. 16, No. 2.

You may have heard about Google Wave, the Next Big Thing in online collaborative tools, if the Internet giant has anything to say about it.  It’s an ambitious and somewhat complicated way to coordinate and record multimedia collaborations.  When Slate columnist and author Farhad Manjoo took Wave for a test drive, he was put off by the fact that others could see his individual keystrokes as he typed in it.  Unlike other text-based applications (like instant messaging), which don’t send anything until you hit Enter.

Few of us type perfect and final sentences every time.  Sometimes we hit the wrong key; sometimes we back up and change words.  If you’re IM-ing and type, “That guy is a nut case unusual,” your recipient will see only the post-editing version, but if you’re typing in Google Wave, other participants will see the whole thing as you type.

Manjoo felt like the application was allowing others to look into his head while he thought.  We picture ourselves having little editor-censors that allow us to think whatever we want, but get to approve the final, expressed version.  In fact, our processing of information is continuous and reiterative, such that the pronouncement of the contents of an information stream at any point as “final” is arbitrary and misleading.  The figure below plots different kinds of cognitive output by the amount of processing that typically goes into each.

The process (or more properly, one cycle of it) begins in thought, where we assemble fragments of ideas into a coherent expression.  Once we’ve chosen the initial words, our fingers start to type, although we’re still building the expression and could change our minds, which is what makes watching a Google Wave comment under construction feel like peering into the skull.  Speech is only a little less granular: e.g., there are nine letters in the word “objective” but only three phonemes.  Multiply the cognitive processing we put into an instant message several-fold and you have the editorial control we typically exercise over blog posts, which are longer and have more elements to coordinate.  As with IM, we don’t hit Enter and publish them until we’re satisfied with the whole thing.

The two rightmost examples go extra-cranial and pool our processing power with others’.  An email thread between a pair of correspondents trying to figure something out has the benefit of both repeated iterations and two different perspectives to refine its conclusion.  An even greater number of authors, and more deliberation and verification, are often (but not always) involved with the creation of a Wikipedia article. 

The output of cognitive processing is typically a representation of objective reality; a truth claim about that reality.  That’s mostly what we think, talk and write about.  On the surface, it might seem that the more time and effort we put into building that representation, and the more heads we can combine to gain the benefit of additional brainpower, the more likely we are to “get it right” – to end up with an expression that faithfully mirrors the objective fact it represents.

But it often doesn’t work out that way.  Additional processing might allow us to crunch more data and catch and correct some cognitive errors and biases, but it can reinforce and magnify others.  If we’re communicating with other people, our goal may not be to collectively discover what’s objectively true, but to manipulate belief.

Even when a thinker with a particularly keen intellect labors over and eventually wins acclaim for solving a problem, the last word may have yet to be writ.  Einstein, Freud and Wittgenstein all repudiated significant portions of their earlier work later in life. Which doesn’t necessarily mean they weren’t right the first time.

Our beliefs and communications hover around the objective truth, but there’s no necessary relationship between how much cognitive work we put into them and how veridical they are.  Additional processing may bring us closer to that truth, cause us to overshoot it, or allow us to steer around it.

We’ve been hearing the word “bubble” a lot in the past year, thanks to the global economic crisis.  The stock market has been prone to bubbles as long as it’s existed; the principle extends to other markets like real estate.  Bubbles happen when buyers overestimate the intrinsic value of whatever they’re buying.  Continuing the metaphor, the excess valuation is sometimes called froth.

Market bubbles are a classic (if not the classic) example of a disconnect between objective and subjective reality.  A commodity’s intrinsic value is its objective value.  Market prices are subjective – they represent the value people believe the commodity has.  The perceived value typically fluctuates around the intrinsic value, pulled either higher or lower by rumors, market buzz, and other belief trends that misrepresent objective facts.

Within a bubble, beliefs are king, and overrule the objective facts they (mis)represent. Suppose the intrinsic value of an ounce of silver is US$15 but a rumor drives the market price up to $30.  It seems trivial to observe that buyers of a fixed amount of silver will see their brokerage accounts drop by twice as much as if they had paid a price based on its intrinsic value, but this fact underscores the power that beliefs have to impose objectively real effects within an information community, even if the beliefs themselves are not objectively true.

But bubbles can, and often do, burst.  False beliefs about the value of a commodity, whether it’s silver, mortgage-backed securities, or something else, are eventually (though not inevitably) “outed,” resulting in a correction of the price to one closer to its intrinsic value.  Through this continuous and iterative process, objective reality keeps our beliefs from straying too far afield and reins them in when they do.

The term “reality bubble” has generalized beyond its original market-based meaning.  The principle really applies to any situation where a person or group of people succeed in sustaining (for a time) a belief inconsistent with objective facts.  The reality inside the bubble is what I call beta reality – beliefs are the “facts” of this reality, and their effects (the actions taken by people based upon them) are the equivalent of the effects of objective facts as enforced by the laws of physics in objective (alpha) reality, which is outside the bubble.

One popular example of a reality bubble is Apple’s Reality Distortion Field (RDF).  The term (borrowed from Star Trek) refers to Apple founder Steve Jobs’ rare ability to create, through charisma and persuasion, the belief that Apple product breakthroughs are more substantial than an objective comparison with competing products would support.  Apple has skillfully leveraged this perception into what numerous polls have proclaimed the world’s top brand.  Multiplied across a market, it’s a belief inflection that translates into billions of dollars of extra revenue.  That’s a hefty beta effect.  And this bubble is not likely to burst anytime soon.

Groupthink is another phenomenon that creates reality bubbles that can burst with disastrous results.  Small belief communities are vulnerable to social pressure to sacrifice cognitive diversity for the sake of group cohesiveness.  This impairment of the ability of the group to construct accurate representations of objective reality is widely acknowledged as largely to blame for the failed Bay of Pigs invasion, the explosion of the Challenger space shuttle, and the failure of General Motors executives to recognize shifts in car buyer preferences.

The Rule Not the Exception

Reality bubbles are talked about like they’re anomalies or exceptions.  On the contrary; the manifold nature of objective reality and the limitations of human cognition mean our mental representations of that world are rarely spot on.  They usually miss the mark in one direction or another, to one degree or another.  Their distance from their objective referents traces the outline of a reality bubble.  Our individual and shared beliefs anchor our beta realities until and unless the bubble bursts and the belief is corrected (see The Correction).

Reality bubbles are thus the rule, not the exception; they characterize the fuzzy accuracy of human information processing.  Each of the following is a snapshot of a reality bubble bursting.  In each case, a false belief and its effects reign for a period before being overruled by the objective world.

• I learn from a local appraiser that the Ming vase that’s been collecting dust in my attic is a knock-off worth only $50, not $5,000 like the one I saw on Antiques Roadshow.  Too bad I already spend the money in anticipation of my windfall.
• A group of music fans stoke each other’s anticipation as they drive to a concert, only to discover upon arriving that the headliner came down with strep throat and had to cancel the gig at the last minute.
• Officials at the Dutch national museum learn that the moon rock that has thrilled many visitors over the decades it’s been on display is actually a piece of ordinary petrified wood.

We typically don’t know we’re in a reality bubble until it bursts.  That means that as long as we’re in one, our beliefs that are false in alpha (objective) reality are true in beta reality, and have all the (belief-based) effects of beliefs that are objectively true.

But some reality bubbles never burst.  You may never know if that autographed Babe Ruth baseball your dad left you was really signed by the Babe.

And you may not want to know.

“Mental representation” is another term for belief.  Our beliefs are models of parts of the objective world that our brains build from information provided by our senses.  But they are incomplete replicas of the things they model.  Mental representations stand for or correspond to their counterparts in the objective world, but they typically don’t capture all the information that’s out there.

Capacity limitations prevent us from holding in our heads and processing every bit of information that objectively exists about whatever we’re modeling (if we could collect it all in the first place).  The previous post examined how our brains solve this problem by throwing some information away, and compared it to the lossy compression used to squeeze digital music files down to a size that will fit comfortably on your iPod.

Lossy compression in itself needn’t compromise the objectivity of a representation.  Take one kind of lossy model of the objective world, the road map, which intentionally omits detail below a certain information threshold (from smaller cities and towns down to pebbles and blades of grass).  A road map can still be considered a faithful, if incomplete, representation of objective reality as long as the relative positions of the cities that are shown mirror their relative positions in the physical world.

Our brains perform something similar to lossy compression when we place mental representations in long term memory, maintain them over time, and retrieve them.  We can’t remember every detail, but we strive to retain the important stuff.  In order to preserve our representations’ fidelity to the objective world, however, each of these steps needs to be conducted in a consistent and unbiased manner, omitting only data below a specified information threshold, and not dropping, adding to, or changing any of the substantive stuff.

If we’ve learned anything from decades of research in human cognition, it’s that we routinely fail to live up to this standard.  A variety of cognitive biases make our mental representations vulnerable to corruption and distortion at every storage stage.  We may fail to record key parts of them to memory through selective encoding.  Over time while stored, the memory traces of elements not regularly reinforced can decay, leading to selective retention.  And recall cues sometimes cause us to remember some parts of a representation and not others through selective retrieval.

Our mental representations are prone to contamination in the process of being compressed, stored and retrieved.  But remembering the phrase, “Garbage in, garbage out,” let’s back up.  How sure are we of their accuracy as mirrors of objective reality before we store them?

Returning to the file compression example, suppose we have a CD recording of a live music performance which we have always considered a faithful representation of the event.  What might compromise its accuracy and objectivity?  Just a few things:

• Microphone placement. Sound waves bounce off surfaces, which selectively absorb some frequencies and reflect others, and they and their echoes arrive at different locations at different intensities.  No two locations in a concert hall receive exactly the same sound.
• Equipment sensitivity. Microphones, magnetic recording tape, and other electronic components are able to capture some parts of the sound spectrum but not others.
• Mixing and mastering. The producer and recording engineer make many decisions about how to combine the individual tracks into what they subjectively judge to be the most esthetically pleasing final mix – take some shrillness off the vocals, add a little more thump to the drums, etc.  Their touching up of a mostly realistic representation is the aural equivalent of airbrushing a photograph, and is a minor act of authorship.
• Analog to digital conversion. Sound waveforms have a continuous range of values.  Digital technology can handle only discrete values (zeroes and ones).  Per the figure below, converting the former to the latter is like forcing a round peg into a square hole.

Even before you apply lossy compression to your CD of James Brown Live at the Apollo to shrink it enough to fit on your iPod, much of the original event has already been lost in translation.   The same is true of our mental representations before we commit them to the further corrupting processes of memory.  Like the microphone, our senses have a limited range, and they can’t collect every bit of data floating around out there.  Worse, we miss much of the stimuli we can perceive, either because we don’t expose ourselves to it (selective attention) or we don’t see it because we’re not looking for it (selective perception).  And our cognition screens, inflects, and otherwise adulterates mental representations in myriad other ways, each one reducing their fidelity to objective reality.

The fact that each of us is exposed to a different set of stimuli from an event in the objective world, from which we shape a mental representation using the filters and distorters of our own unique set of cognitive biases, makes our mental models of the world subjective by nature.  Embedded in each one is the implicit qualifier: seen from this position, constructed in this manner within these constraints.

Before we completely write off our ability to know objective reality, though, we need to note that our information processing deficiencies are mitigated by two important factors, the first of which is the refinement of our representations through error correction and data accumulation.  The construction of mental representations is a continuous and incremental process involving many “reality checks” to detect processing errors and test representations’ accuracy.  Some of those checks are built into our perceptual and other low level cognitive systems in the form of correction subroutines that catch many errors early in the construction process, before they can propagate.

Our mental representations continue to be refined, and their correspondence to objective reality improved, when we pool our knowledge with others’.  This is the purpose of the scientific method, which weeds out many of the errors and biases that can taint our individual representations by requiring detailed disclosure of the steps taken to construct them so that others can verify them.  The scientific method is the formal and systematic expression of a process we routinely conduct more casually whenever we compare notes and look at the world through others’ eyes.

The cognitive errors and biases that taint our mental representations of the objective world are a significant source of what is conventionally called subjectivity.  (I include in this category our use of interpretive schemas or frames to make sense of the world, which selectively filter and inflect environmental data in the act of model construction, though this is a topic worthy of a more detailed discussion than can be given here.)  But pooling information also helps us to overcome positional subjectivity, which is not so much about error correction as it is filling in missing data about the objective world.  An example is the triangulation process by which an object’s geographical position can be accurately estimated with information from several observation points.

By constantly refining our models of the world and filling in missing data, both within our individual information processing systems and collaboratively by pooling our information with others’, we move in the direction of “de-subjectifying” them and making them represent objective reality more faithfully.

The key word in the preceding sentence is “more.”  The factor that most mitigates our flawed information processes is their considerable fault tolerance.  Our models of the objective world virtually never mirror it perfectly.  But too much is made of this, because they don’t have to.  The fact that we have become the planet’s dominant species and have walked on the moon is proof that our fuzzy accuracy is good enough to achieve substantial success navigating objective reality.

Photo by André Karwath

Compression can be applied to any set of information. If you wanted to squeeze a book into a smaller number of pages losslessly, you could lower the font size and decrease the spacing between lines. The compressed book would still have 100% of the information – every word – of the original; it would just be smaller. Alternatively, you could reduce it even more with lossy compression by removing all the articles (a, and & the). Some information would be missing, but articles are relatively expendable because they are redundant of other information in the text – we can figure out what’s missing from the context. If we remove “the” from “The quick brown fox jumped over the lazy dog,” we still know what it’s trying to say. If we were to remove nouns or verbs instead, we’d lose much of the message.

The goal of lossy compression is to maximize efficiency – to reduce the size of a data set as much as possible while retaining as much of its gist as possible. It draws on information theory, a mathematical discipline whose principles apply to a wide variety of domains including communications, finance, physiology and statistics. Using information theory, we can quantify the amount of entropy (or uncertainty, or noise) a particular lossy compression scheme introduces, the goal of course being to keep the signal-to-noise ratio as high as possible.

Human cognition relies heavily on lossy compression in constructing and working with mental representations of the objective world – i.e., beliefs. Our cognitive capacity, like your MP3 player’s memory, is zero-sum – our ability to process and store information is limited, and any data given to our brains to crunch takes up resources that become unavailable for processing anything else. We can’t possibly hold in our heads every bit of information that describes an object in the physical world (if it were possible to collect it all in the first place).

Our ability to know and successfully navigate objective reality, then, relies on the effectiveness of our cognitive shorthand as a stand-in for the objective facts it represents. Our information processing mechanism works not with rich images of the external world but with coarse representations that toss away a lot of the detail, of necessity.

Does lossy compression in and of itself compromise objectivity? Not if the compression process is attribute-neutral and doesn’t introduce any kind of bias. Faced with a decision about which pixels in an image of a flower to discard, if we take them from all over the flower in a way that makes the resulting image consistently fuzzy – as opposed to, say, removing a specific petal – the correspondence of the resulting compressed image to the original changes in granularity, but not in direction. It introduces uncertainty, but diffuses it evenly.

As it turns out, though, we cannot count on cognition’s compression process to be unbiased. Worse, the data our senses collect and pass on for processing is prone to corruption by a host of biases before it even gets to the information-reduction stage, as well as in processing steps after that stage. These challenges to living in the objective world will be the subject of my next post.