David Moore

A Red Herring

2010-01-06T11:15:47-05:00

Mark Blumenthal is right to argue in his recent National Journal article ("The Problem With Polling Cap-and-Trade,") that "politicians should proceed with caution when trying to anticipate public opinion on a complex policy issue." In my opinion, however, he goes too far when he implicitly expresses approval (or perhaps tolerance) for such polling, on the grounds that pollsters are just probing latent public opinion.

As Blumenthal notes, CNN and Pew each conducted polls on this issue, with CNN finding 97 percent of the public expressing an opinion, while Pew reported 89 percent with an opinion. Also, CNN found a 23-point margin in favor of cap-and-trade legislation, Pew an 11-point margin. Blumenthal emailed George Bishop, author of  The Illusion of Public Opinion, asking "What should we make of such findings?"

Bishop responded by writing: "Reliable and valid measures of public opinion on such a complex policy issue cannot be so simply simulated by merely telling respondents what it's about and then asking them to react to it on the spot. Down that road lie misleading illusions and the manufacturing of public opinion - a disservice to the Congress, the president and the press that covers them."

Blumenthal takes issue with Bishop's response, arguing that "policy makers have good reason to want to probe the sorts of opinions that the venerable political scientist V.O. Key once termed latent, those likely to be stirred up should the legislation become law or the focus of a future election campaign. Pollsters can attempt to simulate such hypothetical attitudes in a telephone survey, but the results will be very sensitive to the words they use and, more importantly, to the assumptions they make about the competing arguments voters may eventually hear."(italics added)

So far, I agree with everything Blumenthal says. Note especially the words in italics, which emphasize both that pollsters can "attempt" to simulate hypothetical attitudes, and that such results are sensitive to question wording.

But then Blumenthal argues that "It is possible to anticipate how public opinion on a topic like cap-and-trade may evolve, but proceed with caution." (italics added) Unfortunately, the assertion in italics lacks supporting evidence. In fact, the efforts to predict public opinion on complex issues are more likely to generate wildly conflicting polling results, such as polls on support and opposition for a bailout, or the "card check" (EFCA) bill. The reason: As Blumenthal cautions - because polling results in such situations are highly dependent on question wording and context.

More distressing, however, is Blumenthal's advice to "proceed" - even with caution - which implies 1) that pollsters recognize they are measuring "latent" opinion and 2) they are willing to admit as much to the public. But that is not the case.

It's important to remember that "latent" opinion is not "current" opinion. It is hypothetical, speculative, inconclusive - i.e., it is a concept about what opinion might emerge, depending on the varied ways in which the issue is covered in the press. It would be disingenuous for pollsters to (as Blumenthal says above) "attempt to simulate such hypothetical attitudes" using only one method (question wording) and then declare that the results represent current public opinion. Yet, that is exactly what media pollsters do.

Pollsters do not say they are "simulating hypothetical attitudes." Instead, recognizing that many people may not know anything about the issue (which Pew in fact acknowledges), pollsters feed their respondents information, used forced-choice questions to extract an immediate reaction, and then announce the results as though they represent what the public is currently thinking.

CNN announced its hypothetical results by saying - "CNN Poll: 6 in 10 back 'cap and trade'." Pew was no less assertive in presenting its hypothetical results, saying that "the survey finds more support than opposition for a policy to set limits on carbon emissions." Its sub-headline was also quite definite: "Modest Support for 'Cap and Trade' Policy." In no place did either polling organization admit that these results were based on "simulating hypothetical attitudes."

It's important to keep in mind that once respondents are fed information in the context of a survey, they no longer represent the larger population from which they were drawn - because the rest of the public has not been given the exact same information at the exact same time. Objective questions can often be asked of sample respondents without fatally tainting the sample, but giving respondents extra information about an issue cannot avoid contamination. Such a process inevitably means that, at best, pollsters are trying to simulate what public opinion might emerge. But they don't present their results with such a warning label.

So, it seems to be a red herring to justify polls on complex issues, such as 'cap and trade', by suggesting that policy makers may want to probe "latent" public opinion. Yes, perhaps they do. But that's not what the media pollsters admit to doing.

As Blumenthal writes in a previous post, "If subtle changes in wording can produce such different results, then we can assume that many respondents are forming opinions on the spot rather than sharing pre-existing views on the actual legislation. 'Public opinion' in this sense isn't so much 'fluid' (a favorite pollster cliché) as non-existent."

Yes, indeed. Pollsters converting non-existent public opinion into the appearance of current public opinion, exactly the "manufacturing" charge that Bishop makes.

If the media pollsters want to simulate hypothetical public opinion, they should clearly label it as such, instead of presenting their results as reflective of actual (existing) public opinion. Until pollsters are more candid about the nature of their polls, I've got to side with Bishop in characterizing this kind of polling as "manufacturing opinion," which produces "misleading illusions" about the public and is ultimately a "disservice" to the Congress, the president, the press and the people.

Those Bobblehead Polls on Health Care

2009-09-23T11:30:25-04:00

A recent series of polls brings to mind a bobblehead doll, whose head wags from side to side and from front to back in a random fashion. That disconnected movement seems to be a visual representation of what the polls have been saying about the general public and its views of President Obama's efforts to reform the health care system.

Before the president's nationally televised speech to a joint session of Congress, for example, several polls announced that either a majority or plurality of Americans disapproved of the president's performance on health care. They didn't tell us if the people were dissatisfied because he was trying to do too much, or because he wasn't trying to do enough. All we got was the dismal information about how the public felt let down by the president.

Then, on the Friday after Obama's speech, a CBS poll declared that the public had rallied in favor of the president. A majority of the public now approved of his handling of health care by a 14-point margin, compared to a week earlier when a plurality disapproved by a 7-point margin. That represented a 21-point swing in opinion, which is no mean feat for one speech. (For all poll results, except where noted otherwise, see pollingreport.com.)

But early the following Monday, an ABC/Washington Post poll announced a different opinion. The "bottom line views on health reform," according to the analysis at ABC, "failed to improve since President Obama addressed the nation." His approval rating on health care was essentially unchanged since mid-August, contradicting the CBS finding of a 21-point swing.

Then at 3:00 PM on Monday, CNN revealed that Obama's approval rating showed a 13-point favorable swing over the previous week. Now a majority approved of the president's handling of health care policy, 51 percent to 47 percent, a reversal from a couple of weeks earlier when people disapproved by 53 percent to 44 percent. Not only that, but "strong" opposition to Obama's "plan to reform health care" had dropped by 9 points.

The very next day, a USA Today/Gallup poll asserted that (contrary to CNN's poll) Obama's speech "didn't change minds," and that "for the first time" a majority of Americans disapprove of the way the president is handling health care policy.

Since all these polls were conducted in roughly the same time frame, it makes one wonder if they were all talking about the same public. After all, the pollsters all claim to be using scientific methods of choosing their samples, so that they can interview a representative group of American adults. Could a rational and informed public provide such different scientific measures of its opinion on health care to these four polling organizations?

Perhaps a key to the dysfunctional results can be found in a question that was not reported on CNN's website, politicalticker, but was posted in its archive. When asked how much they knew about the details of Obama's health care proposals, almost six in ten respondents admitted that they knew little to nothing -- after having just opined about them during the previous twenty-minute survey. Only 11 percent said they knew "a good deal."

The ABC/Washington Post poll found a slightly more informed public, with just 44 percent knowing little to nothing about Obama's proposals, and 19 percent saying they knew "a great deal." But a previous poll by CBS perhaps best describes the state of the public: Less than a third said they understand the health care reforms under consideration by Congress, while two-thirds, 67 percent, said the proposals are confusing.

No wonder the polls seem like a bobblehead. Most of the public is confused about the health care debate, and a substantial segment -- a large minority or even a majority, depending on the poll -- doesn't know much about the specific proposals. When these people answer the detailed and arcane questions posed to them in the polls, they respond in an almost random fashion, their answers influenced by the way a question is worded or by the previous questions that have been asked.

Public opinion is not irrelevant in this debate. The public can provide an overall direction to its leaders, and here the polls are useful. They all show a widespread public consensus that the health care system is in great trouble and needs substantial reform. They also show that large majorities of Americans believe that all people should be covered by health insurance and that the government has a responsibility to ensure such coverage. Most Americans also believe that health insurance companies should not exclude people from coverage because of prior medical conditions nor drop people who become sick.

But it is ludicrous to place confidence in polls that ask the general public about the specific details of plans under consideration, when most people don't have that kind of detailed knowledge or at the very least are confused by the many contradictory proposals. The bobblehead public is an illusion created by pollsters who refuse to acknowledge the limits of their craft.

Public Opinion and Health Care: Fearmongering on the Right

2009-09-02T16:29:00-04:00

Recently, New York Times columnist David Brooks has echoed the chorus of right wing noisemakers, who claim that the public is firmly opposed to the healthcare reform efforts of President Obama and the Democrats in Congress, and that if the Democrats persist in trying to pass such legislation, they will suffer electoral disaster in 2010. Such concern for the Democratic Party by these otherwise stalwart partisan foes suggests at least a scintilla of insincerity, but the depth of that disingenuousness cannot be appreciated unless one examines more carefully the actual claims.

Brooks writes, for example, that "public opposition to health care reform is now steady and stable." He adds that according to Republican pollster, Bill McInturff, "public attitudes toward Obamacare exactly match public attitudes toward Clintoncare when that reform effort collapsed in 1994."

In fact, public support for health care reform today is substantial, even according to McInturff's own polls, and it mirrors the strong support for health care reform expressed by the public in 1994, even as the Republicans were killing President Clinton's proposals with their "no compromise" strategy.

McInturff is the Republican partner of the NBC/Wall Street Journal poll, while Peter Hart is the Democratic partner. Contrary to Brooks' claim about the public's opposition to health care reform, the McInturff/Hart mid-August poll for the media organizations found 60 percent of Americans saying that the U.S. health care system needs either a complete overhaul or major reform. These results mirror those found by several other polling organizations, such as the recent poll by CBS News, which shows 82 percent of Americans saying the U.S. health care system needs either to be completely rebuilt or fundamentally changed. It's clear that Republicans in Congress do not agree with the public on this matter.

While general public support for reform is widespread, opponents of health care reform can point to polling questions that ask specifically about Obama's health care plan. Here the results show a decline in support over the past several weeks. Pollsters know that few people genuinely understand the various proposals, so typically they ask respondents to express an opinion based on "what you've heard or read." That's what the NBC/WSJ poll did, and it found 36 percent saying Obama's plan was a "good idea," with 42 percent saying "bad idea," and another 22 percent expressing no opinion.

That apparently is the basis for Brooks' contention that public opposition is now "steady and stable." But the pollsters recognize that many respondents who express an opinion do not know what Obama's plan is. So, in this case, McInturff and Hart wrote a separate question that included a brief summary of the plan. Once respondents heard what the plan was, a 10-point majority expressed support (53 percent to 43 percent). Somehow Brooks failed to mention this finding.

A similar situation prevailed in 1994, when President Clinton's health care plan was defeated. In general, the public expressed widespread support for reform. Over the course of the public debate, people became more confused about the specifics of Clinton's plan and thus more negative. Reform legislation ultimately failed in October, but Gallup polls right beforehand showed even then that more than 60 percent of Americans wanted Congress to pass health care reform -- if not that year, then the next year.

When respondents were asked, "from everything you've heard or read about the plan so far," did they favor or oppose Clinton's health care reform plan, a majority said they opposed it (55 percent to 40 percent). But the same poll showed that a stunning 70 percent admitted they needed "more information to judge the health care plans that have been proposed." And, when read a description of Clinton's plan, 61 percent were favorable toward it and only 37 percent unfavorable.

Today, we find the same confusion. When forced to express an opinion specifically about Obama's plan, most will. But if permitted, respondents will also admit that they really don't know much about it. The CBS poll, for example, reports that only a third of respondents say they understand the health care reforms being considered by Congress, while two-thirds say the proposals are confusing. Earlier, Gallup reported half of respondents saying they don't have a good understanding of the issues involved in the current debate over health care reform. In their confusion over the plan, with conflicting media reports about what it might ultimately entail, many people are cautious and express opposition. Yet, when the NBC/WSJ poll, and earlier a poll for National Public Radio, included descriptions of Obama's plan, majorities expressed support.

Because of implacable Republican opposition to health care reform, some supporters propose that Democrats consider a legislative technique in the Senate to circumvent a likely Republican filibuster, which would require 60 votes to override. Using "reconciliation," some Senate Democrats believe they could get most of the health care reform bill passed with a simple majority vote. Brooks characterizes this strategy as ignoring the "ignorant masses" to "ram health care through reconciliation."

That the "masses" may be ignorant -- or at the very least, misled -- is attested to not just by the CBS and Gallup polls, but by McInturff's own poll (with Hart). They found majorities of Americans believing -- erroneously -- that Obama's plan would cover illegal immigrants, entail a government takeover of the health care system, and provide tax dollars to pay for abortions. Close to half also believe the lie that the government would then be authorized to make decisions about when to stop medical care for the elderly. Given these widespread misperceptions, polls showing specific opposition to Obama's health care plan need to be viewed skeptically.

Still, Brooks argues that allowing for a simple majority vote in the Senate (hardly what democratic theorists refer to as "ramming") would be "suicidal." "You can't pass the most important domestic reform in a generation when the majority of voters think you are on the wrong path." But had Brooks looked carefully at the polls, he would find that most voters don't think Obama is on the wrong track. Not only do they overwhelmingly support reform, as shown earlier, but by a two-to-one margin, the CBS poll shows they also believe Obama has better ideas about health care reform than do the Republicans in Congress.

That the Democrats might actually get a health care reform bill passed worries Republicans, who believe, as did their party colleagues in 1994, that passing no health care bill will ensure a major Republican victory in the mid-term elections. Fifteen years ago, the Republican mantra was "no bill at any cost." And it worked. They crushed President Clinton's efforts to enact any health care reform, and in the mid-term elections they won majority control of the House for the first time in four decades.

Many, but not all, Democrats in Congress have learned the lesson of 1994. They know that whatever the public opinion polls show now, the party will be much weaker in the mid-term elections if it has failed to produce health care reform than if it can enact reform, even if "only" by majority vote. Taking advice from Brooks and his cohorts about what the public wants would constitute the real disaster.

Likely Voters and Mid-Term Elections, Part II (The 1934 Mid-Term Election)

2009-08-21T01:07:11-04:00

In my earlier post (Part I) on this subject, I suggested it would be a political miracle if Democrats did not lose U.S. House seats in the 2010 election. A major reason is that in mid-term elections, the voters most likely to turn out are those who are disgruntled with the policies of the incumbent president, rather than his supporters. In fact, since Democrats and Republicans first began competing against each other for Congress in the mid-19th Century, the president's party has lost seats in mid-term elections, relative to the "out" party, all but three times - 1934, 1998 and 2002.

The two most recent times seem to be special situations that are likely to have little relevance for 2010. The 2002 mid-term election saw the president's party pick up a few seats, probably because the country was still rallying around the flag in reaction to the terrorist attacks the previous year. President Bush's approval rating was still high (Gallup showed it at 68 percent in a poll conducted Nov. 8-10, 2002).

And the 1998 mid-term election found voters quite dissatisfied with the Republicans' vote to impeach President Clinton, apparently a major reason why the president's party was able to pick up a few House seats. In both years, the Republicans enjoyed slim majorities, and the changes did not affect their majority control.

If neither of the two exceptions just noted appear to have much relevance to what might happen in 2010, the third exception (the first chronologically) is a different story. Taking place just two years after President Roosevelt came into office during the Great Depression, the 1934 mid-term election campaign focused largely on the New Deal measures adopted by Congress. The result was a net increase of 9 seats for the president's party - from 313 to 322. Democrats today no doubt hope that a similar debate about the economic stimulus bills and health care reform will be a positive inducement for voters next year.

That's where the differing poll results - depending on whether the pollsters use "likely voters" or not - provide an interesting story. The latest results (noted in my previous article) suggest the Democrats have an overall lead in the congressional house vote of from six to seven percentage points among the general public or registered voters, but are apparently in a dead heat with Republicans based on likely voters.

If indeed, the two-party aggregate vote is about even on Election Day, that would almost certainly result in major seat losses for the Democrats, though it's difficult to say how many. The aggregate two party vote (the total voting for Democrats vs. Republicans in the country as a whole) is not a perfect indicator of how well the parties fare in winning House seats. In 2002, for example, the Republicans won 54.1 percent of the two-party vote nationwide, but got 52.6 percent of the House seats. In 2004, they won a smaller percent of the two-party vote (51.4 percent) but picked up three seats, expanding their majority to 53.3 percent of the seats.

In 2006, the Democrats won 54.2 percent of the aggregate vote, and won about the same percentage of House seats - 53.6 percent. Two years later, their share of the national vote increased by only .4 percent (to 54.6 percent), but they gained 21 seats to hold 59.0 percent of the House seats.

Despite the inconsistency between the percentage of the national vote and the percentage of House seats won by the majority party, a tie vote nationally would likely cause the Democrats to lose a significant number of seats. The Democrats beat the Republicans in 2006 and 2008 by about 8 to 9 percentage points. Currently, not many polls suggest the Democrats will win by that margin in 2010.

Of course, with more than a year to go, much can happen to shape the political landscape. No doubt, the most salient domestic issues will be the stimulus bills and health care reform, and no one knows for sure what will happen to the latter. And then there is always the possibility of some major international event that could influence the elections.

Democrats may hope for a repeat of the 1934 mid-term election, but history tells us the circumstances will have to be quite unusual for that to happen. As indicated in Part I, given the expected low turnout, I'd be especially attentive to the preferences of "likely voters." If they indicate a Democratic lead of 8 to 10 points, that would be unusual indeed.

Likely Voters and Mid-Term Elections, Part I

2009-08-18T17:42:02-04:00

It would be a political miracle if the Democrats did not lose seats in the 2010 Congressional elections, yet the polls so far suggest that scenario is doubtful at best. I think it's because most polls are providing a rosier picture for the Democrats by reporting voting intentions of the general public, or registered voters, rather than the much smaller segment of "likely voters" that will ultimately turn out to cast a ballot.

That the Democrats will almost certainly lose House seats in 2010 is attested to by several factors. The most important, of course, is that since the advent of the current two party system (Republicans and Democrats), the party of the president almost always loses seats in a mid-term election. The best theory for this phenomenon is that disgruntled people (i.e., those who identify with the "out" party) are more motivated to cast a protest vote than the relatively satisfied people (i.e., those who identify with the party of the president) are to cast a vote of support.

The second factor is that, in the wake of the protracted war in Iraq and the sagging economy, Democrats won many seats in 2006 and 2008 that would "normally" go to Republicans. In 2010, with Bush gone and a Democratic administration in charge, Democratic House members in those "normally" Republican seats are going to be quite vulnerable.

The final factor is that as a general rule, Republicans are more likely to turn out than Democrats, because Republicans tend to be higher on the socio-economic scale - generally more educated, with higher incomes, and more actively involved in politics than Democrats.

So, if all of these reinforcing factors suggest the Republicans are likely to gain seats, why aren't the polls showing that? Here are some interesting recent poll results (see pollingreport.com):

July-August Polls 2009 Measuring Support for Congressional Candidates, 2010
Date	Poll	Democrats	Republicans	Unsure/neither/dk	Democratic advantage
		%	%	%	Pct Pnts
Aug 10-13 (general public)	Daily Kos/ Research 2000	36	28	36	+8
July 31-Aug 1 (general public)	CNN/Opinion Research Corp	44	34	22	+10
July 24-27 (general public)	NBC/Wall Street Journal	46	39	15	+7
July 22-26 (likely voters)	NPR/POS and GQRR	42	43	15	-1
July 19-23 (likely voters)	GWU - Tarrance/Lake	43	40	17	+3
July 9-13 (registered voters)	Diageo/Hotline	39	32	30	+7
July 10-12 (registered voters)	Gallup	50	44	7	+6

Note that there is little difference in the lead that polls show for Democrats when the sample is either the general public or registered voters - from six to ten percentage points. However, the two polls that reported results based on "likely voters" show essentially a dead heat (a 3-point Democratic lead or a one-point Republican lead).

Nate Silver (at fivethirtyeight.com) suggests caution in relying on likely voter models this early in the 2010 campaign. Generally, I agree that early polls - especially in specific races (as opposed to the more general generic ballots reported above) - need to be viewed with caution. Many people are undecided 10 to 12 months ahead of the election, though some pollsters obscure that fact by using a forced choice format. See, for example, the contrast between Diageo/Hotline and Gallup above, the former showing 30 percent of registered voters undecided, Gallup showing just 7 percent.

Furthermore, different polling organizations use different screeners to arrive at their presumed "likely voters," some more "aggressive" than others. So, it's difficult to make direct comparisons with polls showing different leads, even if they base their results on likely voters, rather than registered voters or the general public.

That said, I would argue that in general we get a more realistic view of the general sentiment of voters, if the sample has been screened fairly tightly to produce a relatively small segment of likely voters rather than a much larger group of people - the general public or even "registered voters." In mid-term elections, turnout is only about half or so of turnout in presidential elections. Thus, screening out the non-voters is much more sensitive for understanding mid-term elections than presidential elections.

So, contrary to Nate Silver's advice, I would suggest that when polls diverge, one based on likely voters is probably a better reflection of the actual electorate than a poll based on the general population or even registered voters.

(In Part II I will discuss the exceptions to the general rule that the president's party loses House seats in mid-term elections, and whether those exceptions are relevant to 2010.)

Five Reasons Why The Iranian Pre-Election Poll Can't Be Trusted

2009-06-19T16:00:42-04:00

Over the past several days, according to the New York Times and other news sources, millions of protesters in Iran have taken to the streets to express their opposition to the official results of last week's disputed presidential election.

Earlier this week, however, the Washington Post published an op-ed by Ken Ballen and Patrick Doherty, who claimed that a scientific pre-election poll in Iran, conducted "three weeks" before the balloting (in fact, it was four weeks), presaged incumbent President Mahmoud Ahmadinejad's official landslide victory. If we can believe the poll, and the authors' arguments, it's plausible that Ahmadinejad did in fact win the election with more than 60 percent of the vote.

But don't be fooled. The same poll results can be used -- and just as persuasively -- to argue that the Iranian election was stolen, as can be used to argue the opposite viewpoint.

In fact, there are at least five reasons why we can't trust that poll. But first, some background.

What the Poll Showed

The poll was conducted from May 11-20, and incorporated what appear to be rigorous scientific sampling methods. The telephone poll (landline phones) included about 1,000 respondents, the vast majority of whom said they intended to vote. Interviewers called into Iran from outside the country and spoke Farsi. There is little to no criticism of the poll's sampling and interviewing methods.

The results showed that Ahmadinejad led his closest opponent, Mir Hossein Mousavi by 20 points - 34 percent to 14 percent. But more than a quarter of the respondents said they didn't know who they would vote for, and another 15 percent simply refused to answer.

By contrast, the official results http://en.wikipedia.org/wiki/Iranian_presidential_election,_2009show Ahmadinejad getting 63 percent of the vote to Mousavi's 34 percent, while the other two candidates together get just 3 percent.

Five (Somewhat Overlapping) Reasons Why We Can't Trust the Poll

Reason #1: The poll was conducted too far away from the election to have much predictive value.

The poll was conducted in mid-May, on average about four weeks before the election. Too much can happen in four weeks to rely on a poll this old to explain what happened on June 12.

(In the op-ed, the authors claimed the poll was conducted "three weeks" before the election, but that is disingenuous to say the least. In fact, according to their own report, interviewing was conducted over the whole length of the 10-day period (May 11-20). The poll ended three weeks and two days before the election, but the initial interviews began almost five weeks before the election (four weeks and five days). May 15-16 is the mid-point of the interviewing period, which is (on average) four weeks, not three, before the election.)

Reason #2: So many people were still undecided in mid-May, that the "2-1 lead" noted by the authors in their op-ed is close to meaningless.

The poll found Ahmadinejad getting just 34 percent of the vote. Yes, it was more than a two-to-one lead over Mousavi's 14 percent, but more than a quarter of the people (27 percent) said they were undecided, and another 15 percent refused to answer. The poll simply cannot be used as a predictor with so many undecided and undeclared.

Reason #3: The true undecided percentage was almost certainly much higher even than the 27 percent recorded in the poll.

The authors employed a "forced-choice" vote intention question, which asked "if the election were held today, who would you vote for?" Respondents were not explicitly given the option "or haven't you made up your mind?" Thus, they were pressured into coming up with an opinion, even if they didn't have one. In that case, the best known name was Ahmadinejad, and it's quite likely that many undecided voters gave his name solely because he was better known -- at that time. (Of course, 27 percent said they were undecided despite the forced-choice question, but that percentage would have been greater had the question provided an explicit "unsure" option.)

Reason #4: In general, individual pre-election polls are not reliable predictors of election results until just a day or two before the election. They are especially unreliable three and four weeks before an election.

Because most pre-election polls employ the "forced-choice" (who would you vote for if the election were held "today") question, they consistently underestimate the undecided vote and thus the potential for change. We know that polls can't be trusted weeks before an election, because reputable polls come up with contradictory estimates - until right before the election.

A recent example is the 2008 American presidential election. Just a week before Election Day, Pew Research showed Barack Obama leading by 15 points, and CBS reported an 11-point lead. At the other end of the spectrum, Investors Business Daily found Obama up by only 4 points, while the George Washington University Battleground Poll said just 3. A few days later, the latter two polls reported a surge in support for Obama, while the previous two polls showed a sudden decline, resulting in final predictions that were close to each other and to the election results.

The Iranian pre-election poll was a one-time snapshot, and a fuzzy one at that - based on its use of the force-choice question. We don't know what a similar poll would have predicted if interviewing had been conducted up close to the election. But we certainly can conclude that its results four weeks earlier are hardly relevant to what actually happened on June 12.

Reason #5: The Iranian poll report itself suggested the lead was closer than the numbers indicated, and that no candidate would achieve an absolute majority -- much less the 63 percent Ahmadinejad allegedly received.

In May, the authors of the poll report concluded, among other things, that "a close examination of our survey results reveals that the race may actually be closer than a first look at the numbers would indicate" (emphasis added). The reason: Most of those who said they didn't know which candidate they would support favored change or reform in the current political system, and thus were less likely -- according to the report -- to support the incumbent.

The report also noted that it was "likely," given the "current mood," that a run-off election would be necessary, because none of the candidates was likely to win an outright majority of the vote.

News reports of the electoral campaign from mid-May until June 12 hardly suggest the "current mood" had improved for Ahmadinejad. Indeed, one analyst suggested just the opposite, based on other polling in Iran.

Thus, it hardly seems credible for these same authors to argue, a month after their poll, that their results can be used to validate the Iranian election. Anyone who took their analysis seriously in the May report could be persuaded that the alleged 63 percent victory was in fact bogus. The campaign from mid-May to June 12 seemed to be going against Ahmadinejad, a view that seemed to be reinforced by other (though less reliable) polls, yet somehow the president was able to increase his margin from 20 points (in the May poll) to 29 points (in the official tabulation).

But I wouldn't argue that the poll can be used to challenge the election results. Instead, I believe that the most sensible position is simply to admit that the May poll cannot be used either to support or to challenge the election results.

Not Seeing the Forest for the Trees: Important Lessons from a SurveyUSA Experiment

2009-05-04T11:10:11-04:00

In a recent conversation with Jay Leve, founder and head of SurveyUSA, I was alerted to a split sample telephone survey experiment he conducted last October in the San Francisco Bay Area. It was on the subject of the government's plan to bailout or rescue Wall Street.

Though the experiment dealt with an issue that is so last year (!), I'm writing about it now because it demonstrates how questions about specific policy plans can produce misleading results about the public's views of the broader issue - a classic case of not seeing the forest for the trees.

SurveyUSA Experiment

 

Jay tested four different ways of phrasing the bailout question, and each one found mixed to slightly negative results. But then two follow-up questions starkly contradicted these results to suggest a clear majority of the public was supportive of the bailout efforts.

While the SurveyUSA experiment tested four different ways of wording the bailout issue, three are rigorously comparable, and so I mention them first. I'll come back to the implications of the fourth version later in this post.

Each of the following questions was asked of a split sample of just over 500 respondents:    

The government may invest billions to try and keep financial institutions and markets secure. Do you think this is the right thing for the government to do? The wrong thing for the government to do? Or, do you not know enough to say?

The government may spend billions to bail-out Wall Street. Do you think this is the right thing for the government to do? The wrong thing for the government to do? Or, do you not know enough to say?

The government may spend billions to rescue Wall Street. Do you think this is the right thing for the government to do? The wrong thing for the government to do? Or, do you not know enough to say?

TABLE 1 Results of Three Versions of Bailout Question "Right or Wrong Thing to Do?"
	Right	Wrong	No Opinion	N
	%	%	%
A. Invest billions to keep markets secure	39	37	24	532
B. Spend billions to bail out Wall Street	40	42	18	533
C. Spend billions to rescue Wall Street	34	46	20	523

 

Was the Public Ambivalent?

With these results, one would have to conclude that the public was at best ambivalent toward a bailout or rescue of Wall Street. Given the sample sizes, Form A results are significantly different from those in Form C, suggesting "rescue" is the most negative way to phrase the issue and "invest" is the most positive way.

But then come two additional questions that throw a completely different light on the issue. The question below taps into a general feeling about the issue, and finds that people seem to want Congress to do something - and that they're more afraid Congress will do too little than too much.

TABLE 2 More Afraid Congress Will Under-react Than Over-react
	Too Much	Too Little	No Opinion	N
	%	%	%
What concerns you more: that the government will do too much to fix the economy? Or, that the government will do too little?	32	58	11	2100

The problem with comparing the above question to the other three is that this one doesn't explicitly allow for "no opinion," while the first three questions do.

Public Supports Bailout

But the next question is comparable in offering an explicit "don't know" response, and it also suggests that a clear majority of the public wants Congress to do something.

TABLE 3 *Should Congress Support SOME* Economic Rescue Effort?**
	FOR	AGAINST	No Opinion	N
	%	%	%
Do you want your representative in Congress to vote FOR an economic rescue? To vote against an economic rescue? Or, do you not know enough to say?	54	30	16	2100

Note that despite offering an explicit "or don't you know enough to say" option, this question shows a clear majority in favor, with only 30 percent opposed.

Had we depended on only the first three questions, which were asked of separate split samples, for our understanding of the public, we might well have concluded that whether it was "bailout" or "rescue" or "invest," the public was either evenly divided about a bailout or leaning against such an effort. But these last two questions suggest that less than a third of the public was opposed to an economic rescue/bailout plan in principle, while a clear, though small, majority (54 percent) of people were in favor.

New Complications

When we look at the results of the fourth version of the split sample experiment, we find even more support for some type of rescue plan. This version included three substantive options (compared with just two for the other three versions - which is why I'm treating this question differently from the first three versions), followed by the explicit option of not expressing an opinion.

TABLE 4 Fourth Version of Bailout Question
	Pass this plan	Pass different Plan	Take no action	No Opinion	N
	%	%		%
Congress is working on a plan to buy and re-sell up to 700 billion dollars of mortgages. What would you like the Congress to do? Pass this plan? Pass a different plan? Take no action? Or, do you not know enough to say?	31	40	13	16	512

This version shows the least support for the current plan, but its second option - to pass a "different" plan - suggests that more than seven in 10 respondents are in favor of some rescue plan (31 percent the current plan, plus 40 percent for a different plan). These results also suggest that only 13 percent (rather than 30 percent as shown in Table 3) are opposed to some kind of effort.

Important Lessons

These results, based on a telephone survey in the San Francisco Bay Area, demonstrate how variable the survey results can be - even when the survey is about an issue that is the subject of much media attention. The first three versions of the bailout question could reasonably be interpreted to suggest that the Bay Area public was either ambivalent or negative about any bailout plan, while the question reported in Table 3 gives the opposite impression - that the same public was in fact looking for some action by the federal government.

The important lesson: Sometimes, asking about specific plans can blind us to the larger issue of whether some type of action is still desired.

The last question reaffirms that when questions have two options in favor of a policy (pass the current plan, or pass a different one) and one against (take no action), the "opinion" that is measured can be quite different from when a question has just one option in favor of a policy and one opposed.

The last question can also be interpreted to have two options against the "current" plan - only one option says pass the current plan, while two options are against it (pass a different plan, and take no action).

This doesn't mean that three options should never be offered. The example does reaffirm, however, that the more options that are offered in general, the less similar the results will be to questions that offer only two options.

Note: My thanks to SurveyUSA's Jay Leve for pointing me to his very insightful experiment.

Dispatches: Trusting What People Say in Polls

2009-02-19T14:51:35-05:00

This post is part of Pollster.com's week-long series on Stan Greenberg's new book, Dispatches from the War Room.

I'm pleased to join this conversation and to have received Stan Greenberg's book, which is a fascinating insider's look at the way polls were used in five major situations. It's great history by itself, and I would strongly recommend the book to any one at all interested in current events and/or the role of polling consultants in public policy.

I have several comments I would like to make at some point, but this one will focus on one issue raised by Mark Blumenthal and subsequently addressed by Greenberg himself.

Mark pointed to Greenberg's query at one point in the book, where the author asked how much he could trust his own polls on public policy matters. To make it easier for the reader to follow this conversation, I will reproduce what Mark wrote:

...the most candid observation from the book concerns Greenberg's admission that his focus groups and polls misled him on the question of whether Israeli voters would ever accept a division of Jerusalem. At first, two-thirds of voters in his surveys said "it was unacceptable to have a Palestinian state with its capital in East Jerusalem." When Greenberg "saw no movement" when he presented arguments for the division in surveys, and voters "nearly cried" in focus groups, insisting that dividing Jerusalem would be "like taking away your beloved child," Greenberg advised his client that such a policy was a "dead end." [emphasis added]

Yet four weeks after Ehud Barak put that option on the bargaining table at Camp David, despite a negative approval rating and strong opposition in parliament, a majority of Israelis were ready to "go with him" on Jerusalem in Greenberg's polling. Thus Greenberg raises a critical question:

If I cannot believe what people tell me is unacceptable in my surveys on Jerusalem, then what of my findings on other subjects? Why can't a determined leader change these too?

Greenberg's question about the changeability of public opinion is an important one, because much of what polls present to us today are measures of public opinions that appear to be firm - when in fact we know that for many people, polls measure only the most ephemeral of views. What's lacking in most polls are any measures of intensity.

My question to Greenberg would be how intensely did the poll respondents feel about the "unacceptable" response they gave to the interviewers? Greenberg gives no indication that he measured it, though he did get an idea of intensity in the focus groups when participants "nearly cried" about the proposed division of Jerusalem.

Still, as we know, focus groups are poor indicators of general public opinion. The people who are willing to participate are no doubt more engaged in issues than people who can't be bothered (or paid) to participate, and the focus group experience itself can be intense - making the participants completely unrepresentative of the general population.

Greenberg is, of course, quite aware of such limits, readily acknowledging them in his book. Still, as we all know, it's very difficult to ignore numbers and visceral experiences even when we know they are technically unrepresentative of the larger population. The high emotion of the focus group participants could well have made it seem impossible that voters at large could change their minds.

In a series of experiments that Jeff Jones and I designed while I was at Gallup,[1] we discovered that on any given issue about 40 percent to 60 percent of the public had a "permissive" opinion. Though many people may have initially expressed a preference for a policy (saying that they either favored or opposed it), those with a "permissive" opinion then admitted (in response to a follow-up question) that they would not be "upset" if the government did the opposite of what they had just said. This does not mean that later on, once a policy has been implemented, those same people will not hold the leaders accountable if it doesn't work. But it does mean that political leaders have tremendous leeway in what they initially decide to do.

And it also means that on most public policy issues, leaders can probably do pretty much what they want to without being held accountable.

Greenberg refers to that phenomenon when he talks about the importance of public opinion to legislators (p. 395). "The fundamental lesson is that people matter because elections matter. You could only think otherwise if you haven't spent any time close to elected officials or candidates for office...The antics of the Republicans in Washington over the past decade seemed to challenge that presumption when they ignored overwhelming public sentiment on Clinton's impeachment, taxes, and the Iraq war, and escaped accountability at the polls."

What I'm suggesting here is that the "overwhelming public sentiment" described by Greenberg was not, in fact, overwhelming. The polls are misleading us. Such sentiments as Greenberg mentions may be widespread, but thinly anchored. And whether we like it or not, from a democratic point of view, it means that politicians can often get away with ignoring what appears to be a public consensus.

The Iraq War is a good example, though on the opposite side of the issue from what Greenberg is discussing. Did Americans "overwhelmingly" support the war before President Bush launched it? The polls all said yes, by about two-to-one margins or greater. And it would appear that most Democratic Senators, who might have been expected to oppose the war, were influenced by these polls to support the war resolution. But Jeff Jones and I discovered that after measuring intensity on that issue, in fact the public was evenly divided - three in ten strongly supporting the war, three in ten strongly opposed, and a plurality - four in ten - with a "permissive" opinion (not upset either if the United States went to war or didn't go to war). How different the political climate in Washington might have been had this picture of public opinion prevailed, instead of the erroneous depiction of a public hankering for war.

While Gallup has not measured intensity on this issue recently (nor have other pollsters), I suspect that even when it comes to withdrawing troops - which Bush refused to do - the public is more divided than unified. The point is that the public is much more in the middle of an issue, and thus willing to defer to its leaders, than the polls tell us, because most polls ignore the intensity with which people hold their poll-expressed views.

In the conclusion to his book, Greenberg writes that despite his ability, post hoc, to explain why his poll results about Jerusalem might have misled him (p. 422),

...it does not change the question I now must face whenever I see a survey result that sets such dramatic limits on what is possible. How do you know that people will not rethink their starting points? How do you know they will not be moved by a deliberative process that thinks about the problem in new ways?...How do you know you won't discourage a less fearless leader from chancing to be bold?

I do not have an answer to this question, other than to constantly remind myself that opinion is changeable, that I must always simulate changing circumstances, and that I should be wary of telling a leader the public will not join him or her in this.

Like Greenberg, I think there is a much larger portion of the public in the middle of any given issue than might be at first assumed - and that we find reflected in most current polls. It's a point that Kristen Soltis makes in a different way when she praises Morris Fiorina's Culture War?: The Myth of a Polarized Electorate.

In a follow-up commentary about Greenberg's book, Blumenthal  wrote that "public opinion will ultimately limit or control the extent to which policy makers can affect change and achieve their goals, and a wise wonk will want to study public opinion -- both as it exists now and where political leaders can move it in the future."

But political leaders need realistic measures of such opinion. Crucial to that goal is measuring not only the direction of the public's preference, but the intensity with which people hold their views - and thus their potential willingness to be influenced by their political leaders.

Greenberg's Dispatches is a testament to the importance of this dimension of public opinion so often ignored by our major media polls.

[1] David W. Moore and Jeffrey M. Jones, "Permissive Consensus: Toward A New Paradigm for Policy Attitude Research," revision of a paper presented at the annual meeting of the American Association for Public Opinion Research, May 16-19, 2002.

"Reaction" Polling

2009-02-05T11:13:21-05:00

In his post on "Hired Gun" Polling, Mark Blumenthal suggests the need for pollsters to "distinguish between questions that measure pre-existing opinions and those that measure reactions."

He makes an important point. Much of what pollsters offer to the world as "public opinion" is in reality hypothetical, based on giving respondents information that many in the general public may not have and then immediately asking respondents for their reaction to that information.

Such results can be illuminating, but pollsters recognize that feeding respondents information means the sample no longer represents the American public and what Mark calls its "pre-existing" opinion. Unfortunately, many pollsters fail to acknowledge the hypothetical nature of such results, and instead treat them as though they represent the current state of the public's views.

The problem with this kind of approach is illustrated in the case that Mark discussed in his post, dealing with the "card check" bill, the proposed Employee Free Choice Act (EFCA) concerning the authorization of unions in the workplace.

The vast majority of Americans, one can reasonably assume, have little to no knowledge of the provisions of the bill. Thus, to measure "public opinion" on the issue, pollsters feel they need to tell respondents what the bill is all about. A Republican pollster explained the bill one way, a Democratic pollster another way, and - to no one's surprise - they ended up with a "public opinion" that reflected their respective party's position on the issue.

While one may argue the relative merits of the questions used by the two pollsters, the main point is that informing the public of any major policy proposal is intrinsically biased. Pollsters have to decide what is important among all the various elements of the proposal, and they can often come up with quite different conclusions. This problem applies to public policy pollsters as well, who - we can reasonably assume - have no partisan agenda, but who nevertheless can produce what appear to be partisan results.

Such problems have multiplied with the recent public policy polling on the bailout proposals for Wall Street and for the auto industry, and on the stimulus plan being considered by Congress. Most pollsters assume the public has little specific knowledge of such proposals, and thus pollsters provide respondents specific information to measure the public's (hypothetical) reaction to the proposal.

When CNN described the proposal to bailout the auto industry by characterizing it as "loans in order to prevent them from going into bankruptcy," in exchange for which the companies would produce plans "that show how they would become viable businesses in the long run," it found a 26-point margin in favor (63 percent to 37 percent). But when an ABC/Washington Post poll only a few days earlier had mentioned the word "bailout" in its question and did not refer to plans leading to the companies becoming viable, the poll showed a 13-point majority against the proposal (55 percent to 42 percent).

Again, one can debate the relative merits of the two questions, but the tendency of pollsters is to say that each set of results provides different insights into the dynamics of the public's views on this matter. In short, each provides a picture of potential public reaction to the proposal, if the proposal is framed to the general public in the way each polling organization presented the issue to its respondents.

That distinction is generally lost in the news reports. Each polling organization instead announces its results as though they reflect the current views of the public, over which the polling organization had no influence. But the reality is that the polling organization inevitably shapes its results by the very way it presents the issue to respondents.

As Mark argues, such "reaction" polling has a useful role to play in the public discourse on public opinion. However, it's also important that pollsters make clear that their results do not reflect "pre-existing" opinion (opinion before the polls were conducted - though one might instead use the word "extant" opinion), but rather hypothetical opinion under restricted conditions.

It used to be that newspapers made a formal distinction between "hard news" articles and "analysis" articles - clearly labeling the latter as such. That procedure doesn't seem to be followed these days, but it may be a useful analogous model for pollsters. Perhaps, in a similar way, pollsters can devise a method to formally separate their reports of potential "reaction" public opinion from existing public opinion.

I can envision, for example, one article in a newspaper that describes how many (or how few) people are actually aware of an issue and how many express ambivalence about the matter, while another article could explicitly describe how the public might react if the issue were universally framed in one way or another. Pollsters have made such distinctions sporadically, which is why we know that the public is more likely to support "rescuing" the auto industry than "bailing it out."

Still, the need for a formal and widely accepted method of distinguishing "reaction" questions from those that measure existing opinion needs to be found, if pollsters are to avoid the confusion that occurs when highly reputable polls produce wildly contradictory results.

"Manipulating" Public Opinion

2009-01-30T13:09:30-05:00

My colleague, Mark Blumenthal, has recently posted his reaction to an earlier post of mine, in which I suggested that most major media pollsters deliberately manipulate public opinion, in order to make it appear as though most of the public has an opinion on an issue. My examples included polls about the stimulus package, and how the public viewed the Democrats' control of the three branches of government.

In his critical remarks, Mark suggested that I was being a bit narrow in my view of public opinion and unfair in implying a nefarious motive on the part of the pollsters. In a later blog on the same issue, he noted that he had received reactions from two different pollsters, who did not want to reveal their names, one who works on campaigns and the other who works for the media. They provided somewhat different takes on his discussion of public opinion about the stimulus package, takes which I think tend to support my criticisms of media polls.

More about that in a moment. First, let me say that I appreciate Mark's well-considered criticisms, recognizing that they probably also reflect the views of many other practicing media pollsters (though let's hope not all). And I appreciate the opportunity that Mark offers for me to blog on this site about these issues, because I think such conversations are at the heart of the scientific enterprise - as does he. For that I am grateful.

As to the flaws in my concept of public opinion, I think Mark may misunderstand my recent focus on the lack of "no opinion" measures. Like Mark, I think that is just one part of measuring public opinion, but still a crucial one. Mark seems to agree, writing that "Yes, it is important to understand that many Americans lack a specific opinion on the 'economic stimulus' legislation per se, something stressed by too few pollsters. Still, that finding is just one part of 'public opinion' on this issue." (emphasis added)

I couldn't agree more. My view is that at on important policy matters, pollsters should measure at least three dimensions of public opinion: 1) direction of support (from support to opposition), including the magnitude; 2) intensity of views; and 3) the absence of a meaningful view on the matter, or non-opinion. In my recently book, The Opinion Makers, I elaborate more fully on this concept. (For the time being, I will ignore the oft-neglected measure of intensity.)

If measuring the direction of public opinion and measuring non-opinion are both important, why don't we find both measures in most polls? Look at the graph below - these are the poll results that Mark assembled from the various polls in his critique of my commentary. All of them measure direction of opinion, as we would expect, but only one attempts to measure non-opinion (NBC/WSJ). (Mark suggests that Rasmussen may have provided an explicit "no opinion" option, but Rasmussen's topline with the actual question shows it was a forced-choice format, with "no opinion" a volunteered option.)

(The graph below takes the difference between the percentage of people who support and the percentage who oppose the stimulus package, as described in the respective polls, which is then plotted as the "margin in favor" - since all polls showed more people in favor than opposed to the stimulus. The graph also shows the percentage of people without an opinion, as reported by each poll.)

Mark acknowledges that too few pollsters stress the percentage of people who lack an opinion (see italicized part of his quotation above), and his (and my) concerns are amply illustrated in the graph. Only 1 percent have no opinion according to CNN, just 3 percent say ABC/WP, 4 percent says Ipsos, 7 percent say NBC/WSJ, and 11 percent to 12 percent say Gallup and Hotline. Those are hardly credible numbers, if we are referring to a specific stimulus package (rather than to the general idea of some kind of stimulus), as all of these results do.

Indeed, Mark says the consensus of the two pollsters who contacted him in the wake of his critique was that "both imply agreement on one thing: Most Americans know little about the 'economic stimulus plan,' except that the President and the Congress are talking about it." Mark also adds at the end of his commentary that "'tepid' support is about the right phrase to use" to describe public opinion on the issue.

If that is the case (and I tend to agree with it), how did these pollsters arrive at that conclusion? Certainly not by looking at Gallup, Hotline, Ipsos, NBC/WSJ, CNN or ABC/WP. All those pollsters suggest very few people unsure about the specific stimulus plan being considered by Congress, and very large majorities in favor. None of these pollsters suggested "tepid" support and widespread ignorance.

Mark also writes that besides measuring non-opinion, "reactions to new information are also important, as are the underlying values driving responses to all of the questions reproduced above." Again, I don't disagree, though he implies that I do. It is not mutually exclusive to measure non-opinion (which most pollsters fail to do) and also to measure reactions to new information or underlying values.

The question is why do most pollsters fail to measure non-opinion?

(Separately, why do most pollsters fail to measure intensity? That's also an important dimension, but I'll talk about the failure to measure intensity at a later time.)

My response is that most pollsters don't measure non-opinion in general because they don't want to reveal that a sizeable number of Americans don't have an opinion on important policy matters. Mark thinks I'm being unfair, and that I'm attributing nefarious motives to such pollsters.

Here's the dilemma: Mark and I both agree that non-opinion is a crucial part of measuring the public's position on policy matters. We also know that all the major media pollsters do measure non-opinion from time to time. So, why don't they measure non-opinion on such major issues as the stimulus? What criteria do they use to determine when they will, and when they won't, ask questions with an explicit "no opinion" option?

The simple answer is that the news media wouldn't find it interesting to constantly report that large segments of the population don't have a meaningful opinion about the major policy issues facing the country. That's why pollsters feed respondents information and asked forced-choice questions, all in an effort to reinforce the myth of a highly informed, rational, and engaged public. It's a far more newsworthy myth than the reality of a public with many people uninformed and unengaged on issues, and thus lacking meaningful opinions.

I'm not arguing that all people fit that category. I'm only arguing that pollsters and the media should be willing to admit the existence, and measure the size, of such large segments of the public, instead of manipulating respondents to come up with answers so that it will appear as though virtually all Americans have a meaningful opinion.

In 1942, Elmo Roper wrote in an essay for Fortune magazine, titled "So the Blind Shall Not Lead," that even then, less than a decade since the advent of modern polling, "the emphasis in public opinion research has been largely misplaced. I believe its first duty is to explore the areas of public ignorance."[1]

Exploring areas of public ignorance may not necessarily be the pollsters' first duty, but it is certainly an important duty they usually fail to perform.

[1] Elmo Roper, "So the Blind Shall Not Lead," Fortune, 25, No. 2, p. 102, cited in George Bishop, The Illusion of Public Opinion: Fact and Artifact in American Public Opinion Polls (Lanham, Maryland: Rowman & Littlefield Publishers, Inc., 2005), p. 6.

George Bishop's and David Moore's 2009 Top Ten "Dubious Polling" Awards

2009-01-29T10:39:11-05:00

My colleague at the University of Cincinnati, George Bishop, and I have launched what we expect to be an annual listing of the Top Ten "Dubious Polling" reports for the previous year. Posted on Stinky Journalism.Org, we intend this as a satirical look at some of the practices of the major media pollsters.

As the opening paragraph notes: "Every year, poll watchers are confronted with poll results and commentary that defy either logic or science, often raising question about the very utility of polls. Typically, the problems are not with the method of conducting polls, but with the pollsters themselves - as they focus on what they believe is entertaining and appealing to the audience rather than an accurate reflection of public opinion. In the process, pollsters manipulate public opinion or write commentary that makes a mockery of what the public is really thinking."

Each award is ranked, from a low of one set of crossed fingers to a high of five sets. Pollsters generally know in their hearts when all is not right with their polls, but they (figuratively) cross their fingers and hope that no one notices anything amiss. The five crossed-fingers icon is the ultimate in wishful thinking, perhaps the equivalent of football's "Hail Mary pass" for the truly untrustworthy poll.

Our top award - earning the five crossed fingers - goes to all the major media polls[1] for their prediction of Giuliani as the early Republican frontrunner. Collectively this group, beginning more than one year prior to the first statewide electoral contest in Iowa, relentlessly, and without regard for any semblance of political reality, portrayed Rudy Giuliani as the dominant Republican candidate in a fictitious national primary.

Other "Dubious Polling" awards are:

·        Loopiest Poll Award: Pew Research Poll, for weekly pre-election polls in October that showed wild swings in Obama's lead.

·        Shooting Yourself in the Foot Award: Gallup Poll, for publishing two polls on Feb. 25, 2008, that contradicted each other.

·        Over-the-Top Gloating Award: Gary Langer, polling director of ABC, for writing that "What I liked best about the final New Hampshire pre-election polls [which erroneously predicted Obama to win] is that I didn't do any of them" - cleverly completing his polling in the Granite State far enough away from the election to avoid having his results compared with the election outcome.

·        180 Degree Award: CBS News/New York Times and USA Today/Gallup polls, for coming to opposite conclusions about the controversy over Rev. Jeremiah Wright.

·        Waiting for Godot Award: The American Association for Public Opinion Research Committee that still has not issued a report on the erroneous predictions in the N.H. Democratic Primary.

·        Who Knows? Award: Pew Research, ABC News/Washington Post, and Los Angeles Times/Bloomberg polls for contradictory conclusions about public support for Wall Street bailout.

·        Wake-Me-Up-When-It's-Over Award: NPR, Kaiser Family Foundation, and the Harvard School of Public Health Survey for a vague 131-word question.

·        Flip-Flop Award: CNN for two December polls that showed opposite results of the public's support for the auto bailout.

·        For Sale! Award: Peter D. Hart Research Associates, for their General Motors-sponsored poll that found (surprise! surprise!) overwhelming public support of auto industry bailout.

For a full description and rationale for the awards, go to Stinky Journalism.Org.

[1] These include polls by the Associated Press, ABC News/Washington Post, CBS News/New York Times, CNN, FOX, NBC News/Wall Street Journal, USA Today/Gallup, Newsweek, the Los Angeles Times/Bloomberg, and Pew Research.

Why Pollsters Manipulate Public Opinion

2009-01-23T12:10:21-05:00

Two recent polls, one by Gallup and the other by CNN, illustrate how easy it is for pollsters to manipulate public opinion into something different from what it really is.

The Gallup poll, Jan. 6-7, 2009, attempted to measure the public's reaction to a federal government stimulus package, with the question phrased as follows:

Do you favor or oppose Congress passing a new 775 billion dollar economic stimulus program as soon as possible after Barack Obama takes office?

In almost the same time period, the NBC News/Wall Street Journal poll also attempted to measure public opinion about the stimulus package, with a question that provided for a "don't know" option:

Do you think that the recently proposed economic stimulus legislation is a good idea or a bad idea? If you do not have an opinion either way, please just say so.

The results are shown below:

The margin in favor of the stimulus package is virtually identical in the two polls, 16 and 17 percentage points, but instead of being able to report a majority of Americans in favor, NBC and the Journal had to report that a "plurality" of Americans were in favor, with a substantial portion of the public ambivalent or unengaged. Gallup, by contrast, could report (although erroneously) that a majority was in favor.

It's not always the case that the margin in favor of a proposition is always the same in both ways of measuring public opinion, as is illustrated in the following case. When CNN wanted to discover whether the public was copasetic with Democratic control of all three branches of government, it asked a forced choice question (Nov. 6-9, 2008):

As you may know, the Democrats will control both the Senate and the House of Representatives, as well as the presidency. Do you think this will be good for the country or bad for the country?

 

Again, coincidentally, another polling organization, Associated Press/GfK Roper Public Affairs and Media asked a similar question at virtually the same time (Nov. 6-10, 2008), though this question allowed for a middle position:

 

As you may know, the Democrats will now control the House of Representatives, the Senate and the presidency. Do you think it good for the country, bad for the country, or does it not really make a difference that the Democrats now control the House, the Senate and the presidency?

The results of the two polls show two very different publics:

In this case, both polling organizations deliberately manipulated their respondents to come up with an opinion (even if, in the case of the AP/GfK poll, to say the issue didn't matter) by giving them information up front. Why did they need to tell respondents that the Democrats controlled all three branches? Why not find out how many people knew that, and then - among those who knew it - ask whether it was good or bad, didn't it make a difference, or didn't they have an opinion?

But the major media pollsters are generally not interested in realistic measures of public opinion. On the matters discussed here, Gallup and CNN clearly do not want to report how many people don't have an opinion or might want to take a valid middle position on the issue. Instead, these pollsters believe it's more interesting to create a "public opinion" that reflects a highly engaged and decisive public.

For CNN to say that 97 percent of Americans believe Democratic control of the government is either "good" or "bad," and for Gallup to claim that nine out of ten Americans have an opinion about the stimulus package, may fit their journalistic needs - but they know, and we know, it's simply not true.

The Fluctuating Convergence Mystery

2008-12-08T15:58:45-05:00

The "convergence mystery" gets even more mysterious.

In Survey Practice, I initially raised the question of why the national presidential polls showed a great deal of variance in their results during the month of October, but then converged to a relatively tight cluster in the final predictions. Mark Blumenthal then calculated the variance among state polls, showing that they also exhibited much greater variance during October than in their final predictions.

He suggested the phenomenon was probably not a deliberate effort by pollsters to change their numbers. Instead, he proposed that pollsters, whose numbers were outliers, probably looked to see if their polls needed "fixing" - and sure enough, they found reasons to adjust their numbers closer to the mean. Thus, the convergence at the end of the campaign.

My original analysis was a weekly average of the national polls, while Mark's was a weekly average of selected state polls (including 12 battleground states with at least 20 polls in October/November). In a further analysis, I looked at the eight tracking polls from October 4 through November 2. The group includes four daily tracking polls for the whole time period, and four that started a bit later - two on Oct. 6, one on Oct. 12, and the last on Oct. 16.

Shown below is the overall graph of their results.

A quick examination shows a couple of times when the polls converged to a tight cluster before expanding to much greater differences - around Oct. 18 and again around Oct. 28-29.

The next graph shows the same polls, but with the "variance" plotted on the same graph (the pink line). What I hadn't noticed in the graph of all the polls are the three spikes in variance shown below.

The next graph is a scatterplot of the variance. The linear regression line indicates a significant decline in variance over the month of October, though clearly there are spikes.

The last graph shows the day-by-day fluctuation, with three major spikes, all occurring just a couple of days after each of the October debates.

The first spike occurs on Oct. 7, the second on Oct. 12-13, and the last Oct. 20-22. In each case, the spike begins five days after a debate. It's important to keep in mind that the daily tracking polls are typically about 3-day rolling averages, so that means the spike occurs two days after the debate.

These results add to the mystery of convergence, because they 1) show an overall decline in variance over the month, and 2) nevertheless show sudden and temporary spikes in variance, starting just two days after a vice presidential or presidential debate. The largest spike occurs right after the third presidential debate on Oct. 15 - not immediately reflected in the 3-day tracking polls until five days after the debate.

The delayed spikes can be accounted for in this way: The vice presidential debate took place on Oct. 2. The next day, the networks broadcast their interpretations of the debate, and the following day, the polls begin to show quite different results. The debate effect is not complete until the end of the 3-day tracking period, which would mean the first full results would be manifest on Oct. 7, five days after the debate.

Similar scenarios suggest that five days after each of the two succeeding debates, new spikes should occur - and they do. Oct. 12 (five days after the second presidential debate) and Oct. 20 (five days after the final presidential debate) find the beginnings of spikes - the first lasting two days, and the second lasting three days, before beginning the downward movement.

There is one last minor spike, from the end of October to the final prediction figures. It's hard to tell if this is random noise, or part of a predictable pattern.

In any case, the mystery is this: Why do the eight tracking polls show more variance in results following the debates? What is there about the debates that would cause different polls to show greater inconsistencies in results than normal? And why do the polls show a month-long decline in variance, except for the three temporary spikes?

I think that Mark's initial suggestion -- that pollsters with the outlying results tend to "fix" their methodology, and thus have their polls converge toward the mean - may need to be re-examined in light of the tracking poll data. The decline in the variance is gradual over the month, but interrupted by the debate-generated spikes.

Please offer any theories you might have that could explain this phenomenon.

Poll Performances: Crazy October

2008-12-01T14:17:22-05:00

The final presidential contest predictions of the major media polls all came pretty close to the actual results, predicting Obama to win by anywhere from 5 to 11 percentage points (he actually won by 6.7 points).

However, the polls showed a great deal of variability even during the last four weeks of October leading into the election, raising questions about how to measure poll "accuracy" during the election campaign itself.

Shown below is a graph of the results of 10 polls that publicized results for at least the final three weeks of October.

An examination of the daily tracking polls provides no better picture of poll accuracy.  

The differences in the overall trends are quite substantial, as are individual points.

On October 12, IBD/TIPP shows an Obama lead of two points, while DailyKos says it's 12. The next day, IBD/TIPP produces a 3-point lead, while GWU has a 13-point lead. On October 25, GWU's lead is just three points, while DailyKos has it at 12 points. Even right before the election, IBD/TIPP shows just a 2-point lead, while ABC/WP says the lead is 11 points.

More important are the many different pictures of the dynamics of the race. If we single out, say, DailyKos from GWU, one would never know they are measuring the same contest - except that they both converge in their final predictions. Another example: Rasmussen shows only a little variability in the race, between an Obama lead of 3 to 8 points, ending at 6, while GWU goes from 13 points down to 1, finally ending at 5. Likewise, Zogby's description of the campaign dynamics shows a relatively stable race for the first half of the month, followed by a major surge, a big decline, and then a last minute surge. DailyKos usually had the most optimistic Obama leads, mostly double digits, except for the middle of the month, and then at the end when the lead declined to just five points.

What can we say about poll performances when there are such different stories about the October dynamics? The notion that the polls were mostly "accurate" must be modified to reflect how divergent they were during the campaign.

Exit Polls and the Undecided Voters

2008-11-07T08:36:37-05:00

The 2008 exit polls suggest that most the major media pollsters missed an important part of the presidential campaign, as they either failed to measure or mostly ignored the large undecided group of voters just after the major party conventions officially nominated their candidates, and its diminishing size over the next two months.

Unlike in 2004, the 2008 election polls obtained somewhat more detail about how undecided the voters were, and the results support my argument made on pollster.com several times previously, that many voters mull over their decisions until late in the campaign period.

In the 2008 election, the exit polls show that 4 percent of voters said they made up their minds on Election Day, another 3 percent in the previous three days, and an additional 3 percent within the past week - for a total of 10 percent. That's virtually the same as the 11 percent who said they had made their decision in the past week in 2004 - with 5 percent saying the day of the election, 4 percent the previous three days, and 3 percent the past week.

When Voters Made Their Decisions 2004 and 2008
EXIT POLLS (Shown on CNN - click column headings at right)	CNN.com Election 2004	CNN.comElection 2008
	%	%
Day of election	5	4
Previous 3 days	4	3
Past week	3	3
TOTAL (past week)	11	10
Last month/In October	10	15
In September	n/a	14
Before September (2008)	n/a	60
Before October (2004)	78	n/a
TOTAL before past week	88	89

More interesting, the 2008 exit polls suggest that only 60 percent of voters had decided whom to support before September, with about four in ten making up their minds after the major party conventions in August.

While it is certainly difficult for a voter to pinpoint exactly when he or she made a final decision, some pre-election polling data from this year suggests the exit poll results may be pretty good approximations. Of course, most pollsters don't measure the undecided voter directly (which they could do, by asking whom voters intend to choose on Election Day and then asking, "or haven't you made up your mind yet?"), but instead pollsters will often do so indirectly. After the hypothetical, forced choice vote question, for example, the CBS/New York Times poll sometimes asks, "Is your mind made up, or is it still too early to say for sure?" CBS reports that in mid-August 2008, about a third of all registered voters were "uncommitted" - they had either not chosen a candidate initially, or they had mentioned a candidate but then said it was still to early to say for sure if their minds were made up.

A somewhat larger undecided voter group was measured in a 1996 Gallup poll, conducted Sept. 3-5, 1996, which asked voters up front if they had made up their minds - rather than the standard "who would you vote for if the election were held today" question. In that format, 60 percent said they had made up their minds, while 39 percent said they had not, and 1 percent were unsure. Those 1996 figures are similar to what the 2008 exit poll responses suggest as well.

Below are two graphs of voter preferences. The first is based on a reconstruction from the exit poll crosstabs, which show voter preferences including the undecided vote. Of course, such a reconstruction needs to be viewed cautiously. It's difficult for people to remember exactly when they made up their minds, so at best this graph is an approximation of what voter preferences might have looked like for each month.

As you can see, this first graph shows more voters undecided than choosing either of the two major candidates before September, and it shows the decline in the undecided group over time. (Obviously, each time period on the X-axis is not proportional to the number of days in the time period, but the general pattern is obvious.)

The second graph is a reconstruction (averaging) of Gallup's daily tracking poll, using the likely voter results when available, and the registered voter results otherwise. The "last week" results are based on just four days of the week before the election, while the "last 3 days" are based on just those days from the tracking poll. I used this method to approximate the exit poll categories and provide a comparable base of analysis.

In contrast with the first graph, the second graph of the Gallup tracking poll shows no significant change in the undecided voter group from August through Election Day. In fact, Gallup's daily tracking poll, which goes back to March 2008, shows a steady 5 - 6 percent undecided group for the whole seven months - something that not even Gallup researchers can argue (with a straight face) is accurate.

If we believe that the exit polls have any validity in measuring opinion, it's hard to deny the superiority of the first graph in giving poll consumers an accurate picture of the changing electorate during the campaign. The declining size of the undecided vote over the course of the campaign is clearly an important dynamic in the campaign, regardless of whether pollsters will acknowledge it.