Brian Schaffner

Does Survey Mode Still Matter?

2011-05-09T12:04:43-04:00

Over the past several years, pollsters have been working to understand the best approach to conducting survey research in a world where no single technology allows them to reach the entire public. Most reputable polling organizations have adapted their national telephone polls to call both landlines and cell phones in order to deal with the growing cell-phone-only population. Yet, these pollsters still must deal with the fact that response rates for telephone polls are generally under 25%, and adding cell phones to their samples have greatly increased the cost of polling.

At the same time, many firms have been developing technology with the aim of producing reliable and accurate opt-in Internet surveys. These surveys can generally be produced for half the cost of a telephone poll, but they diverge from traditional approaches to survey research because they do not rely on a probability sample. The proliferation of opt-in Internet surveys has generated some controversy within the survey research community (for example, here, here, here, and here). In 2010, the American Association of Public Opinion Researchers (AAPOR) issued a report warning against using opt-in Internet surveys to estimate population values, but also noting that significant evidence on this question was lacking. Indeed, many of the studies that AAPOR relied on when reaching their conclusions were based on outdated data.

About the same time that AAPOR was releasing its report, Stephen Ansolabehere and I were in the field with a study to compare how three different survey modes fared when administering the same questionnaire. The modes we examined were a combined landline/cell telephone sample, a mail survey, and an opt-in Internet survey. We contracted with YouGov/Polimetrix to conduct the study. While we will be presenting the findings from our study on Friday at the annual meeting of AAPOR, our report can be found here.

Overall, we found few, if any, differences between the opt-in Internet survey and the telephone poll. Specifically:

1) For measures that we could validate with government data, both the telephone poll and the opt-in Internet survey produced an average error that was nearly identical in size (about 5 percentage points, on average).

2) For political measures that we could not validate, the differences between the phone and Internet survey were generally small (with a few exceptions). For example, the average difference between estimates of political attitudes and opinions generated from each survey was about 5 percentage points. Furthermore, neither survey was consistently more liberal or conservative on these measures.

3) The correlational structure of the data was not significantly different across the phone and Internet surveys.

Thus, as we conclude in the report:

"Overall, our findings indicate that an opt-in Internet survey produced by a respected firm can produce results that are as accurate as those generated by a quality telephone poll and that these modes will produce few, if any, differences in the types of conclusions researchers and practitioners will draw in the realm of American public opinion."

What Happened to Scott Brown Voters?

2010-11-10T10:30:59-05:00

Whether you want to call the 2010 midterm elections a wave or a tsunami, Scott Brown was at the leading edge of the Republican tide when he scored an unexpected victory over Martha Coakley in the January special election for Ted Kennedy's Senate seat in Massachusetts. But while Brown scored the first major Republican victory of 2010 in the blue-leaning state of Massachusetts, Republicans failed to make much in the way of additional inroads in the state in November. Democrats held on to all 10 House seats and Governor Deval Patrick won re-election over Charlie Baker. These Democratic victories raise the question: what happened to those Scott Brown supporters in this midterm election?

We conducted a statewide exit poll of Massachusetts voters on November 2nd and we asked those voters who they had voted for in the January special election. Interestingly, just 42% of Massachusetts voters said that they had cast a ballot for Scott Brown in January; 44% said that they had voted for Coakley and 12% said they had not voted in the January special election. The last statistic may be the most interesting. After all, turnout in November was only slightly higher than it had been in January. For more than one in ten November voters to have not voted in the special election had to mean that a non-trivial share of special election voters failed to come out to vote this time around. Thus, there appears to be some group of Massachusetts voters that Brown mobilized in January but whom Baker was unable to energize in November.

The chart below compares the gubernatorial votes of Brown voters to those of Coakley supporters and citizens who had not voted in January. Not surprisingly, Coakley voters became supporters of Patrick and Brown voters who turned out in November mostly cast their ballots for Baker. But where Patrick really generated his winning edge was among citizens who had not voted in the January special election. Patrick won that group by a 2-to-1 margin.

So who were these voters that stayed home in January but came out to vote in the November election? Our exit poll results show that they were almost twice as likely to be women than men. They were also more likely to be Democrats or independents and less likely to be Republicans. And these voters were also younger -- almost one in five of those who did not vote in the special election but voted in November were between the ages of 18 and 29.

Voters who did not vote in January but came out in November were also more likely to approve of Obama and disapprove of the Tea Party. Nearly 70% of November-only voters approved of the job Obama is doing as president, compared to 57% among those who voted in both elections. On the other hand, just 27% of those not voting in the special election approved of the Tea Party movement, compared to 43% for voters who turned out in both contests.

Overall, Brown voters did not appear to show up in the same numbers to support Charlie Baker in the November election. Instead, 10% of the November electorate was comprised of people who had not voted in January, and they tended to have views that were much more favorable towards Obama and the Democrats. This allowed Democrats in Massachusetts to do fairly well despite an unfavorable national electoral climate. Just ask Coakley herself how much difference 10 months made in Massachusetts. In winning re-election as Attorney General she received support from 25% of those who had voted against her in the January special election and 66% of those who had not turned out in January, winning easily as a result.

Massachusetts Exit Poll: Deval Patrick Owes His Reelection to Women

2010-11-02T23:52:35-04:00

The National Election Pool did not conduct an exit poll for the Massachusetts gubernatorial race this year. However, I led a team of students at the University of Massachusetts, Amherst to sample voters at 18 precincts across the state today. You can find specifics on our Massachusetts Exit Poll survey, including preliminary results, here.

One finding clearly jumps out from our preliminary results: Deval Patrick (D) appears to owe his victory today to women. As the chart below shows, men divided fairly evenly between Patrick and his Republican challenger, Charlie Baker. But women broke strongly for the incumbent, by a margin of more than 20 percentage points. That provided the winning margin for Patrick.

Not surprisingly, the economy was the most important issue for Massachusetts voters. Interestingly, while 42% of Massachusetts voters said that they thought the economy was worse than it was a year ago, a similar proportion (48%) said it was better, with the rest indicating that it was about the same. Patrick actually carried 50% of the vote among the almost nine in ten voters who said the economy was very important to their vote. Patrick also won among those choosing education and health care as an important issue, with Baker carrying voters who viewed taxes as very important to their vote.

Despite Patrick's victory, the Tea Party movement did appear to demonstrate significant strength among the Massachusetts electorate. Forty percent of Massachusetts voters said that they either strongly or somewhat supported the Tea Party Movement, and about three of every four Tea Party supporters voted for Baker.

Finally, youth turnout did not appear particularly strong in Massachusetts. Our exit poll indicates that about 10% of the electorate was between the ages of 18-29; a similar percentage as what was reported by exit polls in Massachusetts in 2006. This is down from 2008, when 18-29 year olds comprised 17% of the electorate.

Details on the UMass Exit Poll: 835 Massachusetts voters were surveyed leaving the polls in 18 randomly selected precincts across the state on election day.The survey has a margin of error of 3.4% with a 95% confidence level. Margin of error will be greater for subgroups.Calculations are produced using sampling weights to account for the sampling design. Results have not yet been weighted to account for non-response.

Correction: The percentages for voters' evaluation of the economy compared to a year ago have been fixed; they were initially only half as large as they should have been.

Will Hispanic Turnout Really Be Down in 2010?

2010-10-07T16:35:07-04:00

On Tuesday, the Pew Hispanic Center released a report suggesting that Hispanic turnout will likely drop significantly in the upcoming midterm elections. The report has generated a fair amount of news coverage, but the coverage has also been criticized for failing to provide sufficient context for interpreting the poll results. In this post, I hope to provide some of that context.

First some background. The Pew Center report finds that only 51% of registered (the reason for the italics will become clear below) Hispanics say they are "absolutely certain" they will vote in this year's midterm elections, compared to 70% of all registered voters who report the same. It is somewhat informative to learn that registered Hispanics appear to be less likely to vote in this contest than non-Hispanics, but that would not be particularly surprising given that Hispanics generally turn out at lower rates than non-Hispanics. The important contextual information that is missing is how these vote intentions compare to previous midterm elections. Or, as Joshua Tucker writes, "I don't know if 51% of Hispanics planning on voting in this election is better or worse for the Democrats than in previous elections." This is the key piece of information necessary to really judge what the Pew report is telling us.

To put the 51% figure into the proper context, I downloaded data from the Pew Center's early October turnout report from 2006. A word of caution about using this 2006 survey as a baseline for comparison: while the question wording and timing of the 2006 survey is basically the same as for the 2010 poll, the sampling approach in the 2010 survey is somewhat different. Both surveys used a RDD approach, but the 2010 sample is stratified to ensure an over-representation of Hispanic respondents. (More on the 2010 approach in the appendix to this report.)

Even with that in mind, the differences are actually pretty significant. In the 2006 study, 68% of registered Hispanics said that they were "absolutely certain" they would vote (compared to 71% among non-Hispanics). If we assume these figures can be compared, the 2010 survey would be reporting almost an 18 percentage point drop in the share of registered Hispanics saying that they were certain they would vote.

An 18 percentage point decline in the percentage of registered Hispanics intending to vote strikes me as quite large. To confirm that this drop was real, I looked for another poll that I could use to confirm that vote intentions were lower among registered Hispanics this year. I was able to access a June 2010 Gallup poll through the Roper Center Archives. While this poll was conducted a few months ago, it also asked respondents whether they were "absolutely certain" they would vote in the midterm elections. My analysis of the data from this survey reveals that 61% of registered Hispanics said that they were certain they would vote, compared to 75% of registered non-Hispanics. So the values are a bit higher, but the gap is still there. Recall that the 2006 Pew survey indicated no such gap between registered Hispanics and registered non-Hispanics.

Based on this bit of polling triangulation, it appears as though registered Hispanics are, in fact, less likely to vote in 2010 than they were in 2006. The question is why this might be the case. Most explanations for this drop are likely to focus on the numerator (the number of Hispanics intending to vote), with the assumption that something is making Hispanics who participated in 2006 less likely to participate in 2008. But it is also worth considering how much of this may actually be driven by a change in the denominator (the number of Hispanics who are registered to vote). After all, we know that the 2008 campaign did a lot to register citizens to vote, and many of these new registrants were Hispanics. Furthermore, most of the new registrants are also likely to be only occasional voters who would generally skip midterm elections. Thus, it is possible that the same number of registered Hispanics intend to vote in 2010 as did in 2006, but there are simply more registered Hispanics now than there were four years ago (or more of the Hispanics who now end up in surveys are registered to vote). In other words, the percentage who intend to participate might have gone down at least partially because the numerator grew, not because the denominator shrunk.

We can gain some preliminary insight into this hypothesis by returning to the 2006 Pew survey and the 2010 Gallup poll to see what percentage of Hispanics in those surveys reported being registered to vote. In the 2006 Pew poll, only 66% of Hispanic respondents reported that they were registered to vote; in the 2010 Gallup poll the figure was 80%. While I hesitate to go too far with these figures, they do suggest the possibility that the number of Hispanics voting in 2010 may not actually drop; instead, it may simply be the case that more non-voting (or infrequently voting) Hispanics are now registered to vote. Thus, as the political world continues to mull over the reasons for the expected drop in Hispanic participation this year, we should make sure that this decline is real, and not simply a function of an increase in Hispanics who are registered to vote.

Will Risk-Averse Voters Help Endangered Incumbents?

2010-09-28T14:32:54-04:00

There is little doubt that 2010 will be a bad year for Democrats, with many Democratic incumbents likely to lose to Republican challengers in both House and Senate races. The real question is how bad it will be -- will enough Democratic incumbents lose to shift control of the House (and possibly Senate) to the Republicans?

While many things are working against Democratic incumbents this year, the public's tendency to be risk averse may make the difference in allowing some incumbents to win campaigns that they might otherwise have lost. In a recent paper, David Eckles and I show that voters who are more risk averse are more likely to vote for incumbent House members, even when controlling for a number of other factors. The general idea here is that voters will be less sure about what a challenger will actually do once in office, so if a voter is averse to risk-taking, this will make them more willing to support the incumbent even when the challenger appears to be a pretty good alternative. People treat consumer products similarly. For example, one study found that 40% of experimental subjects preferred their "incumbent" skin lotion brand to a "challenger" brand, even though they recognized that the "challenger" brand was superior.

In our analysis of vote decisions in 2008 House races, we find that risk aversion is such an influential force that it often causes voters to vote for the incumbent even though they recognize that the challenger's ideology is significantly closer to their own. Since the vast majority of citizens are generally averse to taking risks, this constitutes a significant advantage for any incumbent candidate.

Of course, many polls are showing that citizens seem to prefer the Republican challenger in their House and Senate races in spite of this risk aversion. The question is whether voters will act more risk averse when they actually go to cast their ballots than they do when they express a vote preference to pollsters. To gain some insight on this question, I looked back at the 2008 Cooperative Congressional Election Study (CCES), which was conducted by YouGov/Polimetrix (disclosure: I was part of a team that sponsored the module to the 2008 CCES that I analyze here). The CCES includes both a pre- and post-election survey of the same voters; therefore, I am able to compare whom a voter said they preferred in October to who they actually reported voting for in November. The chart below shows the percentage of respondents that ended up voting for the incumbent House candidate depending on whether they said in the pre-election survey that they preferred the challenger, the incumbent, or if they were undecided. Overall, those who had settled on a candidate in October almost always stuck with that choice. About 94% of those who planned on voting for the incumbent did so while only 5% who planned on voting for the challenger changed their minds and voted for the incumbent. Those who were undecided in October split their votes almost evenly between the incumbent and the challenger.

The next chart shows how voters broke down when it came to their orientation toward risk. To classify voters as more or less averse, we used a question that asked respondents if they would take a hypothetical job offer under different conditions; see the paper or this post for more details. More risk-neutral voters are shown on the left and more risk averse voters are shown on the right. For the most part, both groups stuck with whichever candidate they had expressed a preference for in the pre-election survey. However, those who were undecided in October broke differently depending on their level of risk aversion. Undecided risk-neutral voters broke overwhelmingly for the challenger while a majority of undecided risk-averse voters ended up voting for the incumbent.

What can this analysis tell us about this year's midterm elections? Ultimately, the fact that the smaller group of undecided risk neutral voters (we classified 28% of voters as risk neutral) broke more heavily against the incumbent canceled out with the smaller break toward the incumbent among the larger group of risk-averse voters (we classified 72% of voters as risk averse). If the 2010 congressional elections go like they did in 2008, then undecided voters may break relatively evenly for the incumbent and challenger, and pre-election polls should prove relatively accurate. However, these findings do suggest a potential strategy for Democratic incumbents* in tight races. Endangered incumbents may benefit from characterizing their challengers as "risky," with the aim of capitalizing on the risk-averse tendencies that most voters have. Such a strategy may help these incumbents win over a large share of late-deciding, risk-averse voters.

------------------

* Appealing to risk aversion is a strategy that either party's incumbents should benefit from. However, there are few endangered Republican incumbents in this election cycle, which makes this a particularly pertinent strategy for Democratic incumbents in 2010.

Framing Risks, Losses, and Costs During the Health Care Reform Debate

2010-04-28T08:42:27-04:00

Brendan Nyhan posted yesterday about his article in the just-released special issue of The Forum on the politics of health care reform. There are several compelling articles in the issue by notable scholars, including Representative David Price.

My own contribution to the issue (along with co-author David Eckles) is an expansion of an earlier post on this blog. Here is the key take-away from our piece:

While a large majority of Americans did see rising health care costs as a problem, very few of these same people thought that reform would improve this situation, and when it came to whether people supported or opposed the reform plan, it was the anticipated costs of the legislation, not concerns about current rising costs, that appeared most salient to Americans. Ultimately, Democrats passed health care reform legislation in spite of their inability to secure significant public support for the plan. Yet their efforts to mitigate the effects of loss aversion on public support for the proposal may have kept even more Americans from opposing the legislation, and if Republicans mount a serious attempt to repeal the reform law, it will be Democrats who are appealing to the public's
aversion to risk and loss.

For the most part, the public agreed that rising health care costs were a major issue and that something had to be done to curtail those costs. However, they also tended to agree with Republicans that the health care reform legislation was not going to help limit those costs. In fact, a significant proportion thought it was going to make them worse. And, as the figure below indicates, prospective views about how the legislation would influence costs had a much more influential role in structuring opinion on the health care reform legislation than did concerns about current rising costs.

Another major point of our article is the importance of loss aversion; that is, the public's tendency to over-value what they already have and under-value what they do not yet own. This tendency worked against Democratic efforts to win public support for health care reform, but it is also why we argue that now that people have been given health care reform, it will likely be quite difficult for Republicans to attempt to take it away.

Check out our piece here, and all of the other great contributions here. (Free registration is required for access).

Dispatch from the Bay State

2010-01-19T16:26:07-05:00

When I moved from DC to Amherst in August I was looking forward to the charm of a small New England college town and the relative affordability of housing (compared to prices inside the beltway, at least). But what I knew I'd miss the most was living at the center of the political universe. Well, for one day at least, I get to re-live the excitement as all eyes turn to the Bay State.

Warning: what follows is entirely un-scientific and is, accordingly, of little use to understand what might happen when the polls close tonight.

I've been surprised over the past few weeks at how much of a ground game Brown seems to have in Western Massachusetts compared to Coakely. Last weekend, I had to drive to nearby Belchertown for a swim meet and we passed countless yard signs for Brown as well as a small rally of Brown supporters. Not a single sign for Coakley. Today I had to travel to Boston for an appointment and I was 40 minutes into the drive before I saw my first Coakley sign.

Of course, once I got into Boston things changed quite a bit. Coakley signs were much more prominent and the handful of polling sites I passed were packed. This is consistent with what the news has been reporting regarding high turnout. But what was most interesting from my vantage point (and the vantage point of any pollster trying to determine who will and will not vote today) was a conversation I had with a young Democratic store clerk who, upon finding out that I was a political science professor, started complaining about how he ended up having to get up early today to go vote when he had been planning all along to skip this election. I've had similar conversations with a number of Democrats over the past few days; people who had no intention of voting a week ago, but now feel compelled to do so. These people are not at all excited about the Coakley campaign, but they suddenly feel as though they have to go out to vote in an election they were planning on skipping.

The interesting question, of course, is how would these voters be treated by a likely voter screen? And would response bias affect the propensity with which these types of voters would turn up in pre-election polls? It will be interesting to see how this plays out tonight. Like Mark, if forced to wager on the outcome, I'd have to put my money on Brown. But there is no way I'd want to put my money down on either candidate in an election like this one.

A Victory for IVR Polling?

2009-11-04T14:26:08-05:00

A friend sent me a couple of links earlier pointing to pundits and pollsters who are taking last night's results as evidence for the merits of IVR polling. First off, as Mark noted earlier, it is a bit too early to be making such comparisons. With regard to the claims being made about IVR polling in particular, I would add the following points:

First, there is no way to control for other reasons that these polls might have generated different results, including different approaches to screening for likely voters and how undecideds are dealt with. With regard to the latter issue, it is important to note that the pollsters using live interviewing in New Jersey were showing more than twice the percentage of undecideds as those using IVR.

This leads to a second important point (related to the first): comparing these pollsters based on the final result presupposes that each pollster that has been entered into this fictitious competition was actually trying to get the final result correct in the first place. If that was the goal, then it seems as though each polling firm would have allocated all of their undecided respondents into one camp or another.

Third, one of the reasons for concerns with IVR polling is that citizens with only a cell phone cannot be reached by these pollsters and these citizens now comprise at least one-fifth of the population. Yet, while the cell-only problem may generally be an issue for IVR technology (and for live interview pollsters who aren't calling cell phones), it is less of a problem for polling on elections, and particularly in low turnout elections. This is because the types of people that do not have landlines are less likely to be voters (and particularly less likely to be voting in low turnout elections). Ultimately, an off-year low turnout election may actually be less of a challenge for IVR-based polls because the non-coverage bias should be smaller for these contests. Where these polls may run into greater challenges is when they attempt to make inferences about the American public rather than registered (or likely) voters.

Loss Aversion and Opinions on Health Care Reform

2009-08-28T13:37:22-04:00

The New Yorker has an interesting piece on how the public's aversion to losses (or loss aversion) limits the extent to which they are willing to favor health care reform. That piece and some others that preceded it are worth reading to understand one reason that Americans may support the general idea of reforming the health care system, but then express far less support when confronted with the possibility that their own health care plans may be affected. The bottom line is that individuals tend to value what they already have to a much greater extent than what they might gain (this is often called the endowment effect). This means, for example, that people are far less willing to part with an item that they already have than they are to forgo receiving that same item if it has not yet been in their possession.

Justin Milner explained the relationship in the Baltimore Sun a few weeks ago:

"In the health care debate, loss aversion helps to color the public's perception of potential reform. A recent Gallup poll found a clear majority of Americans favor health care reform in the coming year. But when pressed on specific aspects of the health care, Americans are decidedly loss averse. Almost 90 percent of Americans want to be able to choose any doctor or hospital they like, and 77 percent of Americans say it is important to have the option to keep the health insurance plan they have now. In sum, we may want change and reform - but not at the cost of any of our current options."

In other words, those that already have health insurance probably overvalue that insurance relative to what would be available to them under health care reform legislation, and this may be driving down support for reform.

Fortunately, the survey released by the Economist yesterday provides a nice addendum to these readings by illustrating how loss aversion can significantly alter public opinion depending on how a question is framed. In this survey, the sample was split randomly into halves. The first half of the sample was asked to choose which of the following plans they preferred:

"A plan with no lifetime limit on beneï¬�ts."

or

"A plan that limited the total amount of beneï¬�ts in your lifetime to $1 million, but saved you $1000 per year."

Four out of five respondents (80%) answering the question framed in this way selected the first option. They'd much rather have a plan with no limit on benefits than save $1,000, but be subjected to a $1 million lifetime limit.

The second half of the sample chose between these options:

"A plan that limited the total amount of beneï¬�ts in your lifetime to $1 million."

or

"A plan with no lifetime limit on beneï¬�ts, but cost you an additional $1000 per year."

Functionally, these options are equivalent to those presented to the first half of the sample. In the first presentation, the limited plan will increase the respondent's wealth by $1000 per year by saving him or her that money; in the second presentation, the limited plan will increase the respondent's wealth by $1000 per year because that respondent will not have to pay the cost of the unlimited plan. However, the different framing of the options (emphasizing "savings" rather than "cost") is critical. Among those choosing from the second set of options opinion was more closely divided--44% chose the plan with limited benefits while 56% chose the unlimited benefits option. In short, more Americans wanted the unlimited plan when it meant forgoing a savings of $1,000 per year than when it meant incurring a cost of $1,000 per year.

It is also important to note that the changes in how the options are framed do not affect all groups equally. In particular, loss aversion appears to be conditioned by income. This makes sense since wealthier respondents may not be as sensitive to a $1,000 per year change in their wealth as those with lower incomes. To demonstrate the relationship, the chart below compares the percentage of respondents who would choose a plan with no lifetime limit depending on whether they received the question with the "savings" frame or the "cost" frame. Respondents are broken down into three income categories.

What stands out from this chart is that respondents in each income category are much more likely to chose the option with no lifetime limit when they received the question with the savings frame. However, under the "cost" frame, responses differed more significantly across income categories. Thus, among respondents making less than $40,000, support for the plan with no lifetime limit was 32 percentage points higher when that plan was presented as a way to forgo a savings of $1,000 rather than incurring a cost of $1,000. The framing effects were much smaller for those with higher incomes.

Of course, these aren't actually the choices being presented to Americans during the health care reform debate, but this survey experiment does provide a neat way of illustrating not only how the framing of health care reform as a potential loss can affect support for the measure, but also among which groups those frames will be most effective. Indeed, the New Yorker article ends by noting that it may still be possible to gain public support for health care reform despite the public's tendency toward loss aversion:

"The key may be to work with, rather than against, people's desire for security. That's surely one reason that Obama has consistently promised people that if they like the health insurance they currently have they can keep it. This promise will make whatever reform we get more inefficient and less comprehensive, but it also assuages people's anxieties. It might even be possible to use the endowment effect and the status-quo bias in the argument for change. After all, although people tend to feel that they own their health insurance, their entitlement is distinctly tenuous...Changing the system so that individuals can get affordable health care, while banning bad behavior on the part of insurance companies, will actually make it more likely, not less, that people will get to preserve their current level of coverage."

For the public to support health care reform, the reform needs to be framed as something that will help keep most individuals (who do have insurance) from losing what they already have. Furthermore, the analysis of the Economist survey suggests that individuals with lower incomes are most likely to respond to such an attempt to re-frame the debate in this way. This is notable since there is much ground to be gained among these individuals. In fact, the same survey shows that respondents in the lowest income group are substantially more likely than others to be unsure about whether the health care reform plan would make them better or worse off. This group appears to have their minds least made up on health care reform and their opinions may be the most susceptible to the efforts by both sides to frame this issue during the coming weeks and months.

The Public's View of Obama and McCain's Campaign Strategies

2009-06-17T15:18:52-04:00

Political pundits generally settle on a shared view of a campaign, one that includes a story about which groups each candidate worked hardest to win votes from. But how does the general public perceive the candidates' campaign strategies?

In 2008, I included a battery on the Cooperative Congressional Election Study which asked 1,000 American adults to indicate which types of people each candidate had focused more attention on (the actual wording of the question was "During the presidential election campaign, which of the following groups do you think [Barack Obama/John McCain] has focused most of his attention on?") Respondents could select up to five groups from a list of 21 and each respondent was asked to complete this exercise for both candidates.

I am using these data for a project I'm working on looking at targeting during campaigns; however, I thought that I'd share some initial results here. The chart below aggregates the responses to these questions to show how the public viewed both candidates' campaigns. The chart shows what proportion of the public that thought that Obama (on the y-axis) or McCain (on the x-axis) had focused on winning the votes of each group.

Groups in the upper left hand corner are those that a large proportion of the public thought the Obama campaign focused on targeting, but only a small share thought McCain did. Clustered far up in that corner are young adults, lower income Americans, and African Americans. Also near that top left corner are liberals. None of these groups are surprising to see in this corner, though they may be there for different reasons.

Groups in the bottom right corner are those that a large share of voters thought McCain targeted but which fewer thought Obama focused on. These groups included whites, conservatives, and upper income Americans. No surprises here either.

In the top right corner are groups that Americans thought both candidates focused on winning votes from. Interestingly, there are very few groups in this area, with middle income Americans standing mostly alone. Aside from this group, the public did not appear to identify too many groups that they thought both candidates were trying to win over.

Some other interesting findings from this chart:

More Americans thought that McCain tried to win the votes of women than Obama. It is interesting to ponder how big a role the Pallin selection was in affecting this perception. It is also worth noting that while more Americans thought that McCain was trying to win the votes of women, the gender gap strongly favored Obama in the actual voting.

Women are not the only group where the public's view of the candidates' strategies didn't quite match with the actual success of the candidate among that group. For example, Obama edged out McCain among Americans earning $150,000 or more. He also won big among those describing themselves as ideological moderates.

It is also interesting to see where Born-Again Christians fall on this chart. Despite the publicity Obama gained for targeting young evangelicals, few in the public actually credited him with trying to win over the votes of this group. However, perhaps more intriguing is the question of where this group might have fallen along the x-axis in the 2004 election. While between 20 and 30% of Americans thought McCain, it seems likely that this number is significantly lower than it would have been for Bush in '04.

Finally, there appears to be some polarization in these perceptions. In particular, note how far apart the income and racial groups are distributed on this chart. The public viewed African Americans as being almost the exclusive domain of the Obama campaign while whites were overwhelmingly viewed as being only targeted by McCain. Furthermore, the Obama campaign was viewed as being the only campaign focusing on lower income Americans while McCain was the only candidate viewed as focusing on those with higher incomes. On the other hand, both candidates were viewed as targeting middle income Americans.

These perceptions undoubtedly vary depending on whether a respondent is or isn't a part of each particular group. For example, those with higher incomes may have been more likely to think that Obama was focusing more attention on those with lower incomes compared to those who actually have lower incomes. I'll be exploring these dynamics when I analyze the data in more detail.

Overall, the chart provides some interesting insight into how the public viewed the candidates' strategies and raises an interesting question...how comparable would these perceptions be to the judgments of journalists and political pundits who follow the campaign for a living?

Understanding the Political Distinctiveness of the Cell Phone Only Public: Results from the 2006 and 2008 CCES

2009-05-12T22:27:19-04:00

A few weeks ago, I highlighted some preliminary findings from a paper written by myself and Stephen Ansolabehere for this week's AAPOR conference. The paper is now finished and you can check out a copy . The data we use for the paper is the 2006 and 2008 Cooperative Congressional Election Study.

One of the major themes in the paper is that understanding the cell-only population is about more than just age. In fact, residential mobility has a strong influence on whether someone has shed their landline. Even after controlling for age and a litany of other demographic variables, we find that respondents who moved within the last year were 24 percentage points more likely to be cell only than those who had lived in the same residence for at least five years. Renters, singles, and those without children were also much more likely to be cell-only.

Our explanation for this pattern:

"There are several reasons that highly mobile Americans may be more likely to go without landlines. First, whenever someone moves from one residence to another, they have an opportunity to reassess their phone needs. Thus, the act of moving provides an opportunity for individuals to shed their landlines. Second, mobile Americans may choose a CPO lifestyle because cell phone numbers tend to be more portable than landlines. When moving from one metropolitan area to another, individuals must change their landline phone number, but do not need to change their cell number. This may provide an incentive for choosing not to maintain a landline in a new residence. Third, those with fewer family and community ties may feel less of a need to have multiple phone lines on which they can be reached by members of their social networks. "

The fact that the cell-only public tends to be more mobile has some important political consequences. Some highlights:

The difference in the percentage of landline and cell-only respondents who reported being registered was fairly small--over 95% in both groups. However, there was a much larger gap in actual registration rates (66.8% versus 53.9%). Since cell-onlys are more likely to have moved recently, they may not have successfully registered to vote at their new addresses despite the fact that they may think they are registered.

Cell-only respondents were significantly more likely to have problems with their registration when attempting to vote. In 2008, over 7% of cell-only respondents indicated that there was a problem with their registration when they attempted to vote, compared to fewer than 4% of respondents with landlines.

Cell-only respondents were more than twice as likely as those with landlines to report that neither campaign contacted them. In short, this group is much less likely to be subjected to mobilization efforts from the campaigns.

Cell-onlys are politically distinct on a variety of measures. However, this distinctiveness is somewhat muted when demographic controls are taken into account. Interestingly, the largest differences between cell-only and landline respondents are not on issues or ideological self-placement, but on reported vote choices.

Ultimately, we argue that weighting for standard demographic measures such as age, education, income, and race may not be sufficient. Pollsters relying on landline samples may want to consider weighting by other factors such as time in residency, renter/home owner, and marital status. But check out the for a more detailed discussion of all of these points.

Pollsters Are Not The Only Ones Who Struggle to Reach Cell-Phone-Onlys

2009-04-26T22:41:20-04:00

The survey research community is focusing intently on the challenges posed by the fast-growing share of Americans who are cell-phone-onlys (CPOs). In fact, there are 40 papers being presented on the topic at the AAPOR conference next month. One of the practical issues faced by pollsters is whether the cost of reaching CPOs is worth the payoff. Last week, Scott Keeter, Mike Dimock, and Leah Christian hosted a forum at Pew during which they discussed this tradeoff. But pollsters aren't the only people who have to make cost-benefit decisions when it comes to deciding whether to attempt to contact CPOs. Campaign organizations must make the same calculation.

So how well did the campaigns do at contacting CPOs during the 2008 campaign? The chart below compares the percentage of those with landlines and cell-onlys who reported being contacted by a campaign representative in 2008. The data comes from the National Election Study (NES), which uses residential sampling and face-to-face interviews to interview both landline and CPO respondents. In the chart below, the blue bars show the percentage of each group that reported being contacted while the black lines represent 95% confidence intervals for these percentages.

The chart shows that CPOs were much less likely to be contacted by the campaigns than people with landlines. Over half of landline respondents reported being contacted compared to less than one-in-three CPOs. This sizable difference holds up even when controlling for age, income, education, partisanship, and a variety of other factors.

The next chart (below) indicates that for those CPOs who were contacted, the contact tended to come overwhelmingly from Democrats. Over 80% of CPOs who were reached by the campaigns were contacted by the Democratic side while just a little over one-third were reached by Republicans. Republicans were significantly more competitive with Democrats when it came to contacting those with landlines.

Unfortunately, the NES did not include questions asking respondents how they were contacted by the campaigns. But a subset of respondents to the 2008 Cooperative Congressional Election Study (which I've analyzed in previous posts) were asked these questions. The chart below plots the responses for those who had landlines compared to CPOs.

CPOs who were contacted by one of the campaigns were significantly less likely to have had that contact over the phone compared to those with landlines. Otherwise, there were not major differences between how landline and CPO respondents were contacted. CPOs were somewhat more likely to get an email while those with landlines were a bit more likely to receive snail mail, but neither of these differences are large. The percentage being contacted in-person or by text message were nearly identical for both groups.

Overall, the findings from these surveys suggest that shedding your landline may help you avoid those pesky campaign calls in future election years. While Democrats were a little more successful than Republicans in reaching CPOs, the cell-only crowd was almost as successful avoiding campaign volunteers as they were hiding from pollsters.

Different Approaches for Reaching CPOs, (Mostly) Similar Results

2009-04-15T17:36:10-04:00

Steve Ansolabehere and I have been working over the past few weeks on a paper we are writing for the AAPOR conference next month. Over the next couple of weeks, I'll share some of our preliminary findings here and I wanted to lead off today by presenting some comparative data we put together from three different surveys with distinct approaches to sampling.

The National Election Study, which has been around since 1948, is a labor- and time-intensive survey that uses a residential sampling approach and conducts face-to-face interviews. The Cooperative Congressional Election Study is an web-based survey conducted by YouGov/Polimetrix. The respondents opt-in to the study and Polimetrix uses a sample matching methodology where they first select a random target sample and then attempt to find a match for each respondent in their pool of opt-in respondents. Finally, Pew's survey work is fairly well known to regular Pollster.com readers. Their efforts to incorporate CPOs into their samples has relied on a dual-frame approach where they randomly select landlines and then cellular lines to build their sample.

So what happens when we compare these surveys, each of which takes a different sampling approach? Each survey I'll present data on here was conducted during the presidential campaign last year. For starters, the chart below compares each survey's estimate of the size of the cell phone only population after sample weights are applied. Note that Pew includes phone status as one of their weighting criteria, but the NES and CCES do not.

CPOs make up 19.7% of the CCES sample after weighting; 17.9% of the Pew sample and 17.3% of the NES sample are CPOs (again, after weights are applied). Thus, the differences are relatively small. Also note that the 95% confidence intervals for each estimate (represented by the darker lines) overlap.

So each of the surveys provides a similar estimate of the size of the CPO population, but what about the composition of that group? The table below compares landline and CPO respondents on a wide array of demographic and socioeconomic factors.

There is a lot of information in the table, but the different surveys are fairly similar across most measures. A few differences do stand out, however. After weighting, the CPO respondents reached by Pew and CCES were less likely to have children, less likely to be married, and less likely to be home-owners than those reached by the NES. Pew's CPOs also tended to have lower incomes and were somewhat more likely to be racial/ethnic minorities. Perhaps NES's face-to-face approach is more likely to pick up CPOs who have "settled down" relative to CCES or Pew.

Speaking of "settling down," one other item deserves attention; that is the information on residential mobility at the bottom of the table. While we weren't able to find this information in any of Pew's surveys conducted in October or November, the data were available for the CCES and NES. Not surprisingly, residential mobility is strongly related to whether one is a CPO or not. Even controlling for age, people who have moved recently are much more likely to have shed their landline and gone with just their cell. In fact, this relationship holds up when you control for all of the other demographic variables in the table. This is something I'll post more about later, but you can probably imagine that there are some significant political consequences arising from the fact that CPOs tend to move around much more frequently.

Risk and Political Preferences

2009-03-28T22:15:11-04:00

Anyone who has ever watched Deal or No Deal has noticed that some people are far more willing to take risks than others. Not only does a person's tolerance for risk affect their decisions about whether to open another suitcase on a game show, but it also influences countless daily decisions like what to eat or whether to drive over the speed limit. But what might a person's tolerance for risk have to do with their political views?

This past Fall, I was part of a team that had a module on the 2008 Cooperative Congressional Election Study. To shed some light on the role that risk might have on political preferences, we borrowed a set of questions developed by economists to gauge an individual's tolerance for risk. Essentially, the questions ask individuals the circumstances under which they would be willing to take a new (and equally good) job. The chart below shows the distribution of responses.

Not surprisingly, the results suggest that the American public is quite risk averse. Most Americans were not willing to take a new job even if the potential for increasing their income was greater than the potential income loss. In fact, over half of the respondents would not take a job even if it offered them an even chance of either doubling their income or cutting it by just 20%. These findings are similar to those of other studies that have looked at how risk tolerant (or intolerant) the public is. But what does this have to do with politics? Take a look at the charts below which show the partisan/presidential vote breakdown among each of the four levels of risk tolerance.

How a person feels about risk is related to that person's choice of party as well as their vote choice. In particular, as one becomes less tolerant of risk, they become more likely to affiliate with the Republican Party (and less likely to be a Democrat). The relationship was even stronger when it came to vote choice. The most risk tolerant respondents preferred Obama by more than a 2-to-1 margin. The most risk averse respondents went for McCain by a margin of 6%.This relationship persisted even when I controlled for other factors that tend to influence vote choice.

When you think about it, there are lots of reasons that an individual's tolerance for risk would influence their political views. It makes sense that the most risk averse Americans were less likely than others to get behind the candidate who was viewed as relatively inexperienced. (Risk averse Americans are probably more likely to vote for incumbents as well.) It also seems logical that those who are more averse to risk would be more likely to affiliate with the party that tends to be more conservative.

There are several recent academic papers on this topic (including ). Yet, despite the growing evidence showing that risk matters for how people think about politics, pollsters rarely include questions that allow us to capture respondents' feelings about risk. Is it time that they started to do so?

Dispatches: Can Pollsters Influence Policy by Determining Whose Voices Are Heard?

2009-02-19T08:43:38-05:00

This post is part of Pollster.com's week-long series on Stan Greenberg's new book, Dispatches from the War Room.

I've enjoyed reading through Greenberg's thought-provoking book over the last several days. The exercise has led me to think not just about the relationship between pollsters and their clients, but also about the role of pollsters and political consultants in our democratic system. Greenberg makes an interesting case that pollsters play an integral role in helping to link elected officials to those they govern, and his argument is convincing.

In his lead-off post on Tuesday, Mark referred to V.O. Key's famous definition of public opinion. The quote was familiar for me (and many others, I'm sure) as I use it as a point of departure for my PhD course on political behavior. Key described public opinion as "those opinions held by private citizens which governments find it prudent to heed." I've added the emphasis to the second part of the sentence because I think it is crucial. For Key, "opinions held by private citizens" do not, on their own, rise to the level of public opinion. Rather, only those opinions that capture the attention of political elites can be classified as public opinion. But which attitudes are these? Or, perhaps more to the point, whose attitudes are these?

Greenberg points out in his book that he initially gained notoriety by reporting on his study of white union members in Michigan (p. 19-20). The report caused a "storm" and led to a lot of focus in Democratic circles about how to win back the so-called "Reagan Democrats." The DLC hired Greenberg to gain more insight into how to win over this group and much of Clinton's appeal was his ability to speak to these working class whites. "Reagan Democrats" were suddenly exercising significant influence over how Democrats would campaign and, ultimately, how they would govern.

But time and resources are finite and politics is a zero-sum game. When pollsters draw politicians' attention to groups like "Reagan Democrats," other groups will necessarily get less attention and, therefore, may be less represented in government. Consider this quote from a story by Politico's Avi Zenilman just after the verdict in our most recent election:

"Unions, Hispanic groups, the Netroots, progressive organizing coalitions, single women, working women, youth, the religious left -- to name just a few -- all claim to have played a vital role in electing Barack Obama. And each says he owes them for that role."

These groups are actively lobbying for elected officials to notice their importance in the last election because they understand that being viewed in that light will lead to more influence over government. But it strikes me that pollsters may have a special role in telling their clients which of these groups have the most valid claims and should be paid special attention to.

This brings me back to how influential the role of a pollster can be during the time between elections. At the end of the book, Greenberg notes that he was always careful to limit his role to reporting on public opinion and avoided devising policy prescriptions (as Clinton himself said, "he wouldn't tell me what to do"). This suggests a relatively innocuous role for a pollster--one that is merely informational. However, if pollsters play a key role in identifying particular groups--"swing voters" or otherwise--that politicians should be particularly concerned with, then doesn't their role become far more influential? After all, the voters that politicians think they need to appeal to will likely have a major influence on the types of policies those politicians promote. By choosing to shine the spotlight on some groups, don't pollsters (and ultimately their clients) leave other groups in the dark?

So I'm interested in hearing the extent to which Greenberg thinks that his role as a pollster was influential simply by identifying some groups as "swing voters" but not others. There are so many different ways to slice and dice the public to identify the next influential group (whether it be "Reagan Democrats" or "married mums"); does Greenberg see any groups who were largely overlooked by government because they never rose to "swing voter" status? And if he was going to go back out into the country now, as he did after the 1984 election, and focus attention on a particular group of citizens, which group does he think would be most deserving of his attention?