Questioning overnight Sarah Palin polls

By Mark Blumenthal

updated 11:17 a.m. ET Sept. 1, 2008

Mark Blumenthal • E-mail

WASHINGTON - What do those "instant response" polls tells us about what voters think about John McCain 's selection of Sarah Palin as his running mate? It may depend on which poll we look at.

The Gallup Organization called 898 registered voters Friday night and found that most voters had "never heard" of Palin (51 percent) or could not offer an opinion (20 percent). The few that had heard of Palin were mostly positive: 22 percent rated her favorably, 7 percent unfavorably.

On the same night, Rasmussen Reports called 1,000 "likely voters" using its automated methodology and obtained a very different result. Four of five voters were able to offer an opinion: 53 percent rated Palin favorably, 28 percent rated her unfavorably, only 18 percent said they were unfamiliar, and 2 percent were unsure.

So, as of Friday night, the percentage of Americans who knew Palin well enough to rate her was either 29 or 71 percent.

Right.

Why the big difference?

First, let's consider the text of the questions. Gallup asked respondents if they had a "favorable or unfavorable opinion of Sarah Palin, or if you have never heard of her."

Rasmussen posed a question with four categories - very favorable, somewhat favorable, somewhat unfavorable and very unfavorable - but they also prompted voters to say if were "not familiar with Sarah Palin."

Both offered an explicit "no opinion" option, although the two "somewhat" choices on the Rasmussen question may have been more tempting for those inclined to offer a soft opinion.

The order of the questions may have been a bigger issue. The favorable rating was the very first question asked by Gallup. Rasmussen, oddly, asked it last and after informing respondents that McCain had selected Palin as a running-mate and then posing other questions about her. Perhaps respondents on the Rasmussen survey were more likely to make inferences about her knowing that she had been selected as the vice presidential nominee.

Video   Is Palin ready for VP role? Aug. 31: McCain supporter and Minnesota Governor Tim Pawlenty (R-MN) discusses the selection of Alaska Gov. Sarah Palin as John McCain’s running mate with NBC’s Tom Brokaw on “Meet the Press.” Meet the Press

Of course, 72 percent of respondents were willing to offer an opinion on the very first question on the Rasmussen survey, whether Palin was the "right choice for McCain to make." Even though some may have been giving McCain the benefit of the doubt, it is still hard to square that result with the 71 percent on the Gallup survey who had never heard of Palin or had no opinion of her.

The Gallup survey interviewed registered voters, while the Rasmussen survey screened slightly further to interview "likely" voters. Could that explain the difference? While the nonlikely voters are probably less attentive to the news, the numbers are simply not big enough to explain the gap between the two polls.

Finally, let's consider the challenge of fielding a survey in just one night. Most pollsters attempt to dial unavailable households at least three or four times over successive days, a technique that is obviously impossible in a single evening.

Thus, as the National Council of Public Polls warns, the response rates for one night surveys "are generally much lower" than for more conventional surveys. As such, and given the possibility that those at home might be different, one night polls "are much more likely to have substantial biases than polls with multiple callbacks over several days."

Is it possible that Rasmussen's automated methodology obtained a significantly lower response rate than the live interviewers used by Gallup? Since neither organization disclosed its response rate, we have no way to know for certain, but live interviewers are usually more adept at securing cooperation than recorded messages. Advantage Gallup on this score.

In any event, the wide differences between these two polls ought to cast some doubt on the wisdom of these instant-reaction polls. Better to set aside the one-night wonders and wait for more reliable data that will be available once the dust of the Republican convention clears.

McCain concedes, urges unity with Obama McCain joins Fey to joke on 'SNL' John McCain still has a shot Subscribe to McCain headlines -

Obama elected 44th president Obama campaign head gets big book deal Tears of joy, dancing in streets over Obama win Subscribe to Obama headlines -

MORE FROM NATIONAL JOURNAL

National Journal Section Front

Walter: GOP can break negative cycle Mercurio: Gunning for a new RNC Cook: How will Democrats govern? Brownstein: The GOP's dangerous imbalance Blumenthal: Pollsters pretty accurate National Journal Section Front