There’s more uncertainty than usual about who will turn out to vote and a candidate whose unpopularity may make his committed voters unwilling to admit their support to a pollster. And many voters are torn between their party and their misgivings about its nominee.

Share story

The Roy Moore race poses exquisite difficulties for pollsters.

If the Democrat, Doug Jones, manages to defeat Moore, the Republican, in deep-red Alabama on Tuesday, control of the Senate will be a true tossup in 2018.

But there is no consensus about whether this is likely. On Monday, a Fox News poll showed Jones up by 10 points; an Emerson College poll showed Moore up by 9 points. Overall, despite allegations of sexual misconduct with teenage girls, Moore appears to maintain a nominal lead in an average of recent public surveys.

What’s going on? The Alabama special election is forcing pollsters to confront just about every major challenge in survey research. There’s more uncertainty than usual about who will turn out to vote and a candidate whose unpopularity may make his committed voters unwilling to admit their support to a pollster. And looming over it all is a big chunk of voters torn between their party and their misgivings about their party’s nominee. No poll can really predict what they’ll do.

Of all of these issues, the pollsters have the most control over how they project the composition of the electorate. But that’s even harder here than usual.

For starters, this is a special electorate — which often has unusual turnout. The high-profile special elections over the last year provide some guidance, though.

The turnout in all of this year’s big special congressional elections — like Montana and Georgia’s 6th — approached or exceeded 2014 levels. So did the big Massachusetts special election for Senate in 2010. As a result, pollsters have a lot of reasons to think they can assume something around typical midterm turnout levels.

The bigger challenge is figuring out who will be among those 1 million to 1.4 million people expected to vote Tuesday. There’s good reason to think that the Democrats will have a better turnout than in recent midterm electorates. Democrats have enjoyed strong turnout in all of this year’s special elections, as well as in Virginia. And this race might see particularly depressed Republican turnout, since the party’s nominee is deeply unpopular. There’s no other reason for Republicans to show up than to vote for Moore.

Pollsters can’t do much about this challenge. There are two major ways that pollsters estimate the likely electorate, and neither is perfect.

One common approach is to screen voters based on whether they say they will vote. In theory, this method would have the advantage of capturing higher Democratic intention to vote. But it is not obvious whether this potential advantage is being realized here. A YouGov/CBS News poll, for instance, found that self-reported likely voters leaned heavily to Moore compared with all registered voters. A Fox News poll found the opposite.

Voters just don’t do a great job of accurately reporting whether they’re going to vote.

There’s a lot of evidence that vote history — whether someone voted in previous elections — is a better predictor than people’s self-reported turnout intention.

But the vote-history approach struggles if there isn’t a comparable past election or if there’s some other reason to think turnout patterns will be different from in the past. A 2004-based turnout model, for instance, could have underestimated black turnout in 2008 when Barack Obama was on the ticket; a 2012-based model could have overestimated black turnout in 2016.

So far in 2017, these kinds of turnout models have underestimated Democratic turnout. That’s in part because they tend to use the turnout data from the last comparable election. In this case, that’s the 2014 midterm, when Democratic turnout was relatively weak. And more generally, young and Democratic-leaning voters have a less robust track record of voting than older and Republican-leaning voters.

One solution might be to estimate turnout using the 2017 special elections, rather than the 2014 midterm in Alabama. But this would mainly increase the estimated turnout among white, Hispanic and Asian-American Democrats, not black Democrats, since black turnout has been comparatively weak so far this year. Because black voters represent around three-fourths of Alabama Democratic primary voters, such a model wouldn’t end up assuming an especially good turnout for Democrats at all. And a 2017-based turnout model would also miss one of the potentially most important dynamics of the Alabama race: low Republican turnout.

So it’s easy to see the turnout challenge for pollsters. A vote-history-based model might understate Democratic turnout. A poll that relies on self-reported vote intention would seem to have a better shot of picking up Democratic strength, but there wouldn’t necessarily be reason to be very confident in its finding.

In a low-turnout election, slightly different assumptions can yield very different results. SurveyMonkey, for example, found a 12-point gap between a typical poll that included registered voters who said they would probably vote and a poll that limited the likely electorate to those who voted in 2014 (or were ages 18 to 20 now, and thus ineligible to vote in 2014, and said they would vote). Similarly, Monmouth found an 8-point gap between a 2014 electorate and a presidential electorate.

Alone, an 8- or 12-point turnout gap could be enough to create unusual variance among the pollsters when you add typical sampling error on top. But there are other sources of variance as well.

It seems, for instance, that Jones has fared better in the live-interview telephone polls, while Moore has fared best in the automated polls without a live interviewer. The two online surveys — SurveyMonkey and YouGov — have split between Jones and Moore.

In general, there are strong reasons to prefer the live-interview telephone surveys from The Washington Post and Fox News. They contact voters with cellphones; the automated pollsters call only voters with a landline. Several automated pollsters now supplement their samples with online data, but some of the automated pollsters in Alabama do not. This is a likely source of a fairly persistent Republican lean in landline-only automated surveys since the 2010 midterm election.

Another reason to prefer the live-interview polls: They tend to be higher-quality surveys, while many of the public automated polls are surprisingly inaccurate. This isn’t true of all automated polls — SurveyUSA, for instance, has long stood out as a notable exception. But many automated polls disclose little about their methodology, while others disclose just enough to reveal serious methodological shortcomings.

A JMC Analytics poll, for instance, put voters over age 65 at 52 percent of the Alabama electorate. This is not plausible; midterm turnout levels would put voters over age 65 in the 30s, and even the August primary was notably younger. Then, just 42 percent of voters were over age 65, according to data from L2, a nonpartisan voter file vendor. An Emerson poll put black voters at 17 percent of the electorate, but a reasonable midterm electorate would be no lower than 23 percent for black voters, based on the L2 data.

Yes, there is a lot of uncertainty in the composition of the electorate, but these figures are not in the ballpark. And if a poll can’t be trusted to be in the ballpark on something as straightforward as age or race in Alabama, it is not clear why it should be trusted on anything else.

Not all the automated polls are so egregious. And in this election, those pollsters have a strong argument in their favor: the possibility that voters might not divulge their support for Moore to a live interviewer, while they might be more likely to give their truthful answer by pressing the number 2 at the invitation of a robot. This type of social desirability bias does seem eminently possible in this election.

Put it all together, and suddenly the huge range in the polls starts to make some sense. On the one extreme, you can imagine an automated poll that under-sampled and under-weighted young and nonwhite voters, with a Republican-friendly, 2014-like electorate. On the other extreme, you can envision a live-interview poll with a highly favorable turnout for Jones that might underrepresent Moore’s support because of social desirability bias.

And this huge range of polling results still doesn’t capture the range of what might happen in the voting booth. In this deeply Republican state, the GOP candidate is assured of a hefty share of votes regardless of controversy. But there is still a big chunk of Republican-leaning voters and poll respondents who face a hard choice about whether to vote, and, if so, whether to vote for Moore or Jones. If they break one way, we might be in for a surprising result. And, for that same reason, it is entirely possible that a pollster that appears to “get it right” might have only stumbled onto it by accident.