With the Iowa Republican caucus only 1 week away, pollsters will poll the state like nobody's business. Some polls will have some candidates higher than others, and some polls will ultimately be more accurate than others. So how do we know which polls we should trust?
Most long-time Iowa watchers will tell you to watch the well known Selzer and Co. Des Moines Register poll. The poll's most recent triumphs include correctly nailing an easy victory for both Barack Obama and Mike Huckabee in 2008. In fact, according to some Washington insiders, it's the only poll to watch.
I disagree. Selzer's polling deserves its praise, but nobody's perfect.
Just look back to the 2010 Iowa Republican gubernatorial primary. Midterm primaries are obviously different than presidential caucuses (though remember Iowa Republicans use a straw-poll procedure that is much closer to a primary than what most think of a caucus). Still, this was a very recent matchup involving mostly Republican voters in Iowa.
In this contest, Selzer's final poll had Terry Branstad winning by 28%, while automated and frequent Iowa pollster Public Policy Polling (PPP) had Branstad winning by 15%. Branstand's final margin was only 9.4%.
PPP was clearly the more accurate pollster in predicting the final margin.
Now, this isn't to say that PPP will end being more accurate in the 2012 Iowa Republican caucus. I am merely saying that no pollster is the one and only.
While there is such a thing as really bad pollsters (Zogby Interactive, I'm looking at you), we really cannot rank most pollsters from best to worst. Mark Blumenthal first demonstrated this point in 2010, Dick Bennett has continuously showed it, and my own work looking back at the 2010 midterm elections confirms Blumenthal and Bennett's finding. What about the argument that PPP and other automated pollsters may be missing cell-phone only voters? There is some evidence that there is a difference in the political opinions of cell-phone only users vs. landline users, and Iowa has a relatively high percentage of adults only reachable by cell-phone.
The real question we should be asking though is how will this affect polls for the 2012 Republican Iowa Caucus.
In 2010, automated polls were on average as accurate as live interview polls. In 2011, we saw automated pollsters accurately predict difficult to poll House special elections in both New York's 9th and 26th districts. And let's not forget, PPP did better than live-interviewer Sezler and Co. in the 2010 Iowa Republican primary.
The evidence from recent history seems rather thin that automated polls will miss something that live-interviewers will capture just because automated polls are automated.
But will 2012 be different?
Earlier this month, Marist polled Iowa and found the following:
You can look at this data in a few ways. First, Ron Paul did better in this polling sample thanks to cell-phones being included*. Second, cell-phones did not greatly impact the poll's findings. In fact, the variation between candidates' percentages in landline only (as automated phones have) and overall voting population was statistically insignificant.
But let's take a look at some more data.
During the 11/27-12/6 window in which the Marist poll was taken, a lot of pollsters were in the field. The race seemed pretty static during this period with Newt Gingrich with a solid lead. We had a good mix of both live-interviewers who called cell-phones and automated pollsters who did not. Did a similar Ron Paul effect happen within this group as in the Marist poll?
Somewhat to my surprise, it did! On average live-interviewers who called cell-phones found Ron Paul's support to be at 17.2%, while automated polls found it to be at only 14%. This is the same exact 3% gap Marist found.
Here's the kicker stat though. PPP was the only automated pollster during this period that did not exhibit the Ron Paul cell-phone gap. At 18%, Ron Paul's PPP percentage of the vote was tied for the highest any pollster recorded Ron Paul at during this period.
In recent weeks, PPP has continued to show Ron Paul flying high. Their polls have shown Paul at a greater percentage than most other pollsters (e.g. automated pollsters Rasmussen and We Ask America).
I think PPP may have something. The lack of cell-phone voters seems to be affecting most automated samples, but PPP somehow seems immune.
When we begin to see more live-interview polls goes into the field later this week, it's probable most of them will find the same Ron Paul lead in Iowa that PPP has. The question of whether that lead holds and which pollster is ultimately the most accurate is still one to be answered, but I think we will have a pretty good idea of the final result going into the caucus.
*Something I know comes to no surprise to his supporters who CONSTANTLY argue this point.
******CORRECTION: CNN's state polls are landline, but do not include cell-phones. Only their national polls include cell-phone only respondents. The recalculated average for the third chart is 17.3% for Paul support among landline with cell-phone polls. In other words, the conclusion is the same.
It has become popular to say that the Iowa caucus is where pollsters' dreams go to die. Experts from Mark Blumenthal
to Nate Silver
have noted that Iowa caucus polling accuracy is less than ideal (and certainly less than in other contests)... But I dare say that our inability to predict the caucuses is somewhat overblown?
First, let's remember that the Republican Iowa caucus is not actually a caucus. It's a straw poll
. The difference? In a caucus (as the Democrats have in Iowa), voters have to gather for hours on end openly declaring their support for a candidate.
In a straw poll, voters merely have to listen to a few speeches and then cast their votes on secret ballots. This makes the act of voting far easier and invites less peer pressure to vote for a specific candidate (i.e. having a voter change their mind to fit what their neighbors are doing).
Another distinction is that in the Democratic caucus voting thresholds exist. Candidates who do not meet this threshold (15%) have their voters reallocate their votes to other candidates. On the Republican side, voters vote once and that's it. No thresholds.
Both of these facts make polling a Republican contest easier in theory than a Democratic contest.
What about the rather convincing charts
indicating relatively high pollster error for past Republican contests in Iowa?
The truth is that we are dealing with a lot of old data. Note that with the exception of 2008, all the Iowa Republican caucus polling data is from 2000 and before. Why is that important?
It turns that polling data has gotten incredibly more accurate
since 2000. For instance, you may remember that the final polls
in 2000 incorrectly indicated that George W. Bush would win a small but solid victory in the popular vote over Al Gore. What you may not remember is that despite being relatively inaccurate, polling in 2000 was better
than in all but two presidential elections since 1960.
Polling in 2004
, on the other hand, came with decimal points of correctly projecting the presidential popular vote. Polling for state contests has similarly
become consistently more accurate in the 2000's.
Despite all the problems with new untested pollsters with supposedly unreliable techniques, polling in 2010 was on-par
with other midterm elections since 2002. In 2011, we had automated pollsters (who are missing the cellphone population and have terrible response rates) correctly forecasting upsets in special Congressional elections (see New York 9
Fitting with this pattern, the 2008 Republican Iowa caucus was more accurately polled than any before it. The average difference between the final polling average and candidates' actual percentage was only 2.7%. The biggest difference in percentage, 5%, was far less than the biggest difference in previous years.
If the trends of accuracy continue, polling in 2012 in both the Iowa caucus and other contests is more likely to be like 2008 than any other year.
None of this is to say that we will not see a "surprise" in Iowa. Considering how tightly the candidates are clustered, a 5% error between the polls and actual margin could make all the difference in the final outcome and, more importantly, media spin. That said, the polls are more likely to get a lot more right than wrong.