Election Update: Leave The LA Times Poll Alone! Instead

Quote:

Election Update: Leave The LA Times Poll Alone!
Instead of arguing about it or ignoring it, adjust for it.

Aug 23, 2016 at 1:54 PM

By Nate Silver

I’m tired of hearing about the USC Dornsife/Los Angeles Times tracking poll.

I’m tired of hearing about the poll from Donald Trump fans such as Reince Priebus, Matt Drudge and Donald Trump himself.1

They frequently cherry-pick that poll because it consistently shows much better results for Trump than the other surveys. As of Tuesday morning, for example, the poll showed the race as virtually tied — Hillary Clinton 44.2 percent, Trump 44.0 percent — even when the national poll average has Clinton up by about 6 percentage points instead.

This has been a fairly consistent difference between this poll and most others. Take the LA Times poll, add 6 points to Clinton, and you usually wind up with something close to the FiveThirtyEight or RealClearPolitics national polling average.

What’s the source of the LA Times poll’s Trump lean? There are good “explainers” from The New York Times’s Nate Cohn and Huffington Post Pollster’s David Rothschild. Long story short: The poll’s results are weighted based on how people said they voted in 2012. That’s probably a mistake, because people often misstate or misremember their vote from previous elections.2

The poll does some other things differently also, some of which I like. For instance, it allows people to assign themselves a probability of voting for either candidate instead of saying they’re 100 percent sure. And the poll surveys the same panel of roughly 3,000 people over and over instead of recruiting new respondents.

That creates a more stable baseline and can therefore be a good way to detect trends in voter preferences, although it also means that if the panel happened to be more Trump-leaning or Clinton-leaning than the population as a whole, you’d be stuck with it for the rest of the year.

But I’m also tired of hearing from the LA Times poll’s critics. I’m not a fan of litigating individual polls, for several reasons. First, in my experience, these critiques tend to involve their own form of cherry-picking. Clinton fans will pick apart the LA Times poll and find a few things wanting — in this case, with good reason (in my opinion).

But they’ll give a free pass to a poll like this one that shows Clinton ahead by 16 percentage points in Virginia, even though it’s also something of an outlier.

You can almost always find something “wrong” with a poll you don’t like, even if you might have approved of its methodology before you saw its result.

It’s probably also harmful for the profession as a whole when poll-watchers are constantly trying to browbeat “outlier” polls into submission. That can encourage herding — pollsters rallying around a narrow consensus to avoid sticking out — which is bad news, since herding reduces the benefit of averaging polls and makes them less accurate overall.

Furthermore, the trend from LA Times poll still provides useful information, even if the level is off. Before the conventions, the poll had Trump ahead by an average of 2 or 3 percentage points. Trump then got a modest convention bounce in the poll and pulled ahead by 6 or 7 percentage points.

But Clinton got a bigger bounce, and she’s been ahead by an average of 1 or 2 percentage points in the poll since the conventions, although it’s been a bit less than that recently, with Trump narrowly leading the poll at times. All of this follows the trend from other polls almost perfectly, as long as you remember that you have to shift things to Clinton by about 6 points.

And that’s pretty much what FiveThirtyEight’s forecast models do through their house effects adjustment. A pollster’s house effect is a persistent lean toward one candidate or another, relative to other polls.

House effects are not the same thing as statistical bias — how the poll compares against actual results — which can be assessed only after the fact. Nor do they necessarily indicate partisan bias. For example, Public Policy Polling, a Democratic polling firm, has a very mild pro-Trump house effect this year.

Calculating house effects is simple, in principle — you compare a poll’s results against the average of other surveys of the same states (treating national polls as their own “state”). In practice, there are a few challenges, which you can read more about in our methodology primer. One of the important ones is defining what the average is. In the case of FiveThirtyEight’s forecasts, the average is weighted based on our pollster ratings.

Put another way, the house effects adjustment seeks to determine what the best pollsters are saying and not just what the most prolific ones are saying. In 2012, that made a difference: the higher-quality pollsters generally projected better results for Obama than the lower-quality ones. This year, any such effects are very minor,3

and neither Trump nor Clinton benefits much from the house effects adjustment overall, although it can matter more in individual states. Polls in Nevada happen to be a Trump-leaning bunch, for instance, so the house effects adjustment slightly helps Clinton there.

Which polls have a big house effect?

In the midst of an election, I’m sometimes reluctant to fixate on the house effects for individual polling firms because I don’t necessarily want to imply that a poll with a strong house effect is wrong.

A house effect is sometimes the sign of a problem and sometimes not; it’s hard to know for sure until after the election has taken place. I also don’t want to encourage herding. Instead, I’d rather pollsters stick with what they’re doing, even if they stand out a bit, than to change methodology in midstream, as at least one pollster (Ipsos/Reuters, which previously had a Clinton-leaning house effect) has already done.

Nonetheless, we talk about polls being Clinton-leaning or Trump-leaning all the time — so here’s some more detail about what that means.

In the table below, you’ll see the house effects for all firms that have conducted at least 5 national polls this year or conducted surveys in at least 5 states.

A couple of technical points: First, although it’s not shown in the table, our models calculate house effects for Clinton and Trump (and Gary Johnson) separately. A poll could be deemed to have both a pro-Clinton and a pro-Trump house effect if it tended to show few undecided voters, for instance. The numbers in the table are net figures.

Also, you’ll see the house effects presented in two ways: as a raw figure and a discounted one. The raw figure reflects the magnitude of the house effect so far, while the discounted one is essentially what the model predicts the house effect will be going forward. The less data we have from a given firm, the more the raw house effect is discounted, since it may reflect statistical noise rather than anything systemic.

Here’s the data with pollsters sorted into three major groupings based on their methodology: internet polls, automated polls (robopolls) and traditional live-caller telephone polls:

http://fivethirtyeight.com/features/election-...oll-alone/



As you can see, the LA Times poll has the strongest house effect of any major pollster: a raw house effect of about 6 points in Trump’s direction, or a discounted one of about 4 points.

Other Internet-based polls have been a mixed bag. The UPI/CVoter tracking poll has also been Trump-leaning. Ipsos/Reuters formerly had a strong Clinton-leaning house effect but, after a methodology change, it has pretty much gone away.5

Other prolific online polling firms, such as Morning Consult, YouGov and SurveyMonkey, don’t have strong house effects.

All the major automated polling firms have Trump-leaning house effects, ranging from moderate to severe, especially in the case of Rasmussen Reports and Gravis Marketing, which have longstanding GOP-leaning house effects.

You might also notice that the various daily and weekly tracking polls, which are either online or automated polls, are mostly a Trump-leaning bunch.

We haven’t had a lot of national polls lately other than the tracking polls, so that’s one reason our national polling average and others that adjust for house effects show a slightly wider margin for Clinton right now than those that don’t.

By contrast, traditional landline telephone polls have been Clinton-leaning as a group, although not uniformly. Quinnipiac University polls had a strong Trump lean earlier in the cycle, for example, although it has dissipated recently.

It’s worth keeping these patterns in mind when you evaluate new surveys. Accounting for house effects, our model thinks a Quinnipiac poll showing Clinton up 8 in Colorado is roughly equivalent to a Marist College poll showing her up 12 there, since Marist’s polls have been Clinton-leaning while Quinnipiac’s have been Trump-leaning.

However, the spread between traditional telephone polls and online and automated polls has been larger recently, with traditional polls generally showing a larger bounce for Clinton. This difference in methodology may explain some of the seeming difference between state polls and national polls.

For whatever reason, online and automated polls have mostly concentrated on national surveys this year, while most swing states have at least a couple of recent, high-quality traditional telephone surveys. Since the conventions, Clinton has done better in state polls (which are in line with a national lead of 7 or 8 percentage points) than in national polls (which show a lead of more like 5 to 6 points).

In essence, that spread between national and state polls may reflect a sort of house effect that the model is not fully adjusting for.

Overall, Clinton has an 85 percent chance of winning the elections according to our polls-only forecast and a 76 percent chance according to polls-plus. Neither figure has meaningfully changed over the past couple of days.

Looking at the polls as a whole — and adjusting for house effects — Trump seems to have gained 1 or 2 percentage points from his post-convention lows, but probably not more than that yet.

Individual polls might give you a different impression, of course — and that’s OK. This is an unusual presidential election and a somewhat challenging time for the polling industry as a whole; we should expect and encourage a bit of disagreement.

If you’re going to browbeat a pollster, do it to a pollster who is doing things cheaply — some of the robopolls qualify — and not one that’s trying to move the ball forward, like the LA Times poll. Besides, every now and then, one of the “outlier” polls proves to be right. But if you want to play the percentages and get the best gauge of where the election is headed, take the average, adjust for house effects if you like, and relax.