Friday, October 02, 2009

FiveThirtyEight vs Strategic Vision

Nate Silver at 538 is suggesting that pollster Strategic Vision is making up polling results. There are some interesting posts, particularly these three: Are Oklahoma Students Really This Dumb? Or Is Strategic Vision Really This Stupid? and Strategic Vision Polls Exhibit Unusual Patterns, Possibly Indicating Fraud and Comparison Study: Unusual Patterns in Strategic Vision Polling Data Remain Unexplained.

3 comments:

Anonymous said...

This is a great example of why people don't trust statistics. I think the OK student one was the most compelling. I don't think it necessarily indicated fraud or made up results or whatever, but certainly it exhibits irregularities which could be investigated. I would guess it was a nonrepresentative sample set issue or the students just didn't take the test seriously.

The other two articles roll out the math, which should be compelling, since math doesn't lie. However, even metioning the Benford distribution seems like a total smoke screen, since, as far as I can tell, it is completely non-applicable to this case. That makes me distrustful of the entire premise.

-K

Richard said...

Let me disagree with anonymous. The I think that statistics (the ones anon doesn't trust) are the thing that shows that Stategic Visions poll results are so unlikely as to have potentially been fabricated. Thus cliche one "don;t trust statistics" isn't applicable here. I think it is better to be skeptical of polling results that don't pass statistical muster.

The Benford distribution is mentioned as a possible expectation of the distribution of the second digit in the polls, though the authors and the some commentors admit that they aren't sure that it applies. "Math doesn't lie" is another semantically null cliche that doesn't get to the issue. Math is a tool.

I think more damning is the fact that the distribution of digits for the Strategic Vision data is significantly different from the other pollsters. This takes away the argument that the distributions should look like something just that they are so different that something is different at Strategic Vision than at the other pollsters.

It might not be active deception. perhaps nonrepresentive samples or students not taking the test seriously are the reasons. Whatever the reason, the polls from Strategic Vision are useless then as indicators of anything and thus no decisions should be based on them nor any conclusions drawn. That is just as good as having no poll at all.

DKB said...

It is not statistics themselves that lie or mislead. Just like lawyers twist the law to their own (or their clients') purposes, statisticians can present the statistical perspective they wish to put forward, rather than that which a complete, accurate and honest analysis of the data would show. There is of course the possibility that the statistician in question might be stupid... "Never ascribe to malice what is adequately explained by stupidity" (pick your speaker... Napoleon, Hanlon, someone said it.) I like that explanation, since I'm a cynical bastard who thinks lots of people are stupid.