Sunday, November 21, 2004
Exit polls: The liar's problem
Thou hast committed—
Fornication: but that was in another country,
And besides, the wench is dead.
Yes, I toyed with the wench of statistics in my youth. But it was long ago and far away, and the wench might as well be dead. And thus have I avoided the exit-poll controversy.
But there is an interesting critique by Mark Blumenthal of the now-famous paper [a link to the PDF may be found here] by Stephen Freeman of M.I.T. The conclusion that has generated the most heat is "that the odds against unusual 'anomalies' in just three states -- Florida, Pennsylvania and Ohio -- 'are 250 million to one.'
Blumenthal properly finds fault with some of Freeman's assumptions, which Freeman acknowledges in a comment. [Read down for Freeman's response.] But while the probability estimate is agreed to be erroneous, the anomaly by no means goes away. Blumenthal writes,
... the observed discrepancies from the actual count in Freeman's data still appear to be statistically significant using the Merkle & Edelman margins of error in Ohio, Florida and Pennsylvania. If NEP were to provide the actual "p-values" (probability of an error) for all three states, and we multiplied them as Freeman did, the real odds that this happened by chance alone are still probably at least 1,000,000 to 1.
Assuming the correctness of that calculation requires—politically speaking—some kind of explanation.
Statistical explanation appears in the form of an "hypothesis," which is then accepted or rejected, based on some set of data, with a probability assigned that the rejection of the hypothesis is correct. One such hypothesis might be that the occurrence of the anomalous data is not due to fraud in the vote count. But there are other hypotheses being floated about, and it is not my point to write about any of these.
I will say, however, that Blumenthal's conclusion that
to continue to see evidence of vote fraud in the "unexplained exit poll discrepancy" is more than wishful. It borders on delusional.
borders on the delusional itself.
The simple truth is that the factors accounting for the discrepancies between the exit polls and the reported vote are not known (at least to the general public or to academics). An open mind is what is required here.
There is a more fundamental problem, however. There are essentially two groups of researchers—(1) those who assume that the vote count is accurate and then study the exit polling data to determine what systematic errors were made in the polling and (2) those who assume that the exit polls are correct and then search for the cause of the systematic errors in the vote count.
The possibilities are these—
- The polls are correct, within their margins of error, and the anomalies reside in the vote count.
- The vote count is correct, within certain narrow limits, and the anomalies reside in the sampling methods and/or statistical assumptions that are made to correct the raw data.
- Neither the vote count nor the exit polls are correct, within reasonable limits.
If you have no way of deciding among these possibilities, the first thing you do not do is come to a conclusion. You must realize that you don't know.
There is one conclusion, however, that is safe to draw. If the vote count is accurate, the inferences drawn from the exit polls are sufficiently imprecise to be useless for predictive purposes, and you have to wonder why the networks are shelling out so much money to create them. For a lot less money they could pay me to appear, appropriately clad in mystic garb, with my tea leaves. The public would be equally well misinformed, and more to the point—I would leave the studio a great deal richer and happier.