Polling and you, a guide #2

Messing up a poll

The first, and least common method of queering a poll's results is in the question. For example a questions such as "Do you support the war criminal John McCain who bombed civilians in Vietnam or the civil rights hero Barack Obama who wants to give you $500 if he's elected?" would tend to produce a result favorable to Obama. This is an extreme question, however there are more subtle hints such as "Do you favor Senator Obama or John McCain?" Just leaving the honorific off of McCain's name will tend to help Obama in a poll. This is the oldest, and crudest method of manufacturing favorable data and is frankly out of date. Pollsters today don't rely on such brute force methods of massaging data... instead they massage samples, as discussed above, by arguing who is a "likely voter" and who is not.

It gets complicated....

As if it wasn't already tough enough to follow. Unfortunately, polls have more uses than predicting the winner of an election. The most common is fund raising. People prefer to give money to a candidate who is winning and will have influence. Those who are seeking influence though their dollars want to spend their money on somebody who will actually have power. Giving money to a candidate who then loses negates any possible influence. This phenomena is why so many big interest groups give money to both candidates... just so that they can say in a lobby somewhere that they supported whoever actually gets elected. Polls fit into this because the campaign can cite them when soliciting money. Imagine a call to the CEO of a industrial farming interest that says "look, this polls says that I'm beating the other guy by 20 points, there's no reason to give money to him and ever reason to give it to me."

Obviously, this second use for polls encourages corruption. Because polls are subject to the whims and judgment of the pollster it is very easy to twist the numbers based on "assumptions" that have their motives in producing a result to raise money rather than an accurate picture of the electorate. The polling organizations are also subject to lobbying by the candidate and simple rooting interests. If a pollster has a biased view of the electorate that is in the gray area where numbers have to be replaced with judgment (for example how many new voters aged 18-20 will actually cast their ballots) the final numbers can be wildly different from another pollster who uses the opposite assumptions. This is why the NYT/CBS poll has Obama up 14 points while Gallop had him up only 3 points last week. The math would suggest that there is something greater than 1 in a million chance that those two numbers could be generated from surveying the same electorate... yet because of all the adjustments that these polls use they end up with those very different results from the same population.

So, if you're trying to use polls to figure out who's winning you have to decide who you trust.

First, throw out most any news organization that runs its own poll. Most news outlets have their biases, are inexperienced (thus tend to take "advice" from campaigns) and are interested in creating news (so will build polls that end up with extreme results). Two examples of news organizations that I suspect of trying to "make news" are the NYT poll cited earlier that had Obama up by 14 and the AP poll last week that had Obama up only 1 point. I frankly think they're both worth throwing out.

Trust the professionals in general. And of the professionals you want to watch those polls that advertise and take pride in getting as close as possible to actual outcomes of elections. These firms make their money by getting it right, not by promoting one message or another. Often campaigns will employ two polling organizations. A public and an internal. The public poll is, frankly, an arm of the media campaign that also runs commercials and raises money by using TV, Internet and radio. It's job is to provide "facts" that promote the candidate. The internal pollster is often a tightly kept secret that is charged with providing an accurate picture of the election to the campaign strategists. Firms like Rasmussen, Zogby and Gallop are often subcontracted as internal pollsters. They never disclose which campaigns they're working for, but that is how they make most of their money. Their public polls are advertising for their internal polling contracts. In this way they have incentives to try to get it just right, regardless of who'd they prefer to see win.

Even these ostensibly more accurate polls can be wildly wrong. Zogby is the most notorious for having such a complicated polling procedure that he mathematically prejudices his results. Basically, he makes so many adjustments, tweaks and compensations (such as those discussed in the post below) that his numbers bear very little resemblance to the original survey data. Remember, every one of these little twists of the numbers might get the reported numbers closer to reality... but they also increase the chance that the poll will be really far away from reality. The danger of tweaking the numbers cuts both ways. So, the famed "margin of error" which is usually reported using the chi-squared of the sample as if it wasn't hand picked, tweaked, and otherwise molested, is in reality often far greater than 3 or 4 points. Mind blindingly this is a mathematical reality without a real utility. Because pollsters who are good at their jobs will often put the right tweaks in, choose the correct samples, and massage the numbers the appropriate direction they will report numbers that can be well within what one would expect with a normal chi-squared analysis simply by making good educated guesses. So, more error, but more accurate poll. Screwy right?

Finally, something to remember about all these folks who love to tell you the electoral vote count based on state polls. Ignore them! At least for the most part. Those state polls are rarely done by the sort of professional organizations needed to get a good idea about where voters will actually come down. If the margin is less than 7 or 8 points you can consider it a toss up. Also, those polls are usually taken over the course of a week or so. Because of this, the conditions of the campaign will have changed from the beginning of the poll to the end. This really messes up the results. Finally, state polls are not taken every day. Because events can drive polls one way, and then the other in the course of a day or two they usually aren't capturing much data that is actually interesting. It's best to just watch the national numbers, and then apply it to geographical regions that are "swing" within the states to get a guess at the EC. Also remember, only twice has the EC gone a different way than the popular vote. Barring a once every 100 years or so occurrence those state counts really don't tell you anything more than the national polls, and the national polls tend to be more professionally done and timely.

Ok, some truths to remember about polls... all polls

-They aren't true, they're a reflection of a truth a the time.
-They are all subjective educated guesses based on what the pollster believes not hard facts
-Combining polls does not make for a more accurate poll
-Not all polls are designed to truthfully describe the electorate be a cynic - if it looks to good to be true it probably isn't
-Trust the professionals who have economic reasons to be accurate. Doing a good poll is very hard, of the big firms only Rassmussen, Zogby, Gallop and Investors Business Daily were within 2 points of both the 2000 and 2004 election.
-Do not trust universities, news outlets, or small state-wide firms. These folks will often have undiagnosed biases, ulterior motives, or too little experience to come up with a good result. The University of Chicago and Pew are exceptions to this general rule... even though Pew had a really screwy poll recently.

Last thought:
Be cynical! Polls are not votes, don't expect anything and don't be upset if polls end up wrong. The one, best and only important poll is on November 4th, it's got such a large sample size that the margin of error is so small it can be dismissed.