Never trust a poll, and never trust a pollster. Just like one should never trust one experimental result.

About the best one can hope for is to try to collect all the available polls together, and make a poll from the polls, in hopes of getting an average that has some tiny thing to do with reality.

This is a good page to read, to remind oneself that thes things we see on the media called "polls" are artifacts whose provenance and political spin can never be trusted.

2010 Top Ten "Dubious Polling" Awards
www.stinkyjournalism.org/latest-journalism-news-updates-169.php

“MANUFACTURED OUT OF WHOLE CLOTH” Award

       

WINNERS: Zogby, Ipsos/McClatchy, ABC/Washington Post, Associated Press/Stanford University, Pew Research Center and CNN/Opinion Research Corporation polls, for their valiant efforts to manufacture a “public opinion” on “cap and trade” legislation, though most people haven’t a clue.

BACKGROUND: Recent poll suggest that so few people are aware of what “cap and trade” programs are, it is pure fantasy to think of a “public opinion” on the issue. Yet, somehow, many pollsters have been able to fabricate the illusion of a public that is so highly engaged and informed, the vast majority of people have a position with respect to the issue.

But let the people guess what “cap and trade” means among three possible answers - healthcare, banking reform, or the energy and environment - and less than one in four can accurately point to the latter. Or ask them how much they’ve heard about “a policy being considered by the president and Congress called ‘cap and trade’ that would set limits on carbon dioxide emissions,” and only one in seven can say “a lot,” while more than half say “nothing at all.”

So, how do pollsters create the illusion of an engaged public? Simple: They feed their respondents information, and then immediately ask what respondents think, using “forced-choice” questions to get an answer.

The key point here is that none of these polls can any longer represent the American people! Once pollsters give information to their respondents, their samples no longer represent the larger population, which has not been fed that same information.

Of course, different polling organizations feed different information to their respondents, which is how the polls run the gamut of opinion. At one end of the spectrum is a Zogby poll (conducted for a global warming skeptic), showing almost a two-to-one opposition to cap and trade legislation, while at the other end of the spectrum is a CNN poll finding about a two-to-one level of support. Other polls fall somewhere in-between. So, fellow pollsters, fabricate away! Just stop pretending you report what the American public is thinking. You know better. We (should) know better, too.

“STONEWALLING/COVERUP” Award

       

WINNERS: The John Hopkins Bloomberg School of Public Health and one of its professors, Dr. Gilbert Burnham, for stonewalling in the face of serious questions about a flawed survey project, which reported more than 600,000 Iraqi deaths from 2003 to 2006. The head researcher was formally censured by the American Association for Public Opinion Research (AAPOR) for covering up his data collection efforts, but the Bloomberg School refuses to investigate the methodology. (Ah, the wisdom of the three monkeys: “See no evil, hear no evil, speak no evil!”).

BACKGROUND: In 2006, the British medical journal, The Lancet, published the results of a survey, designed and supervised by Dr. Gilbert Burnham of the John Hopkins Bloomberg School of Public Health, and his colleagues.* The survey purported to show that about 600,000 Iraqi deaths occurred in Iraq by July 2006, as a consequence of the invasion of Iraq.

A lot of people were against the war, but jacking up the body count with bad studies is not a good tactic for anyone. According to economics professor Michael Spagat of Royal Holloway College, these results were anywhere from seven to 14 times as high as other credible estimates, including those made by the non-partisan Iraq Body Count, a consortium of U.S. and U.K researchers, also concerned about the human toll of the war.

Such large differences in estimates led other researchers to question the methodology of the study. But contrary to scientific norms, Burnham refused to provide details about how the survey was conducted. When a complaint was lodged with AAPOR, its standards committee also tried to obtain such details, but was rebuffed. That led to the censure.

What exactly were John Hopkins Bloomberg School, and Burnham, et. al., hiding? AAPOR asked for the kind of information that any scientist doing this type of work should release: a copy of the questionnaire, the consent statement that interviewees have to see, a full description of the selection process, a summary of the disposition of all sample cases, and how the mortality rate was calculated.

John Hopkins Bloomberg School initially stood behind the study, but then eventually concluded that Burnham had made some unauthorized changes in his methodology, and thus “the School has suspended Dr. Burnham’s privileges to serve as a principal investigator on projects involving human subjects research.”

But the Bloomberg School has not come clean with the problems of the research project. Their press release admitted that their internal review “did not evaluate aspects of the sampling methodology or statistical approach of the study.” Instead, Bloomberg asserts, “It is expected that the scientific community will continue to debate the best methods for estimating excess mortality in conflict situations in appropriate academic forums.”

Let’s see: The Bloomberg School will not attempt to evaluate what experts believe is almost certainly a faulty methodology, saying the scientific community should make the evaluation. But then the school advises Burnham not to release details about his methods, so the scientific community can’t have the information it needs for a definitive assessment.

Sounds like a cop-out and a Catch 22, all rolled into one!

And we thought Richard Nixon was tricky.

* Burnham G, Lafta R, Doocy S, Roberts L. 2006a. ‘Mortality after the 2003 invasion of
Iraq: a cross-sectional cluster sample survey’. The Lancet 368:1421-1428. It can be accessed online at http://brusselstribunal.org/pdf/lancet111006.pdf.