polls and surveys Stories based on public opinion polls must include basic information for an intelligent evaluation of the results. Carefully word such stories to avoid exaggerating the meaning of poll results. (See USA Today story at end)
Every story based on a poll should include answers to these questions:
1. Who did the poll and who paid for it? Start with the polling firm, media outlet or other organization that conducted the poll. Be wary of polls paid for by candidates or interest groups; their release of poll results may be done selectively and is often a campaign tactic or publicity ploy. Any reporting of such polls must highlight the poll's sponsor, so readers can be aware of potential bias from such sponsorship.
2. How many people were interviewed? How were they selected? Only a poll based on a scientific, random sample of a population – in which every member of the population has a known probability of inclusion – can be considered a valid and reliable measure of that population's opinions. Among surveys that do not meet this criterion:
• Samples drawn from panels of people who volunteer for online polls. These cannot be considered representative of larger populations because panel members are self-selected – often including "professional respondents" who sign up for numerous surveys to earn money or win prizes – and exclude people without Internet access. (Online panels recruited randomly from the entire population, with Internet access provided to those who don't already have it, are valid.)
• Balloting via Web sites, cell phone text messaging or calls to 900 numbers. These too are self-selected samples, and results are subject to manipulation via blog and e-mail campaigns and other methods. If such unscientific pseudo-polls are reported for entertainment value, they must never be portrayed as accurately reflecting public opinion and their failings must be highlighted.
3. Who was interviewed? A valid poll reflects only the opinions of the population that was sampled. A poll conducted only in urban areas of a country cannot be considered nationally representative; people in rural areas often have different opinions from those in cities. Many political polls are based on interviews with registered voters, since registration is usually required for voting. Polls may be based on "likely voters" closer to an election; if so, ask the pollster how that group was identified and what percentage of the voting population it totaled. Are there far more "likely voters" in the poll than turnout in comparable past elections would suggest?
4. How was the poll conducted – by telephone or some other way? Avoid polls in which computers conduct telephone interviews using a recorded voice. Among the problems of these surveys are that they do not randomly select respondents within a household, and they cannot exclude children from adult samples
5. When was the poll taken? Opinion can change quickly, especially in response to events.
6. What are the sampling error margins for the poll and for subgroups mentioned in the story? The polling organization should provide sampling error margins, which are expressed as "plus or minus X percentage points," not "percent." The margin varies inversely with sample size: the fewer people interviewed, the larger the sampling error. If the opinions of a subgroup – women, for example – are important to the story, the sampling error for that subgroup should be noted. (Some pollsters release survey results to the first decimal place, which implies a greater degree of precision than is possible from a sampling. Round poll results to whole numbers. However, the sampling error margin – a statistical calculation – may be reported to the first decimal place.)
Also consider the wording and order of the questions asked in the poll. Small differences in question wording can cause big differences in results, and the results for one question may be affected by preceding questions. The exact question wording need not be in every poll story unless it is crucial or controversial.
When writing and editing poll stories, here are areas for close attention:
–Do not exaggerate poll results. In particular, with pre-election polls, these are the rules for deciding when to write that the poll finds one candidate is leading another:
• If the difference between the candidates is more than twice the sampling error margin, then the poll says one candidate is leading.
• If the difference is less than the sampling error margin, the poll says that the race is close, that the candidates are "about even." (Do not use the term "statistical dead heat," which is inaccurate if there is any difference between the candidates; if the poll finds the candidates are tied, say they're tied.)
• If the difference is at least equal to the sampling error but no more than twice the sampling error, then one candidate can be said to be "apparently leading" or "slightly ahead" in the race.
–Comparisons with other polls are often newsworthy. Earlier poll results can show changes in public opinion. Be careful comparing polls from different polling organizations. Different poll techniques can cause differing results.
–Sampling error is not the only source of error in a poll, but it is one that can be quantified. Question wording and order, interviewer skill and refusal to participate by respondents randomly selected for a sample are among potential sources of error in surveys.
–No matter how good the poll, no matter how wide the margin, the poll does not say one candidate will win an election. Polls can be wrong and the voters can change their minds before they cast their ballots.
http://www.usatoday.com/printedition/news/20071019/a_eline19.art.htm
Page 12A
USAToday 10/19/07
You do math: Pols joust over new polls
Some presidential contenders on Thursday were touting the results of recent polls as showing that their candidacies are gaining momentum among voters.
•In an e-mail to supporters, Republican Sen. John McCain's campaign touted a Fox News/Opinion Dynamics Poll as proof that he's the GOP candidate "best positioned to beat Hillary Clinton." The reason: While McCain trailed Clinton 47%-44%, Republican Rudy Giuliani was behind Clinton by 1 extra percentage point: 47%-43%. Other Republicans were further back.
Left unsaid in the McCain campaign's e-mail: Because the poll's margin of error was +/-3 percentage points, that meant both he and Giuliani might be ahead of Clinton. How so? Clinton's support could be as low as 44% (because 47-3=44). McCain's could be 47% (44+3). Giuliani's might be 46% (43+3).
•Another Republican, former Arkansas governor Mike Huckabee, called attention to a Rasmussen Reports poll from Iowa showing him virtually tied for second among GOP contenders. Former Massachusetts governor Mitt Romney led with 25%. Former Tennessee senator Fred Thompson was at 19%. Huckabee was at 18%. The margin of error: +/-4 percentage points.
"It's clear my conservative message is resonating," Huckabee said in a statement touting the poll results.
