Margin of Error

By Robert Niles

Margin of Error deserves better than the throw-away line it gets in the bottom of stories about polling data. Writers who don't understand margin of error, and its importance in interpreting scientific research, can easily embarrass themselves and their news organizations.

Check out the following story that moved in the summer of 1996 on a major news wire:

WASHINGTON (Reuter) - President Clinton, hit by bad publicity recently over FBI files and a derogatory book, has slipped against Bob Dole in a new poll released Monday but still maintains a 15 percentage point lead.

The CNN/USA Today/Gallup poll taken June 27-30 of 818 registered voters showed Clinton would beat his Republican challenger if the election were held now, 54 to 39 percent, with seven percent undecided. The poll had a margin of error of plus or minus four percentage points.

A similar poll June 18-19 had Clinton 57 to 38 percent over Dole.

Unfortunately for the readers of this story, it is wrong. There is no reasonable statistical basis for claiming that Clinton's lead over Dole has slipped.

Why? The margin of error. In this case, the CNN et al. poll had a four percent margin of error. That means that if you asked a question from this poll 100 times, 95 of those times the percentage of people giving a particular answer would be within 4 points of the percentage who gave that same answer in this poll.

(WARNING: Math Geek Stuff!)
Why 95 times out of 100? In reality, the margin of error is what statisticians call a confidence interval. The math behind it is much like the math behind the standard deviation. So you can think of the margin of error at the 95 percent confidence interval as being equal to two standard deviations in your polling sample. Occasionally you will see surveys with a 99 percent confidence interval, which would correspond to 3 standard deviations and a much larger margin of error.
(End of Math Geek Stuff!)

So let's look at this particular week's poll as a repeat of the previous week's (which it was). The percentage of people who say they support Clinton is within 4 points of the percentage who said they supported Clinton the previous week (54 percent this week to 57 last week). Same goes for Dole. So statistically, there is no change from the previous week's poll. Dole has made up no measurable ground on Clinton.

And reporting anything different is misleading.

Don't overlook that fact that the margin of error is a 95 percent confidence interval, either. That means that for every 20 times you repeat this poll, statistics say that one time you'll get an answer that is completely off the wall.

You might remember that just after Dole resigned from the U.S. Senate, the CNN et al. poll had Clinton's lead down to six points. Reports attributed this surge by Dole to positive public reaction to his resignation. But the next week, Dole's surge was gone.

Perhaps there never was a surge. It very well could be that that week's poll was the one in 20 where the results lie outside the margin of error. Who knows? Just remember to never place too much faith in one week's poll or survey. No matter what you are writing about, only by looking at many surveys can you get an accurate look at what is going on.