Why we should be skeptical of how the media reports polls.

Many things irritate me about mainstream media, and their inaccurate reporting of public opinion polls is rather close to the top of the list. While this article by the CBC’s online team is not bad (http://www.cbc.ca/news/canadavotes/story/2008/09/12/elxn-poll.html), there is crucial information that should be included but isn’t.

Readers are asked to compare a poll by Harris/Decima to one by EKOS without knowing which questions were asked in both surveys. Invariably, if different questions were asked, people will respond differently. If this is the case, comparing the response categories (i.e. political parties) of the questions would be inappropriate. The Harris/Decima question wording is (sort of) included, but the EKOS is not. As a result, we don’t know if the comparison that’s a significant part of the article is appropriate.

This particular article does better than some, as does this article (http://www.cbc.ca/canada/story/2008/09/07/full-poll.html). The second article is particularly interesting as it *actually* includes the margins of error for the regional breakdowns used in the analysis of the poll. These changes in regional accuracy matter: while this particular poll has a margin of error of +/- 2 percentage points nationally, the margin of error jumps to +/- 7.9 percentage points for the Prairies. Any analysis of differences, say, between parties on the Prairies that were not greater than 8 percentage points would be inaccurate. Most polls reported in the media DO NOT include these breakdowns nor does the analysis take the increased regional margins of error into account.

So when a Canada-wide poll of 1000 Canadians includes analysis of BC’s lower mainland, we should be suspicious. Similarly, most analysis of the last Quebec provincial election was suspect as the differences between all three political parties was smaller than the margin of error. Technically, then, the parties were in a statistical dead heat, but the news media reported nothing but how one party was gaining or losing compared to the others. Such coverage is often inaccurate at best and grossly misleading at worst.

The bottom line is that the media needs to be more upfront about the national and regional margins of error for any given poll they report, and they (and the polling firms that produce the polls) need to own up when the differences between parties (or issues or whatever) are not satistically significant.