UPDATED: Dubious Poll Reporting Achievement of the Week Award: Sun Media/Leger Marketing

by kadyomalley on Monday, September 28, 2009 8:19am - 14 Comments

Honestly, ITQ isn’t trying to pick on Sun Media or Leger Marketing here, but — well, actually, she sort of is, but only because this story — on support for the Afghanistan mission —  is such a perfect example of how media outlets can unwittingly (at least, she assumes it’s unwitting) render the findings of even a fairly stark and straightforward poll utterly meaningless:

Almost half of Canadians say our troops should remain in Afghanistan, but only if the mission changes from a combat role to a training and development mission.

A Leger Marketing poll says 45% of Canadians support staying for a non-combat mission, while 12% want the troops to stay until the war is won.

Okay, let’s do some math. 45 +12 = 57. 100 – 57 = 43.  So what did the 43 percent of respondents — just 2 percent fewer than those who back a continuing, if non-combat role for Canadian troops in Afghanistan — think about the future of the mission? We don’t know, because – somewhat unbelievably – that result simply isn’t provided. Seriously, ITQ reread the article three times, thinking she was  somehow missing it, but no, it’s just not there. The fact that it isn’t, and that we don’t even know what the third option*–  if any — might have been, makes this poll virtually useless as far as presenting anything approaching an accurate snapshot of Canadian public opinion.

UPDATE:ITQ partially retracts her accusation, and takes full responsibility for her sloppy interpretation of the regional data, buried in which is the fact that 37 percent of respondents want to see Canadians leave “immediately,” although that still leaves six percent unaccounted for. But she maintains that that number — which is higher than she would have expected, frankly — should have been presented with the rest of the topline data. So there.  (She also now wonders why there was no option that would have troops leave in 2012, with no continuing mission — “development and training”-focused or otherwise, and can’t help but think that including that as a possible outcome could have substantially altered the results. But that’s a polling nitpick, not a reporting one.)

Oh, and the story also fails to give field dates, sample size or margin of error — all of which is helpful, if often overlooked information when attempting to determine how much credibility to give a particular poll, but that’s a comparatively minor sin. Excluding the views of 43 percent of respondents, on the other hand, is pretty much indefensible, not to mention inexplicable.

*For the record, ITQ suspects that there was, in fact, a third choice presented to respondents, although really, even if there wasn’t, the fact that such a sizeable contingent was undecided, or went with ‘none of the above’ would still be noteworthy; based on previous polling on the Afghanistan issue, it almost certainly proposed a scenario in which Canadian troops would remain until 2012, but then withdraw on schedule, and withdraw completely, not simply reconfigure the mission into one focused on ‘training and development.’

Bookmark and Share
  • http://intensedebate.com/people/VinceClortho VinceClortho

    Good for you. Poll reporting is often lacking in depth or sometimes reality….good to see you trying to raise the bar amongst your peers.

    Bravo.

    • http://intensedebate.com/people/KadyITQ KadyITQ

      On my fourth read of the story, I did manage to uncover 37 percent of the missing respondents, and take full responsibility for not noticing it before, although honestly, it should have been in the topline results. I've updated my post accordingly.

      • http://intensedebate.com/people/VinceClortho VinceClortho

        Nothing wrong with expecting clarity. It shouldnt be such a chore. But this is also because poll analysis generally requires more than a headline….not to be critical, but well suited to a longer form in a newspaper. But I suspect th issue isnt amoutn of ink, paper or bits but the number of "brains per minute" analyzing and explaining.

        Keep trying to raise the bar.

  • jarrid

    Also in the Leger/Sun poll, some federal national numbers confirming a recent spate of polls:

    Conservatives 36
    LIberals 30
    NDP 17
    Greens 8
    Bloc 8

    Tories lead 40-34 in vote-rich Ontario

    Is Canada headed in the right direction? 48% of Canadians say yes while 30 % say no.

    • http://intensedebate.com/people/KadyITQ KadyITQ

      Yes, those results can be found here — again, no sample size, dates or margin of error, which really does drive me nuts, but the results are consistent with what we've been seeing lately, so it was spared my wrath as far as the reporting thereof.

      • http://intensedebate.com/people/VinceClortho VinceClortho

        and no sub region breakdowns…grrrrrr

      • … jjm …

        No sample size, dates, margin of error … another frustrating aspect is that the methodology wasn't reported. Was this an online poll or a telephone poll? if it was an online poll, the MRIA (Market Research Intelligence Association) explicitly prohibits members from posting a margin of error for an online poll result because, one could argue, an online poll is not "truly" random. Mind you, many firms, including Leger, ignore this.

  • http://intensedebate.com/people/SeanStok SeanStok

    From the linked story:

    ""It's surprising there is as much as 12% who want to stay until the war is won because that is about as open-ended a mandate as we could have," said Drapeau."

    I hate to break it to Drapeau, but you can usually count on 12% of respondents to support most anything. Particularly when the question would imply to some that the war is winnable (which is not to get into a debate about the veracity of such a claim, simply to note that going the extra mile and asking if we should stay until the war is won plants something of a seed in minds of some who hear the question.)

  • http://www.intensedebate.com/people/SeanStok SeanStok

    From the linked story:

    ""It's surprising there is as much as 12% who want to stay until the war is won because that is about as open-ended a mandate as we could have," said Drapeau."

    I hate to break it to Drapeau, but you can usually count on 12% of respondents to support most anything. Particularly when the question would imply to some that the war is winnable (which is not to get into a debate about the veracity of such a claim, simply to note that going the extra mile and asking if we should stay 'until the war is won' plants something of a seed in minds of some who hear the question.)

  • http://www.intensedebate.com/people/SeanStok SeanStok

    From the linked story:

    ""It's surprising there is as much as 12% who want to stay until the war is won because that is about as open-ended a mandate as we could have," said Drapeau."

    I hate to break it to Drapeau, but you can usually count on 12% of respondents to support most anything. Particularly when the question would imply to some that the war is winnable (which is not to get into a debate about the veracity of such a claim, simply to note that going the extra mile and asking if we should stay 'until the war is won' plants something of a seed in the minds of some who hear the question.)

  • Anon

    Why would ITQ — to continue on this interesting third-person vein — even pay attention to anything in the Sun papers? Even Nanos won't poll for them anymore, so they've to go to Leger which is really a Quebec polling company.

    • http://intensedebate.com/people/SeanStok SeanStok

      I've worked in a lot of places (blue collarish) where the Sun was the only paper people would read. It's very much worth paying attention to what they publish, if for no other reason than understanding the worldview of many voters.

  • knick

    All of which would seem to suggest that polls are only as reliable as knowing the complete wording of all of the questions and objective reporting of the results. Too often the media tries to manipulate the result by headlining one aspect which, if taken on it's own, misrepresents the total result.

  • http://intensedebate.com/people/Tridus Tridus

    What is this "non combat mission" people speak of? This isn't Toronto, how are you there at all without soliders to defend you?

    Are we just going to pull everybody back into a fortified compound and never leave? That'll cut down the casualties for sure, but it's also completely ineffective and pointless.

    This is a great example of how useless polls are. How many people gave that answer because it sounds nice and didn't put any thought into if its even possible?

From Macleans