Pollsters scramble to explain how polls could be so wrong in three provinces
By The Canadian Press - Thursday, May 16, 2013 - 0 Comments
OTTAWA – Canada’s pollsters have struck out and are now scrambling to explain how…
OTTAWA – Canada’s pollsters have struck out and are now scrambling to explain how their predictions in three consecutive provincial elections turned out to be so wildly wrong.
Strike three came Tuesday when Christy Clark’s Liberals came back from the polling dead to easily recapture government in British Columbia.
The polls were similarly, though not quite as spectacularly, wrong in last September’s Quebec election and the April 2012 Alberta election.
Ipsos Reid, the country’s largest pollster, conducted an exit poll in B.C. that it says explains why its poll one day before Tuesday’s election — which had given Adrian Dix’s NDP an eight-point lead —was so far off the mark.
It suggests 11 per cent of B.C. voters made up their minds in the polling booths. That, combined with low voter turnout and a shift in voters’ priorities over the course of the campaign to the economy from wanting change, resulted in Clark’s stunning, come-from-behind victory, according to the Ipsos analysis of the exit poll.
“In this regard, our last (pre-election) poll should have had more attention paid to those who intend to get out and vote as opposed to just those who issued a voter preference,” the analysis says.
But Harris Decima chairman Allan Gregg said it’s just not plausible for pollsters to blame their failure to accurately predict results in three different provincial elections on a last-minute decision or shifts among voters to the winning party.
“The shortest answer is this should not happen, it literally should not happen,” Gregg said in an interview Wednesday.
“It’s the law of large numbers. If you do enough polling using properly framed samples and properly crafted questions, you will get the right answer. And if you’re not getting the right answer then obviously you’re doing something wrong.”
Gregg believes a “confluence” of factors are responsible for the problem: unreliable methods used to contact survey respondents, inexperienced, fly-by-night pollsters and badly-framed questions.
Traditional telephone surveys are no longer reliable since many people don’t answer the phone or have given up their land lines for cell phones. They tend to be skewed in favour of older voters who are most likely to support conservative parties, Gregg said.
Online polls — such as the B.C. polls done by Ipsos and Angus Reid — survey self-selected respondents who sign up to participate in a pollster’s Internet panel. Gregg said they tend to be skewed toward younger, urban, educated voters who are most likely to support the NDP.
As well, Gregg said a lot of inexperienced pollsters have jumped into the field, particularly in provincial campaigns, offering their surveys to the media for free as a way to promote their fledgling companies.
“Basically, charlatans who really don’t know what they’re doing or don’t have the resources or tools to do it properly.”
But even among the heavy-hitters in the polling industry, Gregg said there’s a misunderstanding of how campaigns work and what motivates people to vote.
Polls typically ask respondents which party they intend to vote for, which Gregg noted is not the question on the ballots voters actually cast. Ballots give the names of the local candidates running for election in each riding, which tends to give an advantage to well-known incumbents, especially in non-urban ridings.
“So what (polls) are doing is they’re wildly under-influencing the impact of incumbency,” said Gregg, who used to be the Conservative party pollster during the Brian Mulroney era.
That may explain why polls were so wrong in Alberta, where Premier Alison Redford was deemed a goner but pulled out a majority win, and in Quebec, where Jean Charest managed to come within a hair of hanging onto government despite polls predicting his Liberal party would be humiliated.
Sophisticated polls conducted for political parties plug in the names of local candidates when surveying voters, Gregg said, whereas polls conducted for the media — for free — generally can’t afford to go to such lengths.
Better internal party polling may explain why Clark confessed Wednesday that she “wasn’t as surprised as everybody else” by the B.C. result.
Ipsos senior vice-president John Wright disputed suggestions that there’s a problem with polling methodology. He said it’s just a reality that in some elections, such as B.C.’s, campaigns are hard-fought and voters “make up their minds in the voting booth.”
Clark appeared to agree with that assessment.
“The polls do not tell us how people are going to vote because voting day is the only day that they vote. It’s like me asking you what you’re going to have for dinner a month from now,” she said.
Still, Wright agreed with Gregg that the relatively recent influx of fly-by-night pollsters is helping to discredit the public opinion research industry and journalists’ addiction to polls has exacerbated the problem.
While he doesn’t favour banning polls during elections, he said: “We need to have a discussion about the use of polls in election campaigns.”
In Wright’s view, polls should not be published unless they’re accompanied by complete data tables detailing the response rate, the amount of weighting that was done to make the sample reflective of the population as a whole, and other factors.
“With no disclosure, there should be no exposure.”
Gregg suggested the country’s most credible pollsters should join forces during election campaigns to conduct credible national polls for the media on a not-for-profit basis, using a hybrid of land-line phone, cell phone and online surveys.
To allow the proliferation of unreliable campaign polls to continue unchecked “is completely dangerous” for the health of our democracy, he added.
“It absolutely changes the (media) coverage … and media coverage influences voting behaviour, there’s no question about that at all.”
-
Polls: Not to be believed except when they are to be believed
By Aaron Wherry - Wednesday, May 15, 2013 at 11:15 AM - 0 Comments
Monday night: the difference between earlier public polling and the election result demonstrates that Justin Trudeau is in over his head.
Tuesday night: the difference between public polling and the election result demonstrates that polls are not to be given much credence.
Actually, Forum Research’s last poll in Labrador was fairly reflective of the final vote—and Conservatives could point to that as evidence of Mr. Trudeau driving voters away, but then the 20-point drop they claimed on Monday night becomes a nine-point drop (from 57% in early April to 48% on by-election night).
-
Trudeaumaniaish
By Aaron Wherry - Friday, May 10, 2013 at 10:46 AM - 0 Comments
The April polling averages from threehundredeight.com.
If the Liberals continue on the trend line of the last few months, they should be at 100% by the fall.
-
Checking in
By Aaron Wherry - Thursday, February 14, 2013 at 1:11 PM - 0 Comments
The January poll averages from threehundredeight.com.
-
Checking in
By Aaron Wherry - Monday, December 10, 2012 at 1:19 PM - 0 Comments
The latest monthly polling averages from threehundredeight.com.
-
Is it possible to manipulate a poll?
By Aaron Wherry - Tuesday, November 20, 2012 at 12:24 PM - 0 Comments
Dave Climenhaga suggests the Greens are trying to game the polls in Calgary Centre.
So it is significant – though impossible to criticize – that a Green Party organizer emailed committed supporters a note headed “There is another poll tonight – be sure to pick up,” not long before the latest survey. “Word from Chris Turner’s Head Quarters is that another poll is being conducted at this very moment,” said the email from Green Party Volunteer Co-ordinator Natalie Odd to committed Turner supporters. “Please be sure to pick up any calls your receive this evening!”
The emails were followed up with phone calls to supporters, although the pollster actually appears to have called a day later than the party expected. In addition to such emails and calls, Mr. Turner’s supporters posted similar messages on Facebook and some people distributed the call-display number the polling company was using.
This bit of gamesmanship seems to involve two assumptions: that it’s possible to manipulate a poll and that a good showing in a poll can precipitate a good showing on election day. The sample sizes used so far in Calgary Centre have been relatively small, but I’m not sure what the relative odds are that something like this could be pulled off. I can imagine that poll numbers could influence turnout and the result, but what are the odds that alerting supporters to the possibility of a poll would result in enough people responding to a survey who normally wouldn’t to significantly impact the results of that poll? I invite any and all mathematicians in the crowd to sort that out.
-
Will you return to the NHL when the lockout ends?
By macleans.ca - Friday, October 26, 2012 at 10:56 AM - 0 Comments
…
-
China can’t buy Albertans’ love
By Charlie Gillis - Wednesday, October 24, 2012 at 5:25 PM - 0 Comments
New poll suggests the province’s residents are cooling to Beijing, despite all that cash spent in the oil patch
Photograph by Jeff McIntosh/CP
China’s thirst for oil is Alberta’s insurance policy—a guarantee of investment and jobs amid economic turmoil in the U.S. and Europe. Small wonder, then, that Albertans have been reliably sanguine when it comes to forging close economic ties with China. More sanguine, at least, than other Canadians.
That might be changing.
In an Alberta-only poll released today, 64 per cent of respondents said they disapproved of Chinese investment in the province if it took the form of full ownership of assets; a lukewarm 37 per cent said they found partial ownership acceptable. That’s a sharp break from one year ago, when a healthy majority agreed with the proposition that they should welcome Chinese investment.
-
Checking in
By Aaron Wherry - Tuesday, October 9, 2012 at 11:51 AM - 0 Comments
Both the Conservatives and New Democrats slip a little nationally. The Conservatives take a lead in British Columbia and maintain leads in Alberta, the Prairies and Ontario. The NDP leads in Quebec and the Atlantic.
-
Things look good around here — for now
By Paul Wells - Monday, September 17, 2012 at 11:31 PM - 0 Comments
A word cloud from the 2011 federal budget: economic, action, plan.
So it turns out Ipsos has been polling monthly, since 2007, in 24 countries around the world, about attitudes toward the local economy. The latest round landed in my emailbox tonight, and it’s very interesting reading. It helps explain why incumbent governments have been doing well in Canadian elections, and why economic insecurity is still a big ingredient in our politics. Let’s take a look.
A hefty .pdf of the annoyingly named “Global @dvisor” poll’s latest results is here. Let me highlight a few results.
Respondents were asked how they think their country’s economy is doing; how the “local” economy is doing in their area; and whether they expect things to get better or worse during the next six months. Canadian respondents were near the top in their perception of their country’s economy: 65 per cent think the situation in Canada is “very good” or “somewhat good” (“somewhat bad” and “very bad” were the other options), the fifth-highest result out of the 24 countries surveyed. Saudi Arabia (84 per cent), Sweden (81 per cent), Germany (69 per cent) and India (68 per cent) were ahead. Continue…
-
Checking in
By Aaron Wherry - Monday, August 20, 2012 at 12:26 PM - 0 Comments
A slight tightening in the national numbers for July.
By region, it’s a tie in the Prairies, while the NDP leads by six in British Columbia and the Conservatives lead by seven in Ontario.
-
Checking in
By Aaron Wherry - Wednesday, July 11, 2012 at 2:03 PM - 0 Comments
Eric Grenier has updated his monthly polling averages for June at threehundredeight.com.
The New Democrats are first in British Columbia (+6), Quebec (+19) and Atlantic Canada (+10). The Conservatives are first in Alberta (+41), the Prairies (+7) and Ontario (+4).
(Editor’s note: For basically as long as this blog has existed, I’ve more or less imposed a ban on “horse-race” polling; my view being that the fussing over every new poll was generally unnecessary and often unhelpful. Especially during the minority parliament years, I attempted to maintain some kind of high-minded approach, avoiding the clamour over every little twitch and hiccup in the party numbers. I think I also once tried to avoid providing free time to the latest party adverts.
I’ve slowly come to abandon those principles.I’ve long since abandoned that ban on ads. And while I still don’t think polls should generally dominate the discussion, I’ve realized it’s also silly to ignore them. I also think Eric’s monthly numbers and historical charts provide important perspective. So from here on, I’ll be checking in once per month with Eric’s latest averages.) -
Projecting the projections
By Aaron Wherry - Friday, April 29, 2011 at 1:46 PM - 1 Comment
Paul Adams considers the seat projection predicament.
For the record, I am generally a believer in the usefulness of seat projections, and continue to be so in this election, which promises to be historic in some ways. However, I also think that there are reasons to be more cautious about them in elections where there is potential structural change, and where there is volatility late in the campaign.
-
Why newspapers were a bit slow to respond to Layton’s big surge
By Josh Dehaas - Friday, April 29, 2011 at 7:45 AM - 9 Comments
‘It’s possible they thought the first poll was an anomaly’
Jack Layton made history last week when a CROP poll showed the NDP in first place in Quebec. It was the kind of shift that would surely alter media coverage overnight (and the results of his Federal Election Newspaper Analysis along with it), thought McGill University political scientist Stuart Soroka. “I thought Layton was going to spike in the volume of coverage and that the coverage was going to be more positive,” says Soroka, who has been crunching his numbers for Maclean’s each week. But his usual analysis, which captured 665 stories written from April 18 to April 24, showed only tiny increases for Layton. He was baffled.
But the results for Easter Sunday and Monday showed a clear shift. “The media was just slow to adjust,” says Soroka. “It’s possible they thought the first poll was an anomaly.” On Sunday and Monday, Layton’s share of “first mentions” (a tally of how often a leader’s name comes first in a story) doubled from 10 per cent to 20 per cent. While Ignatieff was up a tiny bit over the previous week—from 19 per cent to 21 per cent—Harper lost the most, falling from 66 per cent to 55 per cent.
The “net tone” results (more positive words near a leader’s name in articles equates to a higher score) also show an initially slow, but then sudden shift in favour of Layton. All three English party leaders earned slightly more negative press last week, but on Sunday and Monday, the media turned on Harper and Ignatieff. Harper’s net tone score dove from 0.77 last week to 0.54. Ignatieff plummeted from 0.84 to 0.49. Layton, on the other hand, improved from 1.27 to 1.55.
More telling is the sheer number of times Layton’s name appeared in print on Sunday and Monday. He went from 0.5 mentions to 1.2 per article, tying him with Ignatieff for the first time. And though he still trails Harper, who’s at 1.9 per article, Layton is suddenly impossible to ignore.
-
From the magazine
By Aaron Wherry - Thursday, April 21, 2011 at 4:51 PM - 4 Comments
Here is the magazine piece on Michael Ignatieff’s current situation. Here is the math.
In order to do so, the Liberals first need their supporters to return. According to analysis from Alice Funke ofpunditsguide.ca, the loss of Liberal seats in 2008 had less to do with other parties than with a drop in the Liberal vote from 2006 levels. The 800,000 voters that failed to materialize in 2008 are key to Liberal hopes in 2011. In tandem, the Green vote must decline—in 29 of the 31 ridings the Liberals failed to retain in 2008, Funke finds, Green support increased.
Even then, there is the small matter of the NDP and the current reality of political fragmentation. A plurality of Canadians—according to Innovative Research Group’s Canada 20/20 online panel for Maclean’s and Rogers Media—may agree with Ignatieff on student aid and a majority may agree with him on corporate taxes and pension reform, but while Harper is alone on one side of the argument, Ignatieff is competing for such voters. (For complete poll results see macleans.ca/electionpoll.) And NDP support has proved resilient. In the wake of Jack Layton’s performance in the leaders’ debates, the New Democrats have even risen in some polls.
-
Adventures in probability
By Aaron Wherry - Monday, April 18, 2011 at 4:06 PM - 25 Comments
Michael Veall cautions against reading too much into sudden poll changes.
So even if a party is up say 3.5 percentage points comparing a new poll with a previous poll, if each poll had a margin of error of 3.1 percentage points, the 3.5 percentage point increase should be compared to a margin for error of about 4.4 percentage points. There is a reasonable chance that the party’s vote intention share in the total population did not change at all: all that happened was that the pollsters randomly happened to choose more of the party’s supporters in the second poll.
The margins of errors for changes in leads can be twice as large again. If a party is leading by 5 percentage points in one poll and then by 9 percentage points in the next poll, the margin of error around that 4 percentage point gain could be over 8 percentage points. While probably the lead increased, there is still a significant chance that the lead decreased.
-
Harper's game
By Paul Wells - Monday, April 4, 2011 at 2:05 PM - 273 Comments
“Friends, remember. The global recovery is fragile,” Stephen Harper told a room full of Conservatives in St. John’s. It wasn’t a big room but it was reasonably well-packed; the friendly audience had the Prime Minister surrounded on every side.“Yes, Canada is doing relatively well,” Harper said. “But a sea of troubles is lapping at our shores.”
Reporters who’d been travelling with the Conservative leader longer than I had rolled their eyes. “Still lapping,” the guy from the CBC said cheerfully.
“Have you heard about the sea of troubles yet?” the lady from the Canadian Press had asked me that morning in Moncton. Apparently it’s a fixture of the Harper stump speech, although I had managed to miss it so far.
Here was my chance to catch up. Harper described the contours of the trouble sea to his latest audience: “Disaster in the pacific, chaos in the Middle East, debt problems in Europe, and all kinds of challenges — some very serious challenges — south of our border. Canada — this country — is the closest thing the world has to an island of stability and security. And we’ve got to keep it that way.”
So far Harper had been reading from a teleprompter, or perhaps by now reciting from memory, his voice brisk but flat. Now he spoke with real emotion. “What would the world think, were we as a country to suddenly head off in some high-tax economic direction, led by a reckless coalition without a coherent program or even basic national principles?”
This is the Harper pitch for 2011. He varies it at each stop. Sometimes he leaves the sea of troubles out. But it’s always the same argument. Life is not perfect in Canada but it’s getting better. Peril lies all around. If Canadians throw off the protective embrace of Harper’s Conservatives… well… well then the Visigoths will descend, won’t they? And by the time they are done with us, everything Canadians cherish will lie in ruin. Continue…
-
Would you like to see a one-on-one debate between Stephen Harper and Michael Ignatieff?
By macleans.ca - Wednesday, March 30, 2011 at 4:19 PM - 61 Comments
…
-
Harper's popularity slides in leadership poll
By macleans.ca - Monday, March 21, 2011 at 12:16 PM - 46 Comments
Government facing spate of scandals
A new poll shows that trust in Stephen Harper has declined in recent weeks following a series of political scandals. The Nanos Research poll shows Harper’s leadership index score has fallen from 99 in February to 83 in March. Michael Ignatieff did not benefit much from the drop, as his score only increased from 37 to 40. Meanwhile, NDP leader Jack Layton’s score jumped from 44 to 51. Harper’s decline in the poll is mainly attributed to a spate of political scandals, including party officials being charged with breaking election financing laws, and revelations that Bruce Carson, a former Harper aide, lobbied for dodgy water contracts that would have benefited him and his fiancee, a former escort. The government is also facing the prospect of being found in contempt of Parliament over two separate incidents.
-
Statistical probabilities
By Aaron Wherry - Thursday, March 3, 2011 at 9:18 AM - 51 Comments
Pollsters continue to debate the meaning and prominence of their work.
Gregg said the proliferation of sometimes conflicting polls and the hypeventilating analysis that frequently accompanies them does not strengthen democracy. On the contrary, he said: “Rather than have a public that’s informed, you have a public that’s misinformed.” He said he’s not arguing that polls should be ignored; only that their import needs to be interpreted much more cautiously. Rather than pontificate on weekly fluctuations in individual polls, he said it makes more sense to average the results of various surveys and look at the trends over longer periods of time.
It is probably important to consider, as Eric Grenier did this week, how much and how often polling responses change when an election campaign is conducted. Consider, for instance, that the last three changes in government were not obviously foretold by publicly available polling data released immediately before the election was called. Continue…
-
In defence of the humble pollster
By Aaron Wherry - Friday, February 25, 2011 at 12:53 PM - 14 Comments
Last week, Darrell Bricker and John Wright defended their profession against recent criticism. Today, Michael Adams makes his own case.
Any serious research firm is capable of providing an accurate, representative snapshot of the attitudes and opinions of ordinary Canadians. Election numbers repeatedly affirm that pollsters can produce good numbers. Considering those numbers with a reasonable degree of literacy – an understanding of the capabilities and limitations of the science – is not just a job for the number crunchers. It’s a job for all of us.
-
Coyne v. Wells on the uprising in Libya
By macleans.ca - Thursday, February 24, 2011 at 7:24 PM - 8 Comments
Should western leaders have dealt with Gadhafi in the past? Also: why the Tories have pulled ahead in the polls
RELATED: Read Paul’s column—‘What Canada can do about Libya’—in the March 7 issue of Maclean’s
-
On a scale of 1 to 5, how much do you worry about the economy?
By macleans.ca - Thursday, February 24, 2011 at 2:46 PM - 12 Comments
…
-
What do you think of the CBSC decision to change the lyrics of Dire Straits' "Money for Nothing"?
By macleans.ca - Friday, January 14, 2011 at 11:07 AM - 109 Comments
…
-
New Year brings drop in Conservative support
By macleans.ca - Wednesday, January 12, 2011 at 1:38 PM - 18 Comments
Tories lose ground in Ontario, Harper tied with Layton in approval rating: poll
A new survey conducted by Vision Critical and Angus Reid shows support for the Conservative government has dropped slightly to 34 percent, while Prime Minister Stephen Harper’s approval rating is now tied with Jack Layton’s at 26 per cent. The NDP, meanwhile, dropped 1 point to 17 per cent. The Liberals, now at 28 per cent, gained slightly among the 1,008 Canadian adults polled, but Michael Ignatieff, now at 12 per cent, has seen a 20-point drop in public approval. Canadians are most upset with the House of Commons, which earned a 47 per cent disapproval rate. The poll concludes that “the start of 2011 did not provide a boost to the Conservative Party, with a noticeable drop across the country and in Ontario—the key battle ground for the next federal election.” The Bloc and the Greens each gained one point and now sit at 11 and eight per cent, respectively.
