Trudeaumaniaish

By Aaron Wherry - Friday, May 10, 2013 at 10:46 AM - 0 Comments

The April polling averages from threehundredeight.com.

If the Liberals continue on the trend line of the last few months, they should be at 100% by the fall.

Checking in

By Aaron Wherry - Thursday, February 14, 2013 at 1:11 PM - 0 Comments

The January poll averages from threehundredeight.com.

Checking in

By Aaron Wherry - Monday, December 10, 2012 at 1:19 PM - 0 Comments

The latest monthly polling averages from threehundredeight.com.

Mind the ‘credibility’ gap

By Colby Cosh - Monday, October 29, 2012 at 11:05 AM - 0 Comments

(J Pat Carter/AP Photo)

Over the weekend, the estimable David Akin was talking U.S. politics with Ipsos’s Darrell Bricker on Twitter when he noticed an unfamiliar verbal oddity in a Reuters report on the polling firm’s recent survey of early voters.

Obama leads Romney 54 per cent to 39 per cent among voters who already have cast ballots, according to Reuters/Ipsos polling data compiled in recent weeks. The sample size of early voters is 960 people, with a credibility interval of plus or minus 3.5 percentage points.

Huh, what’s this “credibility interval” business? Sounds like a different name for the good old margin of error! But why would we need a different name for that? This question, it turns out, is the pop-top on a can of worms.

The polling business has a problem: when most households had a single land-line telephone, it was relatively easy to sample the population cheaply and well—to estimate quantities like voter intentions in a clean, mathematically uncomplicated way, as one might draw different-coloured balls from a single urn to estimate the amounts of each colour amongst the balls on the inside. That happy state of affairs has, of course, been reduced to chaos by the cell phone.

The cell phone, increasingly, does not just divide the population into two hypothetical urns—which is basically how pollsters originally went about solving the problem. Its overall effect (including the demise of the telephone directory) has affected the math of polling in several ways, all of them constantly intensifying; declining response rates to public surveys (“Get lost, pal, you’re eating up my minutes”) are the most obvious example. Put simply, individual members of the public are no longer necessarily accessible for polite questioning by means of a single randomizable number that everybody pretty much has one of. The problem of sampling from the urn has thus become infinitely more complicated. Pollsters can no longer assume that the balls are more or less evenly distributed inside the urn, and it is getting harder and harder to reach into the urn and rummage around.

So how are they handling this obstacle? Their job, at least when it comes to pre-election polling, is becoming a lot less like drawing balls from an urn and more like flying an aircraft in zero-visibility conditions. The boffins are becoming increasingly reliant on “non-probability samples” like internet panel groups, which give only narrow pictures of biased subsets of the overall population. The good news is that they can take many such pictures and use modern computational techniques to combine them and make pretty decent population inferences. “Obama is at 90 per cent with black voters in Shelbyville; 54 per cent among auto workers; 48 per cent among California epileptics; 62 per cent with people whose surnames start with the letter Z…” Pile up enough subsets of this sort, combined with knowledge of their relative sizes and other characteristics, and you can build models which let you guess at the characteristics of the entire electorate (or, if you’re doing market research, the consumerate).

As a matter of truth in advertising, however, pollsters have concluded that they shouldn’t report the uncertainty of these guesses by using the traditional term “margin of error.” There is an extra layer of inference involved in the new techniques: they offer what one might call a “margin of error, given that the modelling assumptions are correct.” And there’s a philosophical problem, too. The new techniques are founded on what is called a “Bayesian” basis, meaning that sample data must be combined explicitly with a prior state of knowledge to derive both estimates of particular quantities and the uncertainty surrounding them.

A classical pre-election voter survey would neither require nor benefit from ordinary knowledge of the likely range of President Obama’s vote share: such surveys start only with the purely mathematical specification that the share must definitely be somewhere between 0 per cent and 100 per cent. A Bayesian approach might start by specifying that in the real world Obama, for no other reason than that he is a major-party candidate, is overwhelmingly likely to land somewhere between 35 per cent and 65 per cent. And this range would be tightened up gradually, using Bayes’ Law, as new survey information came in.

This is probably the best way, in principle, to make intelligent election forecasts. But you can see the issues with it. Bayesianism explicitly invites some subjectivity into the art of the pollster. (Whose “priors” do we use, and why?) And in making the step from estimating the current disposition of the populace to making positive election forecasts, one has to have a method of letting the influence of old information gradually attenuate as it gets less relevant. Even nifty Bayesian techniques, by themselves, don’t solve that problem.

Pollsters are trying very hard to appear as transparent and up-front about their methods as they were in the landline era. When it comes to communicating with journalists, who are by and large a gang of rampaging innumerates, I don’t really see much hope for this; polling firms may not want their methods to be some sort of mysterious “black box,” but the nuances of Bayesian multilevel modelling, even to fairly intense stat hobbyists, might as well be buried in about a mile of cognitive concrete. Our best hope is likely to be the advent of meta-analysts like (he said through tightly gritted teeth) Nate Silver, who are watching and evaluating polling agencies according to their past performance. That is, pretty much exactly as if they were “black boxes.” In the meantime, you will want to be on the lookout for that phrase “credibility interval.”  As the American Association for Public Opinion Research says, it is, in effect, a “[news] consumer beware” reminder.

Checking in

By Aaron Wherry - Tuesday, October 9, 2012 at 11:51 AM - 0 Comments

Both the Conservatives and New Democrats slip a little nationally. The Conservatives take a lead in British Columbia and maintain leads in Alberta, the Prairies and Ontario. The NDP leads in Quebec and the Atlantic.

Checking in

By Aaron Wherry - Wednesday, September 12, 2012 at 3:14 PM - 0 Comments

Only two national polls in August, but the Conservatives gain a slight advantage.

A year ago, the Conservatives went into the fall with a five-point advantage. Two years ago, the Liberals were at 29% and the NDP was at 16%. The New Demorats have now led the Liberals for 16 months.

The increasingly inexact science of polls

By Aaron Wherry - Thursday, September 6, 2012 at 3:13 PM - 0 Comments

Peter Loewen considers the recent failures of political polling.

The problem is not with the first part. Pollsters have developed very clever ways to obtain people’s honest opinions. They can use live people on the phone. They can find respondents over the Internet. They can use “interactive voice recognition,” in which a poll is conducted via a pre-recorded phone message and respondents key in their answers. Data collection is more affordable than ever, and text messaging and smartphone apps are promising even swifter data collection.

The problem is the second part. Pollsters simply do not know enough about who responds to polls via some media, who replies through others, and what kinds of people ignore polling requests entirely. The problem isn’t getting sample, it’s getting good sample. Simply knowing a respondent’s demographic information is not enough to correct for bad sampling. The result is that we cannot extrapolate with sufficient accuracy from our samples to the whole population. We cannot, in other words, know with much confidence the likely outcome of an election before the votes are cast.

The most popular decision a politician ever makes

By Aaron Wherry - Thursday, July 19, 2012 at 1:16 PM - 0 Comments

Bob Rae’s popularity is surging on word that he will be going away.

Bob Rae, the interim Liberal leader, meanwhile, jumped from 32.2 on the Nanos leadership scale to 41.5 when judged on the same qualities. Mr. Rae announced in June that he was out of the race for the leadership of his party. “Many times when leaders step down, people have a more favourable opinion of them,” said Mr. Nanos. Mr. Rae “has a little bit of an exit honeymoon.”

Checking in

By Aaron Wherry - Wednesday, July 11, 2012 at 2:03 PM - 0 Comments

Eric Grenier has updated his monthly polling averages for June at threehundredeight.com.

The New Democrats are first in British Columbia (+6), Quebec (+19) and Atlantic Canada (+10). The Conservatives are first in Alberta (+41), the Prairies (+7) and Ontario (+4).

(Editor’s note: For basically as long as this blog has existed, I’ve more or less imposed a ban on “horse-race” polling; my view being that the fussing over every new poll was generally unnecessary and often unhelpful. Especially during the minority parliament years, I attempted to maintain some kind of high-minded approach, avoiding the clamour over every little twitch and hiccup in the party numbers. I think I also once tried to avoid providing free time to the latest party adverts. I’ve slowly come to abandon those principles. I’ve long since abandoned that ban on ads. And while I still don’t think polls should generally dominate the discussion, I’ve realized it’s also silly to ignore them. I also think Eric’s monthly numbers and historical charts provide important perspective. So from here on, I’ll be checking in once per month with Eric’s latest averages.)

A sensitivity measuring contest

By Aaron Wherry - Friday, June 29, 2012 at 10:29 AM - 0 Comments

Nanos polling for the CBC—see here and here —finds a plurality of Canadians consider the NDP to be the party most sensitive to the needs of seniors, students, new Canadians and the unemployed. The Conservatives are deemed most sensitive to the needs of small business.

I think I’d like whether Canadians just generally associate the NDP with the adjective “sensitive.”

This 2010 poll by Abacus suggests that could be the case: “On the positive side, the NDP was identified as being in touch with Canadians, honest, having a good leader, and standing up for working people.”

Reality check

By Aaron Wherry - Monday, May 28, 2012 at 11:50 AM - 0 Comments

Harris-Decima checks in with the electorate.

Recently, NDP leader Thomas Mulcair has made comments regarding the impact of the oil sands on the Canadian economy. Have you read, seen or heard something about these comments?

Yes. 44%
No. 55%

By party affiliation, only a majority of Conservative voters (53%) were aware. By age group, only a majority of those over 50 were aware. Awareness was lowest among women, those 49 and younger and those in Quebec.

Luckily for voters, they will have the next three years to become acquainted with the particulars of the resource development debate.

The motivation gap

By Aaron Wherry - Tuesday, January 17, 2012 at 3:36 PM - 0 Comments

Eric Grenier notes an interesting distinction made in the latest polling data from Ekos.

This is the first publicly released report of federal vote intentions from EKOS since the May 2011 election, and the poll shows a much closer race between the Conservatives and the New Democrats than we’ve seen elsewhere. The Conservatives finished with 31.4% support in this survey, compared to 29.5% for the New Democrats and 24.8% for the Liberals … But EKOS also filtered these numbers out according to who voted in the May 2011 election, weighing them accordingly. With those weightings, EKOS pegs Conservative support at 36.7%, with the NDP at 27.8% and the Liberals at 21.9%.

That is quite a big difference between the voting intentions of the general population and the voting population. It does not surprise me that there would be a disparity, though this is larger than I would have expected.

A Senate if we must, but must we have a Senate?

By Aaron Wherry - Tuesday, July 12, 2011 at 12:37 PM - 6 Comments

It is obviously noteworthy that the Canadian public largely supports Stephen Harper’s proposed reforms to the Senate, but when the options are put side-by-side, the Canadian public is still relatively split.

Which of these statements comes closest to your own point of view?

Canada does not need a Senate, all legislation should be reviewed and authorized by the House of Commons 36%
Canada needs a Senate, but Canadians should be allowed to take part in the process to choose senators 40%
Canada needs a Senate, and the current guidelines that call for appointed senators should not be modified 5%
Not sure 19%

For that matter, as many Canadians support Stephen Harper’s calls for term limits and elections (70% and 72%) as support Jack Layton’s call for a national referendum (71%).

Advising the Governor General

By Aaron Wherry - Friday, April 29, 2011 at 2:30 PM - 36 Comments

Ekos asks what should happen if a minority government is defeated soon after the next election.

The EKOS-iPolitics survey finds two-in-five Canadians – 43 per cent – think the governor general should call on the leader of the Official Opposition to form a new government if the next prime minister’s party is immediately defeated. Only 19 per cent of Canadians think another election should be called. The remaining 38 per cent either had no opinion or refused to respond.

Leger asks a similar question and gets a similar response.

Projecting the projections

By Aaron Wherry - Friday, April 29, 2011 at 1:46 PM - 1 Comment

Paul Adams considers the seat projection predicament.

For the record, I am generally a believer in the usefulness of seat projections, and continue to be so in this election, which promises to be historic in some ways. However, I also think that there are reasons to be more cautious about them in elections where there is potential structural change, and where there is volatility late in the campaign.

Projecting backward

By Aaron Wherry - Friday, April 29, 2011 at 11:00 AM - 53 Comments

Alice Funke challenges the seat projection phenomenon.

The inherent bias in seat prediction methodologies to favour previous election results means they tend to overly favour parties set to lose seats, such as the Liberals and Bloc Québécois in the last election. They also tend to miss the likelihood of parties on the rise to gain seats, such as was the case with the Conservatives in the last election. Only the NDP, whose vote intention numbers showed little gain by the end of the 2008 campaign, saw seat count predictions on both sides of its eventual total.

Another problem for the seat projection methodologies is that they are backward-looking. They’re using days-old polling data at a time of incredible movement in the polls, and laying that on top of results from the last election when incumbency was a factor for some political parties’ votes that is no longer at play.

From the magazine

By Aaron Wherry - Thursday, April 21, 2011 at 4:51 PM - 4 Comments

Here is the magazine piece on Michael Ignatieff’s current situation. Here is the math.

In order to do so, the Liberals first need their supporters to return. According to analysis from Alice Funke ofpunditsguide.ca, the loss of Liberal seats in 2008 had less to do with other parties than with a drop in the Liberal vote from 2006 levels. The 800,000 voters that failed to materialize in 2008 are key to Liberal hopes in 2011. In tandem, the Green vote must decline—in 29 of the 31 ridings the Liberals failed to retain in 2008, Funke finds, Green support increased.

Even then, there is the small matter of the NDP and the current reality of political fragmentation. A plurality of Canadians—according to Innovative Research Group’s Canada 20/20 online panel for Maclean’s and Rogers Media—may agree with Ignatieff on student aid and a majority may agree with him on corporate taxes and pension reform, but while Harper is alone on one side of the argument, Ignatieff is competing for such voters. (For complete poll results see macleans.ca/electionpoll.) And NDP support has proved resilient. In the wake of Jack Layton’s performance in the leaders’ debates, the New Democrats have even risen in some polls.

Adventures in probability

By Aaron Wherry - Monday, April 18, 2011 at 4:06 PM - 25 Comments

Michael Veall cautions against reading too much into sudden poll changes.

So even if a party is up say 3.5 percentage points comparing a new poll with a previous poll, if each poll had a margin of error of 3.1 percentage points, the 3.5 percentage point increase should be compared to a margin for error of about 4.4 percentage points. There is a reasonable chance that the party’s vote intention share in the total population did not change at all: all that happened was that the pollsters randomly happened to choose more of the party’s supporters in the second poll.

The margins of errors for changes in leads can be twice as large again. If a party is leading by 5 percentage points in one poll and then by 9 percentage points in the next poll, the margin of error around that 4 percentage point gain could be over 8 percentage points. While probably the lead increased, there is still a significant chance that the lead decreased.

Vague notions

By Aaron Wherry - Thursday, April 7, 2011 at 1:55 PM - 49 Comments

Fifty-four percent of Canadians prefer a Liberal-NDP coalition to a Conservative majority, but a plurality of Canadians feel uneasy about the idea of coalition government.

When you think about the idea of a coalition government in Canada do you have a positive, somewhat positive, somewhat negative, or negative impression?

Positive 18%
Somewhat positive 22%
Somewhat negative 17%
Negative 32%
Unsure 12%

Statistical probabilities

By Aaron Wherry - Thursday, March 3, 2011 at 9:18 AM - 51 Comments

Pollsters continue to debate the meaning and prominence of their work.

Gregg said the proliferation of sometimes conflicting polls and the hypeventilating analysis that frequently accompanies them does not strengthen democracy. On the contrary, he said: “Rather than have a public that’s informed, you have a public that’s misinformed.” He said he’s not arguing that polls should be ignored; only that their import needs to be interpreted much more cautiously. Rather than pontificate on weekly fluctuations in individual polls, he said it makes more sense to average the results of various surveys and look at the trends over longer periods of time.

It is probably important to consider, as Eric Grenier did this week, how much and how often polling responses change when an election campaign is conducted. Consider, for instance, that the last three changes in government were not obviously foretold by publicly available polling data released immediately before the election was called. Continue…

In defence of the humble pollster

By Aaron Wherry - Friday, February 25, 2011 at 12:53 PM - 14 Comments

Last week, Darrell Bricker and John Wright defended their profession against recent criticism. Today, Michael Adams makes his own case.

Any serious research firm is capable of providing an accurate, representative snapshot of the attitudes and opinions of ordinary Canadians. Election numbers repeatedly affirm that pollsters can produce good numbers. Considering those numbers with a reasonable degree of literacy – an understanding of the capabilities and limitations of the science – is not just a job for the number crunchers. It’s a job for all of us.

Everything you know is of questionable accuracy

By Aaron Wherry - Monday, February 14, 2011 at 9:05 AM - 27 Comments

In a pair of stories, Joan Bryden effectively questions the basis for 85% of political discussion in popular media.

There’s broad consensus among pollsters that proliferating political polls suffer from a combination of methodological problems, commercial pressures and an unhealthy relationship with the media … ”The dirty little secret of the polling business . . . is that our ability to yield results accurately from samples that reflect the total population has probably never been worse in the 30 to 35 years that the discipline has been active in Canada,” says veteran pollster Allan Gregg, chairman of Harris-Decima which provides political polling for The Canadian Press.

Further thoughts from Alice Funke and David Coletto.

We remain torn over hypothetical options

By Aaron Wherry - Tuesday, February 1, 2011 at 9:12 AM - 103 Comments

When Nanos polled a thousand Canadians in January 2009 about what should happen if that year’s federal budget was defeated, 49% favoured an election, 42% said the opposition should be invited to form a coalition.

About two weeks later, when Ekos asked a thousand Canadians to choose between a Conservative government and a Liberal-NDP coalition, 50% favoured a coalition government, 43% favoured a Conservative government.

Last June, when Harris-Decima surveyed a thousand Canadians, 55% of respondents indicated some support for at least some kind of cooperation between the Liberals and New Democrats.

And now, in a new Ipsos Reid poll of a thousand Canadians, 55% state a preference for a Conservative minority government, 45% favour a Liberal-NDP coalition.

The more you know

By Aaron Wherry - Thursday, January 27, 2011 at 9:38 AM - 125 Comments

Abacus asks Canadians how they feel about the corporate tax rate.

Once respondents were made aware of Canada’s tax position relative to the United States, Germany, Japan, and Britain, two statements were presented – the federal government’s argument and the opposition parties’ argument. “This finding suggests that as Canadians become aware of how low Canada’s tax rates are compared to other countries, it becomes more difficult to convince them to support them.”

“Right now, public opinion is firmly aligned with the opposition parties,” said Coletto.  “Only 21% of respondents buy the job creation argument when given the alternative to spend more on health care or to reduce the deficit.” The survey then asked Canadians if they support or oppose the government’s plan to continue with the corporate tax cuts.  In total, 52% strongly or somewhat oppose the government’s plan, while 26% support or strongly support it.