Beyond The Commons

Beyond The Commons

Aaron Wherry covers all the goings-on in and around Parliament Hill. Follow Aaron on Twitter: @aaronwherry

What went wrong with the projections?

by Aaron Wherry on Tuesday, May 3, 2011 3:58pm - 126 Comments

Eric Grenier explains where his system failed.

ThreeHundredEight.com projected two things: the popular vote and how that would transform into seats. It failed at the first, which means it failed at the second. 

Bookmark and Share
  • Ariadne

    Another reason is blind media leading blind Liberal insiders/handlers/hacks. Liberals listen only to but* lickers (pardon my language).

  • Guy

    I wonder why pollsters don't have a control sample? Each poll taken is with a different group of people and these groups are compared to one another. There is actually no common reference point between the groups. The pollsters could set up a large control group (say 3 times the usual sample size) and interview them on an ongoing basis during the campaign. Their regular polls can then be compared to the control group. Comparing how the control group changes versus the regular polling could highlight trends and such.

  • rick

    CBC had all along predicted a majority for the Conservatives.
    so not all projections were wrong. Just yours.

  • Darren

    Pollsters and media and biased toward the liberal and NDP agenda…they do not understand the pulse of the Canadian people.

  • Dot

    Does he have three hundred and seven more excuses?

  • Anon

    The polls were all over the place too because nobody could correctly model the left-right split in Ontario. In the end, the fear of an NDP government was just too much for some on the centre and the right, and Iggy was too right wing for those on the left.

  • avr

    I can only hope that Frank Graves, in particular, is laughed out of the industry.

  • http://secondthots.blogspot.com Dennis_F

    Or the pollsters are biased against or don't account for conservative support. The Conservatives have outperformed the polls in just about every election they've been in. Rob Ford decimated the pollsters' predictions in the Toronto mayoral race. Funny that.

  • http://secondthots.blogspot.com Dennis_F

    Didn't he use the term "undeniably accurate" for his last laugher of a poll?

  • MTB

    From now on, I'm going to forget looking at polls and just listen to what Chantal Hebert is predicting.

  • Orson Bean

    I think part of that is "please the pollster" bias. In some circles in this country, it's not considered trendy, fashionable or socially acceptable to be, or to be seen to be, conservative. It's well-documented in polling studies and literature that some people are reluctant to tell pollsters that they're going to vote a certain way if those people perceive that that's not what the pollster wants to hear.

    It reminds me of university, where left-of-centre students are often very vocal about their political views, constantly speak up in class about their views, and tend to be visible and active in campus politics. Right-wing students, not so much.

  • Jan

    But the 'split' means that the Conservatives benefit from the vote divided up amongst the opposition. He's talking about the modeling of that. So just measuring Con support doesn't help to predict. Are you paying attention at all?

  • http://ragingranter.blogspot.com Raging_Ranter

    Colby Cosh nailed it too. He predicted 160 seats. Only 7 seats out.

  • Iccyh

    Projections are nice and all, but why is anyone at all surprised they all got it so wrong last night? Basically all pollsters and predictors admitted that their models aren't set up to deal with the unprecedented happening.

    Really, if anyone actually manages to get a working crystal ball, please let me know. I'd like to borrow it for a bit.

  • hosertohoosier

    I see two problems. Firstly, seat projectors used old data (discounted somewhat), which simply didn't reflect the situation on election night. Grenier increased the polling decay rate, but even that wasn't enough. As a result, seat projectors are usually behind, and overestimate declining parties (eg. the Liberals in 2008 and 2011). We also lack a good mechanism for figuring out where a swing in the vote is going to happen – uniform swing is a clumsy substitute. Simulation models can help with this (calgary grit's simulation model was tied for second, as far as I know). Demographics may work as well, and of course, sub-regional polling would be of great help.

    Secondly, at the level of pollsters, I don't think we have good or consistent ways of screening for likely voters. Ekos was particularly harmed by this. While their topline results were way off, here is what you would get if you just looked at voters that indicated they would not change their minds:
    CPC: 38.64%
    NDP: 30.05%
    LPC: 18.94%
    Green: 5.1%
    Bloc: 5.7%
    Other: 1.5%

    This is almost exactly what we saw on election night. Of course one problem is that applying a likely voter screen of a particular level of strength may not work the same in every election. In a municipal election, where few people vote, you would want a very tight screen. In a high turnout election, you might want a loose one. The varying intensity of support among Canada's political parties are a more recent development, so the polling industry may not have had sufficient cause to adapt its assumptions.

  • hosertohoosier

    Chet was closer!

  • http://ragingranter.blogspot.com Raging_Ranter

    Really, must you force us all to eat humble pie? I actually like Chet, I just tire of his repetitive posting on the same subjects. Mind you, I'm tired of my own repetitive posting on the same subjects, and no doubt others are too. Perhaps time for a few months off. :)

  • http://ragingranter.blogspot.com Raging_Ranter

    Perhaps EKOS tried to be too fancy and applied too many fitlers.

  • http://halooverride.blogspot.com/ Halo_Override

    You like chet? I used to like Emily.

    Spoiler alert: the novelty wears off.

  • Missy

    hahahah, yes, we all need to change the channel, take a deep breath & relax knowing that PM Harper will get the job done. :)

    We've all had quite the adrenaline pump these past couple of weeks. Royal Wedding, bin Laden's death,
    Conservative Majority, NDP as opposition & the death of Libs & Bloc.
    Yep, we sure have had a bumpy fun ride, haven't we! hahahah

  • http://ragingranter.blogspot.com Raging_Ranter

    I used to like Emily.

    Never had that problem. :)

  • http://secondthots.blogspot.com Dennis_F

    But shouldn't pollsters then take that bias into account in their sampling? Instead, they seem to bask in feeding into it, or even use it as a form of push polling.

  • TwoYen

    It reminds me of when I was in BC when WAC Bennett was in power. It was not considered proper to support the Socreds, who were too populist for the elites but Bennett kept winning.

  • s_c_f

    That's the funniest part.

    His words:

    "it appears that what was previously thought unimaginable has become a point of consensus."

    "As shocking as those numbers would have seemed just a few weeks ago, they are now more or less stable and undoubtedly accurate. The only real question remaining — and, frankly, this is more a matter of anecdotal curiosity at this stage — is how these translate into the new seat distribution in Canada’s next Parliament."

    He sounded absolutely certain. No talk about his shoestring budget. Just absolute undeniable certainty about the accuracy. Sort of like the global warming zealots.

    LOL
    http://ipolitics.ca/2011/05/01/frank-graves-last-…

    Now he says:

    "We were almost spot on about the popular vote, but where we really screwed up was with the Tory majority"

    as if that makes any sense whatsoever, and also:

    "This is done on a shoestring budget, and while we do our best to try and understand our failures it requires analysis and time and nobody is really interested in that"

    That's exactly the worst thing you could hear from a pollster.

  • Trudeau lover

    Pollsters are snake oil salesmen employed by corrupt media organizations in order to manipulate and deceive the electorate. Pollsters are always wrong in their predictions, but their use comes when they enable the media to give everyone their "expert" opinions on the reading of the polls, and the possible outcomes that the polls suggest. Polls are for dogs and strippers.

  • Colin

    Social desirability bias (or the Bradley effect as it is sometimes called) is really not a measurable bias because it changes according to the current climate of public opinion about the subject at hand. Some pollsters do count this as one reason to use robo-calling services instead of live operators, or for the use of internet polls. I actually doubt that it is very important in this case, more likely is a failure to capture turnout effects.

  • Blacktop

    Nor me;<)

  • Niceguy

    Small 'c' conservatives, men, and older people are more private. They hang up on pollsters, or tell them to f$%k off. Lefties, young people, and chicks can't wait to tell the world how they are going to vote…and how much more wonderful this makes them. This is why polls always short-change the vote for conservative parties and inflate the vote of left-wing parties.

  • Crit_Reasoning

    I can't say I've ever had that problem either.

  • http://ragingranter.blogspot.com Raging_Ranter

    How about a simpler explanation? They were wrong. Some overestimated Conservative support. Most others under-estimated it. It's a very inexact science, and it is not surprising that some got it wrong. Some were close. They aren't likely corrupt or even biased. Just working in a field that isn't all the accurate.

  • Jan

    'Lefties, young people and chicks' – the Harper majority has begun in earnest. Only 4 more years of this.

  • http://ragingranter.blogspot.com Raging_Ranter

    I spent a good portion of today listening to progressives talking with grave concern about the election results. Couldn't wait to share with each other. I remained silent – as I always do on political issues at work – until someone dragged me into it. When I said how I voted, one guy – who is usually quite friendly and reasonable – told me I was part of the "stupid 40".

    This exchange seems typical. Progressives are wonderfully warm and friendly as long as they think you're one of them. When they find out you're not, they call you stupid and worse. I merely smiled and told him that while I may be stupid, reminding me of that fact is not likely to make me come around to his point of view. Left him with nowhere to go. How does one respond to something like that? :) He blushed so red I could almost feel the heat coming from him.

    This is a big problem for progressives. They think everyone is like them, because most vocal people are like them. The conservatives tend to be very belligerent online, but socially very quiet about their views. There are exceptions of course. But I rarely express mine publicly. (British pollsters called this the "Shy Tory" effect after John Major's surprise majority victory in 1992). Unlike progressives, we don't eat, sleep and breathe politics and government. In fact, most of us go long periods of time without thinking of it at all. Unfortunately for progressives, we aren't shy about voting when the time comes. And we aren't likely to vote for those who dismiss us as stupid. You'd think they'd clue in sooner or later. Evidently, it will be later.

  • FVerhoeven

    Stop the predictalbe polling results……..marked with an x

  • http://ragingranter.blogspot.com Raging_Ranter

    Chet has one he might not be using anymore.

  • Thwim

    "Unlike progressives, we don't eat, sleep and breathe politics and government. In fact, most of us go long periods of time without thinking of it at all."

    You say that as if it's a good thing, though.. that's what confuses me. Isn't thinking about how our government is handled something that all of us should be concerned about.. and not just come elections?

    Interesting point about being belligerent online, but not socially. Why the difference? What makes those online less worthy of consideration?

  • Iccyh

    I think it would be safer to wait and see before asking to borrow it. I have a suspicion it'd tell me to buy IBM, or to bet on Winnipeg, or perhaps on the Jays based on belief rather than their actual prospects, you know?

  • Jan

    But the parties use them – how do you explain that?

  • Colin

    To clarify I don't mean its existence can't be measured in other kinds of studies, I just mean that it can't be measured in such a way as to fix it on a poll to poll basis.

  • http://secondthots.blogspot.com Dennis_F

    No. He's saying the polling was wrong. Graves had the Tories at 35% and weak minority. They got 40% and a strong majority.

    Do you care about the truth? Of course not. Next.

  • http://secondthots.blogspot.com Dennis_F

    But wouldn't any pollster worth their money, or the profession in general, be interested in dealing with this anti-conservative bias? Instead, they seem intent on leaving it alone, or even use it as a form of push polling. Reason #1488 to despise pollsters?

  • http://ragingranter.blogspot.com Raging_Ranter

    I believe they were saying when Harper first got in in 2006 that, "Thank God we've only got a few years of this." Read Paul Wells today. "The victims of Harper's resilience are piling up." There's the thing you want, and then there's reality. Harper won't last forever, but he's had some impressive staying power.

  • http://ragingranter.blogspot.com Raging_Ranter

    We also sometimes plan weekend events around long term weather forecasts, even though they aren't much good. Pollsters are the best tool available. They aren't perfect, but they aren't completely useless either. In the absence of crystal balls, we rely on pollsters. A little too much it seems, but there it is.

From Macleans