Inkless Wells

Inkless Wells

Paul Wells on all the latest out of Ottawa—along with the occasional post about jazz. Follow Paul on Twitter: @InklessPW
He also offers his thoughtful perspective of Stephen Harper’s last 10 years in his recent eBook, The Harper Decade.

Something's happening to my cell phone. I keep getting an Ekos.

by Paul Wells on Monday, June 1, 2009 1:13pm - 11 Comments

Frank Graves, president of Ekos, sent this note on what ITQ readers will already know is a fairly poll-intensive day. Graves argues for his methodology and promises more poll-related fun over the next few days.

Hi Paul

I think the real value and implications of this poll may be missed. It has a huge sample. As you know it also includes cell phones( i believe you received a call from our robot). [True story — pw] One unreported finding is that those who don’t include the growing cell only population (nearly one in four households in the United States now and Canada is trailing but following that pattern) will have increasing difficulties covering key parts of the population . Liberal supporters, for example, are higher in the cell only segment. This trend to cell only households is also much more pronounced in Quebec adn amongst 18 to 40 year olds of moderate and lower socioeconomic status.

What is unusual, and we think valuable about the poll is the ability to drill down and get a very detailed demographic and regional profile of the eligible voter population. Because of the very large sample sizes we can also examine daily trend without resorting to the 3 day moving average approach. This allows us to see some interesting shifts and test some key hypotheses. For example, we saw the Liberals opening up a five point lead since the announcement of the 50B dollar deficit. It’s possible some other factor explains this highly significant shift but it’s highly unlikely. It is also notable that the Tory “attack” ads don’t appear to be bearing any immediate fruit for them.

There are a range of analyses we will be rolling out in the coming days. For example we will be analysing where voters have migrated since the last election adn who is doing well with first time voters.

The economy is clearly a crucial factor but different parts of the economy are having diverse effects in different segments of the population. For instance, It appears that a sense that the economy may be turning the corner is overall working mildly in favour of the Prime Minister and the CPC. Although Canadians are not highly optimistic a sense that things look better than they did three months ago is growing amongst older more affluent voters, who had defected from the Tories but were starting to return . Alarm over the deficit numbers may be extinguishing this recovery for the conservatives. This mild recovery of confidence in the economy is not being felt by younger voters who are more tuned into labour markets (which continue to deteriorate) as opposed to the equity markets that affluent older voters are following. These contradictory forces are crucial to understanding the shorter and medium term prospects for the parties adn they should factor into election timing questions.

In a shameless plug for the new suite of polling methodologies we are developing I note that the in depth analysis of diverse demographic segments is beyond the reach of most on-line polls which are restricted to self selected group which opts in to the “panel” . These on line approaches also excludes a very large number of households which don’t have broadband internet access. The approaches we are trying to bring together are based on the conviction that all members of the population should be able to appear in one’s sample and that the only method for recruitment is through a random invitation to participate. Problems with rising rates of refusals are no reason to abandon probability sampling . We will be linking this work with more detailed diagnostic work using our hybrid online-phone panel which allows in depth questioning of a more lengthy issues outside the reach of an IVR (robo-poll). The problems aren’t the online method of contact; it can work very well. It is the problem of assuring random probability sampling while using those methods. In our view, all polling should be based on random, probability sampling that includes the entire eligible population.

Cheers

Frank

Bookmark and Share
  • Dot

    Did the Roman empire do extensive polling before it declined?

    • http://intensedebate.com/people/Jack_Mitchell Jack_Mitchell

      Actually it was illegal to consult astrologers about when the Emperor would die.

  • Pundits' Guide

    The regular Carleton University book on the 2008 Canadian Federal Election just came out, and in the chapter on media and polling, it is noted that while the Ekos tracking polls looked like outliers during the campaign, later study suggested that they in fact had a 3-day or so lead over the other pollsters which they attribute to this new IVR technology.

    Now, I believe one of the Carleton folks was also affiliated with Ekos at the time, but anyways that was their take on it.

  • Stats PhD

    Most recent data on cellphone-only incidence from CWTA/Decima in Nov 2008 (http://www.newswire.ca/en/releases/archive/Novemb… and StatsCan in Dec 2007 (<http://www.newswire.ca/en/releases/archive/Novemb… shows that we're at 6% of households being cellphone-only. And, it was 5% in 2006, according to both sources. Plus, this proportion of the population tends to be largely 18-29 year-old youth in demographic composition – a small segment of the population, and one that does not vote in high numbers.

    So, I'm afraid it's going to be long, long time before you can make a solid argument that telephone surveys are inaccurate because they are missing this cellphone-only segment of the population.

    IVR and online may be valid ways to conduct surveys, but let's not play fast, loose, and vague with these cellphone-only stats in order to make the case. The American stats on cellphone-only penetration and their growth rates are significantly higher on both scores than what we see in Canada – and perhaps surprisingly so. But, we cannot point to their rates/trends and ignore our own.

    • Frank Graves

      The rates of growth of cell only households are quite dramatic and while Canada still significantly trails the United States it is closing the gap and will catch up. In Quebec our evidence suggests that the incidence may be approaching 20%. So the 2007 Stats Can data (which is based on a still earlier survey) is hopelessly out of date.
      Excluding the cell phone only population is increasingly permitting a controllable source of survey bias to persist. Depending upon the population parameter being estimated the effects can indeed be significant , both statistically and substantively . This is partiularly true when there is a correlation bewteen the cell only demographics and the parameter being estimated. In the case of examples such as BQ support in Quebec the effects are large.
      Also, telephone surveys Canada can include the cell only population and lets remeber IVR is also a telephone survey, albeit using a computer rather than live interviewer to collect the data.
      I do agree with the commentators last point

      • Stats PhD

        I think your new technique is innovative and exciting, Mr. Graves. But, for other reasons than the cellphone-only argument. I want to be careful re: Canadian cellphone-only data.

        Specifically, the StatsCan data I refer to above is actually from a survey conducted in December 2007, not sooner. I apologize for posting the wrong link – here it is for your perusal: http://www.statcan.gc.ca/daily-quotidien/080423/d… The incidence of cellphone-only households is 6.4% in their survey, up from 5.1% in December 2006. And, the CWTA/Decima survey I linked to above was conducted in Late 2008, which found 6% incidence, up from 5% in their 2006 survey.

        • Stats PhD

          So, as of the end of 2007 or 2008 Canada was at 6% incidence. Not very high. And, based on this study (http://www.canada.com/windsorstar/story.html?id=a… 68% of the cellphone-only is in the Under 35. Historically, youth do not turnout to vote in high proportions. I'm not questioning your political polls. I just don't think the youth are a large enough segment of the population or of actual voters specifically for their inclusion or exclusion in political polls to be an issue re: the results you get.

          I agree that the rates of growth are worth noting, in terms of percentage increases. i.e. the increase from 5% in 2006 to 6% in 2008 in the Decima/CWTA survey is a 20% increase in 2 years. But, really, it's a slow, consistent inching upwards – no major jumps. And, it's starting from nothing.

  • Stats PhD

    Most interestingly, I wonder if the future does hold cellphone-only households given the finding that 69% of the current cellphone-only population is male. Given the youthful character of this segment, I'm going to assume a high proportion of single people. When they settle down into a long-term relationship with a significant other, will they give in to the need for a combined telephone line and will it be a land line? Will the signficant other make that decision for them?

    And, given the slow march (of 1-2 point increases every 1-2 years) on this front over the past few years, I would find it shocking to see an increase from 7.7% in Quebec in Dec 2007 (http://www.canada.com/windsorstar/story.html?id=a… – StatsCan) to 20% in Spring 2009. That's a 175% increase in under 2 years, without a major consumer event to provoke it.

    Great to speak with you – I do find your new technique exciting, innovative, and interesting. And, I find this topic very intriguing.

    • R in calgary

      Most young couples I see tend to move to two cellphones with no landline (or Skype on their broadband when required). And the house I moved into in Calgary didn't have a landline installed – the couple there hadn't bothered as they both just used their cellphones. Even banks have had to make adjustments to allow their lending systems to use cellphones for loan applicants that have no landline (having worked for a bank I know this firsthand).

    • Frank Graves

      thanks for the helpful links and thoughtful comments. T his is a very interesting aspect of what is a true revolution going on in survey research methods. I am optimistic (but not convinced ) that new hybrid data collection methods and better sampling methods can allow us to have true probability sampling in the internet era. I find the vast majority of what's out there doesn't even come close to this standard and the fact that exotic matching and weighting systems can calibrate to provide reasonable estimates of vote outcomes doesnt make me believe it is a valid sample

  • Stats PhD

    I'm interested in the same – although there really is no such thing as true probability sampling. I mean, when only 10-15% of people picking up the phone agree to do a telephone survey, on average … and the fact that there are many not even answering because of Caller ID … there is no true probability sample.

    But, IVR robocall-surveying sounds so intriguing. I'd love to see an independent study on this, perhaps by Pew or other academia? How are the cellphone contacts reacting to the calls? Higher or lower response rates as phone? Because they are paying for the call – so, I'd expect them to be angry and issuing refusals. And, given that many Canadians are anti-automated CSR systems, do these surveys have to be 5 minutes maximum? On the plus side, I imagine that respondents may be more honest when we remove the human interviewer, and you'd get more reliable results on reported awareness and personal and financial information etc.

    Anyway, many exciting things afoot to study!

From Macleans