Beyond The Commons

Aaron Wherry covers all the goings-on in and around Parliament Hill. Follow Aaron on Twitter: @aaronwherry

Categories: Beyond The Commons, Capital Read

‘The data will almost certainly be biased’

by Aaron Wherry on Friday, September 30, 2011 5:32pm - 23 Comments

New figures show a 69% response rate for the National Household Survey—higher than Statistics Canada expected, but still not sufficient to replace the long-form census.

“You simply can’t get reliable data from a voluntary survey,” said John Brewster, who teaches statistics at the University of Manitoba and is president of the Statistical Society of Canada. ”I mean, this is something we teach in every course in statistics, for example. The data will almost certainly be biased. And we don’t really know at this time the magnitude or direction of that bias.”

Tags: census, John Brewster, long-form census, National Household Survey, statistics, Statistics Canada

More by Aaron Wherry

Renee S

Since the Harper government doesn’t do well with evidence-based decision-making anyway, this can only be good news for them. No more pesky facts to get in the way of their ideological decisions!
TonyAdams

Maybe someone should tell/teach Brewster that coercion does not lead to reliable data either.

Also, since last year when Cons first eliminated long form census, I have asked people at a few dinner parties about filling in census. I have not done it myself but I have asked about 50 other people and not one person has said they filled out entire form truthfully.

Even people who agree that census is good/necessary said there was a least one question that they thought was too private and did not answer truthfully.

Lucky Luke

I don’t believe you actually know 50 people.

BTW…Arsenal is going down…go Lilywhites! Audere est Facere!
Anonymous

Way to miss the point Tony. Lots of people who don’t like the census did so anyway when they thought they had to. Now they know there will be no repercussions, they really don’t give a shit. I wasn’t going to fill in the census myself as a form of protest, until it was made clear to me i would likely be costing my small community money and services. Making the census no longer manitory is simply a lincense to indulge in childish form of selfishness and petty communal vandalism.

TonyAdams

Is there really tens of thousand Jedis living in Canada or were they lying because State should not know your religion?

CTV, May 2003:

An astonishing 20,000 Canadians declared themselves to be followers of the religion of Jedi, the guardians of peace and justice in the Star Wars flicks, Statistics Canada reported Tuesday in the latest, and final, data to come from the 2001 census.

Anonymous

A non-voluntary census can account for margins of error due to known sample and response rates.

A voluntary census cannot.
Anonymous

http://en.wikipedia.org/wiki/Jedi_census_phenomenon

It took me 5 minutes of googling to establish this was not just an astonishing spontaneous phenomenon as you imply, but actally part of an organized international protest. Of course it doesn’t take away from the fact a good number of people don’t liked to be asked the question[ i don't see the purpose of it myself - but i;ll bet there is not a conspiratorial one world govt marxist hegemony behind it] Nonetheless i’m prepared to bet most of the drivers of the movement are loopy libertarians and haters of govt in general – doubtless Palin is on the list somewhere

Anonymous

All taken into consideration, Tony. It’s the sampling they royally screwed up.
Anonymous

Until you demonstrate an elementary understanding of statistics, best stay out of the conversation. You’re only embarrassing yourself.

Anonymous

no kidding.

Tony, even if everyone in Canada lied a bit under the old system, at least you have a hundred years of data that can be validated agaist previous data.

Under this system we have a major break in the data set! and for sure, as you correctly indicated, an over representation of “dinner party” people and an under representation of poor.

The only justification I can see for this is so that the con’s can fake their way into believing they have made Canada better.

Anonymous

“Maybe someone should tell/teach Brewster that coercion does not lead to reliable data either.”

That is precisely wrong, and it is that kind of gut-feel logic that had the world convinced for thousands of years that the world was flat.

On the contrary, mandatory responses are essential to ensure reliable data as they ensure there is no self-selection bias.

To illustrate, suppose each citizen in Canada has a probability X that they will fill out a voluntary form, 0 < X < 100. Some citizens have higher values of X, some lower values of X. And most importantly, each person's value of X will be influenced by their own socioeconomic factors (e.g. higher education implies higher X, fluency in English implies higher X, retirees may have higher X than young people, etc).

If we allow the census to be voluntary, the resulting data will thus be biased because it will be overrepresented by people with higher values of X and underrepresented by people with lower values of X. And because X is correlated with various socioeconomic factors, it will thus be biased along these factors also.

The way to avoid this bias? Make responses mandatory, and you set 20% of the people's X at 100% and the other 80% at 0%. You then ensure there is no correlation between the 20% who do respond and the 80% that don't as the only factor that determines which group you are selected into is due to the randomness of the selection.

TonyAdams

Thomas Sowell ~ Quest For Cosmic Justice:

We have seen how easy it is to go wrong by wide margins when dealing with American history. It is equally easy to go wrong with contemporary statistics. If one goes through enough numbers, one will eventually come upon some statistics that seem to fit one’s vision. These are what might be called ‘Aha’ statistics. Other statistics which suggest opposite conclusions bring no ‘Aha’ but are more likely to be glided over and forgotten.

Wall St Journal ~ When We See What We Want:

In 1981, Harvard paleontologist Stephen Jay Gould published “The Mismeasure of Man,” a fierce critique of various scientific attempts to measure human intelligence. Mr. Gould began the book with a takedown of “craniometry,” a popular 19th-century technique that attempted to find correlations between skull volume and intellect …

The larger lesson of the Gould-Morton affair is that bias is everywhere, that many of our studies are shot through with unconscious errors and subtle prejudices. To Paul Simon, we see what we want to see and disregard the rest.

In recent years, it’s become clearer that these psychological shortcomings are a serious societal problem. Because we believe we’re impervious to bias—we’re blind to our own blind spots—we assume that our judgment isn’t affected by financial incentives or personal opinions.

Anonymous

Is quoting random people with no connection to the topic of self-selection bias your idea of a rebuttal?

Lame. I’m interested in talking with actual people, not 30 year old quotes.
Anonymous

Furthermore, having now read the Wall St Journal article, the mere fact you seem to have confused the concept of statistical bias with bias in the sense of prejudice (note: the terms are completely unrelated) shows me you have absolutely no clue what you are talking about.

Anonymous

Priceless. After holding forth on what constitutes reliable data, Tony says: “I have asked people at a few dinner parties about filling in census.”

Next up, he’s going to lecture us on irony.
Anonymous

Just goes to prove that you and your entourage are less than very honest.

TimTayler

hmm, released late on a Friday afternoon, not in the StatCan’s Daily. Well hidden on their website.
How much of that response was those partly completed forms that statcan decided to accept, and how partial is partly complete? Did anyone respond besides internet users?

Anonymous

I expect Stephen Gordon will be weighing in on this.

Anonymous

I declined the opportunity to complete the long form because a) I knew it was no longer a statistically useful instrument and b) I wouldn’t waste my time lending credibility to the Tories’ half-a$$ed notion of public policy research.

Anonymous

exactly. Why should i put my good healthy stats in thier database?

I’d prefer if everyone making 18k a year filled it out

JimmyJames

Quietly posted “Preliminary report on methodology options for the 2016 Census”
At http://www12.statcan.gc.ca/strat/index-eng.cfm
JimmyJames

Quietly posted “Preliminary report on methodology options for the 2016 Census”
At http://www12.statcan.gc.ca/strat/index-eng.cfm
JimmyJames

Quietly posted “Preliminary report on methodology options for the 2016 Census”
At http://www12.statcan.gc.ca/strat/index-eng.cfm