Newsmakers of the week

By macleans.ca - Thursday, November 15, 2012 at 5:30 AM - 0 Comments

Lukas Coch / Reuters

The full-bore FAQ

The royal family still feeds Prince Charles now that he’s 64—just not seven eggs at breakfast, as per popular myth. That and other long-held beliefs about the Prince of Wales were laid to rest this week in an FAQ released by Clarence House on the occasion of Charles’s birthday, as part of the royals’ ongoing effort to put a more normal face on their sometimes remote heir. He doesn’t duck taxes, advocate use of dangerous alternative therapies or loathe modern architecture, according to officials. And he doesn’t spend any—repeat, any—time thinking about being king. All of which is too bad: those were things that made him interesting.

Now, put that wand away

No sooner is Barack Obama re-elected than his first Supreme Court appointee is out spreading his radical anti-princess agenda. Sonia Sotomayor appeared on Sesame Street to confront a pink muppet named Abby who was dressed as a Disney-style princess, telling her that pretending to be a princess “is definitely not a career,” and encouraging girls to be “a teacher, a lawyer, a doctor, an engineer and even a scientist” instead. But her profession hasn’t been very helpful to Kevin Clash, the puppeteer who plays Elmo on the iconic kids’ show. He took a leave of absence after being accused of sexual misconduct, an accusation that was then recanted in a statement by the accuser’s law firm. Maybe he’d be happier if more people became princesses, not lawyers. Continue…

274.5? 315? A U.S. election apéritif

By Colby Cosh - Tuesday, November 6, 2012 at 3:44 AM - 0 Comments

New York magazine’s Adam Pasick has put together a charming collection of electoral vote forecasts from American political notables. Because most of these people have some known prior commitment to one side or the other, the table makes for an interesting diorama of America’s political camps: the Republicans and conservatives are all over the map, with as many predicting massive triumph for Mitt Romney as there are imagining disaster, and the Democrats and liberals/leftists are united behind a party line of certain victory, though nobody thinks it will be too impressive. Most of the latter are in a band between 285 Obama electoral votes and 305—keeping in mind that the president got 365 last time out. The most pessimistic Romney backer of the bunch, who I guess would have to be Buzz Bissinger, has a higher number for Obama than any of the overt Obama voters I can identify. Granted, he’s called Buzz for a reason (and that reason is that he is approximately three-quarters crazy on a good day).

If you throw out Jim Cramer’s prediction, which he made explicitly to set himself apart from the crowd and give himself a longshot chance of looking like a lone genius 24 hours from now (good luck with that), the mean of all guesses is 274½ electoral votes for Obama. The Republicans are running about 20 below that on average, the Democrats about 20 above. It may be noteworthy that absolutely none of the Democrats and liberals is willing to place Obama as high as FiveThirtyEight.com’s Nate Silver, whose mean EV estimate for the incumbent at this hour is 315 and rising.

Screening Pasick’s “pundits” for general cool-headedness, insider knowledge, or just having strong incentives to get the call right doesn’t seem to help extract a signal from the noise. Bowtied eminence George F. Will, who is a conservative but hardly the model of a death-or-glory demagogue, submitted exactly the same numbers as Glenn Beck. Slate’s Dave Weigel, perhaps the only person on the list who has officially declared He Is Not Voting For Either Of These Bozos, is predicting a narrow 276-262 Romney win.

In lieu of a prediction, because I am short on insights into this particular election and there’s no reason you should care either way, I would offer one warning from this spring’s Alberta vote: it is dangerous to attempt to infer the true state of a political race from the last-minute behaviour of the candidates. The Progressive Conservatives, widely perceived to be behind on the final weekend, appeared to be defending what ought to have been relatively safe ridings in Calgary and the province’s northeast. Although I was cautious and emerged from the election only lightly bespattered with facial egg, watching Premier Redford move about encouraged me to think the PCs really were in serious danger.

In fact, if you think about it, the ridings—or states—where a candidate can do the most marginal good with a late appearance are not necessarily the ones closest to parity or 50-50 overall. If a candidate is blitzing a state with TV ads, that may just mean the TV audience in that state is especially promising in some respect. If a candidate is visiting in person, he may be forsaking a closer but less tractable state race for one in which a weak organization needs the personal touch, or the youth vote has an unusual quantity of undecideds, or… well, you can imagine an infinity of scenarios yourself.

It is tempting to regard late candidate activity as a form of revealed preference, a Fool Killer that smashes through verbiage to the truth. Sometimes, though, it is not telling you what you might think. In this election, a late rush by both sides toward Pennsylvania, a vote-rich state that Obama won by 10 points in 2008, has people wondering if Romney really might be ahead nationally and putting the president on the ropes. Well, for all I know he might be. But the real signal is probably simpler than that: “Hey, Pennsylvania doesn’t have early voting.“

Now is the time to argue about Nate Silver’s method

By Jaime Weinman - Monday, November 5, 2012 at 4:13 PM - 0 Comments

Eric Gay/AP

My colleague Colby Cosh got a lot of favourable notice for this piece about Nate Silver. Well, to paraphrase Patty and Selma from The Simpsons, I believe the best way to write a post is to leech off the popularity of another post. But also, I had something I wanted to say about Silver. Or more about perceptions of Silver.

Arguing about Silver has suddenly become a big thing in the last few weeks. He’s been taking criticism from at least three different directions. First, there are the people who don’t so much have a beef with the man himself as with the idea of him as an oracle. Silver does not claim to be an all-knowing prognosticator. But there are people out there who see him as such. I know a few people who have told me the U.S. presidential race can’t be close because Silver gives Obama a 75 per cent chance of winning, or who simply refer to Silver alone to tell them what’s going to happen. Silver has never set himself up as a prophet, and he can’t be blamed for his adherents, but I do think that the way he expresses his findings is very vulnerable to misinterpretation. It’s true that Silver isn’t literally saying that Romney has almost no chance of winning; he’s talking about probabilities. But they do lend themselves to those talking points, even if that’s not his fault.

The second anti-Silver faction consists of Republicans and conservatives who detect a bias in his work. You can see one such argument here. To some extent, this may be misdirected: the polls themselves, particularly the swing state polls, have tended to show Obama in the lead. But there is an argument that any system with a subjective element — and while Silver has stayed true to his system, it includes subjective decisions about how things are weighted — has an element of bias in it.

From a Republican point of view, Silver may have stood in the way of developing a media narrative of Romney momentum after Denver; as the national polls started shifting toward Romney (many of them have since shifted back again) Silver continued to say that Obama was the favourite based on swing-state polling. I don’t know how harmful that actually was to the Romney campaign, but perceptions of momentum are quite important to some of the people who run campaigns.

Finally, there are the pundits and reporters, many of whom see Silver as an annoyance at best and an enemy at worst. Here’s the article that mentions some of the anti-Silver sentiment brewing among pundits; one of the people from Politico also mocked people who think Silver has some kind of “secret sauce,” when he’s actually just “averaging public polls.” Though, I don’t think he, as opposed to some of his fans, have claimed he has any “secret sauce.”. As many people have already noted, this is developing a lot like arguments over baseball statistics: who has the better perception of the game, the guy who goes out and talks to the players and has inside knowledge of what goes on, or the guy who sits at home with the statistics and plugs them in?

It’s not as simple as that in election forecasting, because analyzing baseball statistics is about analyzing things that have happened, while polling is about things that haven’t happened yet. With predicting the future, it is probably true that inside knowledge can help you see things the stats don’t — just as someone who knows about a baseball star’s drinking or drug problem will do a better job than the sabermetrician of foreseeing his upcoming decline. An example from 2010: Jon Ralston of the Las Vegas Sun, who predicted Harry Reid would be re-elected at a time when Silver gave Reid’s opponent “a better than three-in-four chance,” thanks to polls that were turning in her favour. Ralston didn’t have a lot of evidence to give us, but he did have his reputation as a clued-in, plugged-in observer of Nevada politics, and what he observed was that Reid’s political operation was as strong as ever, and that his opponent wasn’t being carried along as strongly as she should have been by that year’s Republican wave. This is the sort of thing you can probably see that the polls can’t – if you’re intimately familiar with the political workings of a particular area or state.

But most people who forecast elections, of course, have no such familiarity. Even people who live in a state, while their local perspective is almost always more insightful (for example, a local can tell you not only what ads are on television, but what they’re saying on local news and the weirdly political world of sports radio), are going to have limited knowledge of what’s going on. Other people just tell you that someone must be winning, no matter what the polls say, because he had huge turnout at some rally, or the locals seem to be getting really excited about him. And then there’s the most problematic of all these little subgenres: talking to interested parties and asking them if they think they’re winning. Of course they think they’re winning, and can give you all kinds of reasons why. But why on earth would that be more useful to us than looking at an average of the polls?

It always seems counterintuitive and wrong that a guy staying at home with the numbers, never setting foot in a state, could have more insight into the situation than someone who does shoe-leather reporting on the ground. And in one sense that’s true: the number-crunchers would be nothing without the people in the field doing the polls. But in terms of the actual process of figuring out who’s likely to win, this is probably one of those situations Bill James described in response to criticisms of the sabermetric method: told that sabermetricians can’t see the forest for the trees, he pointed out that the trees aren’t in a good position to tell us how tall they are. To go and report on baseball up close, you find out a lot of things, but you still need hard cold statistics to put the season into perspective for you and find out stuff like, well, who’s ahead in the standings. If you ignore the stats and just “trust your gut,” you get something like this piece from Peggy Noonan, a full-blown pundit in good standing, where she argues that the polls don’t matter because she’s hearing a lot of people have signs in their yards:

There is no denying the Republicans have the passion now, the enthusiasm. The Democrats do not. Independents are breaking for Romney. And there’s the thing about the yard signs. In Florida a few weeks ago I saw Romney signs, not Obama ones. From Ohio I hear the same. From tony Northwest Washington, D.C., I hear the same.

Is it possible this whole thing is playing out before our eyes and we’re not really noticing because we’re too busy looking at data on paper instead of what’s in front of us? Maybe that’s the real distortion of the polls this year: They left us discounting the world around us.

This is the kind of predicting that, even if it turns out to be right, is completely useless. It’s useless because it’s not based on anything; it tells us nothing except that humans will pick out the signs and portents that tell them what they want to hear. With Ralston’s prediction about Harry Reid, we could at least look back on it after the election was over and learn something about the way politics works in Nevada. But most gut-feeling punditry, I think, is closer to the Noonan quote: someone is going to win because I feel it in my bones, or a particular candidate has the “momentum.”

This is why I think now is right time to argue about whether Silver’s method makes sense, rather than after the election. There are many reasons why he might wind up calling the election wrong (along with a lot of other poll aggregators, pundits, and so on). There are also flukey reasons why he might be right. It doesn’t exactly matter a lot, since the actual election renders all advance polling completely irrelevant. The question is, though, before the election, when predictions are all we have to go on, which predictions are useful? Which methods shed some light on the state of the race at a particular time? Which posts seem like they might have something useful to tell us about where the polls stood, even if things change?

I think there are some ways in which the Silver method makes the race more confusing, creating the impression that races are less volatile than they really are, and under-stating the chances of surprises like the Harry Reid/Sharron Angle race. And I think it’s important to take the polls in conjunction with some bigger-picture reporting. But that’s not the choice we usually have: the choice we have is between poll aggregation and analysis, and pundits reporting “SHOCK POLL: OBAMA LOSING [name of state]” or telling us how David Axelrod thinks things are going. With a choice like that, no wonder people turn to Nate Silver.

Tarnished Silver: Assessing the new king of stats

By Colby Cosh - Sunday, November 4, 2012 at 4:19 AM - 0 Comments

The whole world is suddenly talking about election pundit Nate Silver, and as a longtime heckler of Silver I find myself at a bit of a loss. These days, Silver is saying all the right things about statistical methodology and epistemological humility; he has written what looks like a very solid popular book about statistical forecasting; he has copped to being somewhat uncomfortable with his status as an all-seeing political guru, which tends to defuse efforts to make a nickname like “Mr. Overrated” stick; and he has, by challenging a blowhard to a cash bet, also damaged one of my major criticisms of his probabilistic presidential-election forecasts. That last move even earned Silver some prissy, ill-founded criticism from the public editor of the New York Times, which could hardly be better calculated to make me appreciate the man more.

The situation is that many of Nate Silver’s attackers don’t really know what the hell they are talking about. Unfortunately, this gives them something in common with many of Nate Silver’s defenders, who greet any objection to his standing or methods with cries of “Are you against SCIENCE? Are you against MAAATH?” If science and math are things you do appreciate and favour, I would ask you to resist the temptation to embody them in some particular person. Silver has had more than enough embarrassing faceplants in his life as an analyst that this should be obvious. Continue…

Mind the ‘credibility’ gap

By Colby Cosh - Monday, October 29, 2012 at 11:05 AM - 0 Comments

(J Pat Carter/AP Photo)

Over the weekend, the estimable David Akin was talking U.S. politics with Ipsos’s Darrell Bricker on Twitter when he noticed an unfamiliar verbal oddity in a Reuters report on the polling firm’s recent survey of early voters.

Obama leads Romney 54 per cent to 39 per cent among voters who already have cast ballots, according to Reuters/Ipsos polling data compiled in recent weeks. The sample size of early voters is 960 people, with a credibility interval of plus or minus 3.5 percentage points.

Huh, what’s this “credibility interval” business? Sounds like a different name for the good old margin of error! But why would we need a different name for that? This question, it turns out, is the pop-top on a can of worms.

The polling business has a problem: when most households had a single land-line telephone, it was relatively easy to sample the population cheaply and well—to estimate quantities like voter intentions in a clean, mathematically uncomplicated way, as one might draw different-coloured balls from a single urn to estimate the amounts of each colour amongst the balls on the inside. That happy state of affairs has, of course, been reduced to chaos by the cell phone.

The cell phone, increasingly, does not just divide the population into two hypothetical urns—which is basically how pollsters originally went about solving the problem. Its overall effect (including the demise of the telephone directory) has affected the math of polling in several ways, all of them constantly intensifying; declining response rates to public surveys (“Get lost, pal, you’re eating up my minutes”) are the most obvious example. Put simply, individual members of the public are no longer necessarily accessible for polite questioning by means of a single randomizable number that everybody pretty much has one of. The problem of sampling from the urn has thus become infinitely more complicated. Pollsters can no longer assume that the balls are more or less evenly distributed inside the urn, and it is getting harder and harder to reach into the urn and rummage around.

So how are they handling this obstacle? Their job, at least when it comes to pre-election polling, is becoming a lot less like drawing balls from an urn and more like flying an aircraft in zero-visibility conditions. The boffins are becoming increasingly reliant on “non-probability samples” like internet panel groups, which give only narrow pictures of biased subsets of the overall population. The good news is that they can take many such pictures and use modern computational techniques to combine them and make pretty decent population inferences. “Obama is at 90 per cent with black voters in Shelbyville; 54 per cent among auto workers; 48 per cent among California epileptics; 62 per cent with people whose surnames start with the letter Z…” Pile up enough subsets of this sort, combined with knowledge of their relative sizes and other characteristics, and you can build models which let you guess at the characteristics of the entire electorate (or, if you’re doing market research, the consumerate).

As a matter of truth in advertising, however, pollsters have concluded that they shouldn’t report the uncertainty of these guesses by using the traditional term “margin of error.” There is an extra layer of inference involved in the new techniques: they offer what one might call a “margin of error, given that the modelling assumptions are correct.” And there’s a philosophical problem, too. The new techniques are founded on what is called a “Bayesian” basis, meaning that sample data must be combined explicitly with a prior state of knowledge to derive both estimates of particular quantities and the uncertainty surrounding them.

A classical pre-election voter survey would neither require nor benefit from ordinary knowledge of the likely range of President Obama’s vote share: such surveys start only with the purely mathematical specification that the share must definitely be somewhere between 0 per cent and 100 per cent. A Bayesian approach might start by specifying that in the real world Obama, for no other reason than that he is a major-party candidate, is overwhelmingly likely to land somewhere between 35 per cent and 65 per cent. And this range would be tightened up gradually, using Bayes’ Law, as new survey information came in.

This is probably the best way, in principle, to make intelligent election forecasts. But you can see the issues with it. Bayesianism explicitly invites some subjectivity into the art of the pollster. (Whose “priors” do we use, and why?) And in making the step from estimating the current disposition of the populace to making positive election forecasts, one has to have a method of letting the influence of old information gradually attenuate as it gets less relevant. Even nifty Bayesian techniques, by themselves, don’t solve that problem.

Pollsters are trying very hard to appear as transparent and up-front about their methods as they were in the landline era. When it comes to communicating with journalists, who are by and large a gang of rampaging innumerates, I don’t really see much hope for this; polling firms may not want their methods to be some sort of mysterious “black box,” but the nuances of Bayesian multilevel modelling, even to fairly intense stat hobbyists, might as well be buried in about a mile of cognitive concrete. Our best hope is likely to be the advent of meta-analysts like (he said through tightly gritted teeth) Nate Silver, who are watching and evaluating polling agencies according to their past performance. That is, pretty much exactly as if they were “black boxes.” In the meantime, you will want to be on the lookout for that phrase “credibility interval.”  As the American Association for Public Opinion Research says, it is, in effect, a “[news] consumer beware” reminder.

Do the math

By Aaron Wherry - Friday, October 7, 2011 at 9:44 AM - 3 Comments

Nate Silver measures the impact of campaign advertising.

Campaign ads matter more when a candidate can outspend the opponent. This simple fact sometimes gets lost because people fixate on the content of ads. But the volume of ads may matter more. Consider the 2000 presidential election. In the final two weeks of the campaign, residents in battleground state were twice as likely to see a Bush ad as a Gore ad. This cost Gore 4 points among uncommitted voters. The same thing happened in 2008, when Mr. Obama vastly outspent his Republican opponent, Senator John McCain.

'We need more of this kind of leadership and ideas exchanged between the parties'

By Aaron Wherry - Monday, February 8, 2010 at 12:52 PM - 14 Comments

In lavish detail, Nate Silver explains how Washington might create its own Question Time. Meanwhile, former McCain campaign strategist Mark McKinnon posits that a regular QT might help leach “partisan poison” from Capitol Hill.

All of which may serve to remind that it is not necessarily the institution or the idea of Question Period that is the problem here in Ottawa, so much as how it is used.

Mr. Overrated

By Colby Cosh - Tuesday, January 19, 2010 at 2:56 AM - 32 Comments

I hope, though I doubt, that Nate Silver’s performance during the stretch drive of the Massachusetts special Senate election will finally lead to him being downgraded from “All-seeing HAL-9000-esque quantitative wizard” to “Just another guy with a computer”. Armed only with the traditional maxims of psephological interpretation, which teach that a late polling break away from the incumbent party is a very unfavourable omen, one could have figured out ten days ago that repulsive Democratic candidate Martha Coakley was in a heap of trouble. Silver, with his revolutionary disregard for everything but the polling numbers, was still arguing as late as Thursday afternoon that Coakley was the clear favourite; he changed his mind at midnight that evening and acknowledged that Scott Brown had a puncher’s chance.

He continued to soothe jumpy Democrats throughout the weekend in the manner of a government radio station denying rumours of a coup d’etat, writing on Sunday night that the election was still a “toss-up”. (He had already cheated on his mechanical bride by citing traditional eyeball analysis from NBC’s Chuck Todd: “If this were any other state we’d say this one was over”). There followed another handful of balm. Tonight, as the national party prepares for the probably loss of its congressional supermajority, he’s still distributing it.

Silver (or “Nate P. Silver”, as I will always think of him) may still be “right”, in the limited sense that a probabilistic prediction about an event that will only happen once can be “right” at all. But even if Coakley does surprise everyone by pulling this election out, the gruesome lack of robustness in Silver’s approach should be evident. We don’t need an advanced proprietary model to tell us what the polls are saying with a five- or six-day lag time built in. I have always understood Silver’s core claim of special expertise to inhere in the ability to give useful information about the future. Boasts like “[the model] correctly predicted the outcome of all 35 Senate races in 2008” are nothing but distracting hype, since 95%+ of Senate elections are easy to call on the morning before they happen. And as Silver certainly knows, a model that delivers probabilistic estimates of outcomes of one-time events has to not only be “right”, but right in such a way that bettors using it would, in the long run, outperform other bettors or prediction markets not equipped with that model. That’s the only appropriate test, and I know of no evidence that he has passed it.

Vive le nerd

By Aaron Wherry - Wednesday, November 12, 2008 at 3:55 PM - 1 Comment

While we’re casting about for our own Obama, could we also find our own Nate Silver?