The FAP trap

rendering of Alpha Centauri system

In this artist’s rendering, the exoplanet Alpha Centauri Bb looms in the foreground, with the Alpha Centauri binary system in the background.

Almost one year ago, a team of astronomers announced a detection of a rocky exoplanet right next door in the star system Alpha Centauri, the closest to our own solar system. Yes, Alpha Centauri—that near-mythical system that has such a hold on our imagination, its fictional appearances have their own Wikipedia article.

Ok, ok, so this planet, named Alpha Centauri Bb, wasn’t actually habitable. It was too close to its star, more like a scorched, oversized Mercury than Earth. But the fact that a small rocky planet was right next door boded well for the likelihood that rocky planets were everywhere. Debra Fischer, a Yale exoplanet researcher, told the New York Times it was the “story of the century.” If Joe Biden were an astronomer, he’d have called it a big fucking deal.

Except…the detection wasn’t quite a slam dunk. The team, based in Geneva and led by astronomer Xavier Dumusque, found the planet by detecting the wobble that its gravity exerts on its star. But that wobble was so small that its signal was buried deep, deep within the noise of the data. They had to attempt to control for 23 different effects that could have thrown off their measurements—things like the star’s pulsations and magnetic spots. It was only after stripping them away, one by one, that a signal started to emerge. Here’s what it looked like:

alphacenbb

Dumusque et al. (2012), Figure 5

All those scattered little dots that seem almost random—that’s the post-analysis data. But the red dots are what you get when you group data points that are close together and average them. That’s how the team was able to recover their signal. They reported that the odds that the data in the plot could have been a fluke of nature (a statistic called the False Alarm Probability, or FAP) were pretty slim: one in a thousand.

This was a key point that many journalists picked up on and quoted the authors repeating it in a press conference to bolster the case for the planet. To wit:

Mike Wall at Space.com: “Udry, however, said that the team’s statistical analyses show a ‘false alarm probability’ of just one in 1,000 — meaning there’s a 99.9 percent chance that the planet exists.”

Ian Sample in The Guardian: “The astronomers told a press briefing that the chance of their discovery being false was about one in 1,000…”

And Camille Carlisle in Sky & Telescope: “Study coauthor Stéphane Udry (Geneva Observatory) noted in a press conference earlier this week that there is one chance in 1,000 that the signal his team sees is a fluke.”

Well, that sounded like pretty good odds to me. That is, until early this summer when exoplanet astronomer Artie Hatzes published a paper in which he did his own analysis of the same data, and found nothing. In fact, he concluded that if you assumed the planet was there, he should have found it with a confidence of 99%.

So hang on a second. According to the Geneva team, they have only a 1/1000 chance of being wrong. But Hatzes finds the opposite, and says there’s only a 1/100 chance that Geneva are right. So who’s “correct”? What do those numbers even mean?

So I asked Debra Fischer. Her answer confirmed my thinking. That False Alarm Probability of 1/1000? That’s the probability that the data in that plot is a fluke—but remember, that’s the data after all of their analysis. In other words, the 1/1000 figure holds only if you assume that their analysis of those 23 parameters is absolutely perfect. It’s a comparison of the signal against the flukey nature of reality, but says nothing about the confidence in the analysis that led to that signal in the first place!

Yikes. That’s a difference with a big distinction, and one that got very little play in the media. (And it’s a point I didn’t call out when I wrote about Hatzes paper for Sky & Telescope.)

Now, that doesn’t mean the analysis is junk. Dumusque and his team weren’t trying to hide anything about their analysis—quite the opposite, in fact. They released their data publicly, inviting scrutiny; that’s what enabled Hatzes to do his independent analysis. And Dusmusqe’s team did a check of their analysis as part of their original study to see if it might introduce a false signal and concluded it did not. So Alpha Centauri Bb is not dead—not by a long shot. Both Dumusque and Fischer are currently analyzing fresh observations to try to get that slam-dunk confirmation. (Peter Edmonds has written an excellent blog post taking a look at the whole saga.)

But it does mean that it’s difficult to quantify how convincing the data are as they stand, and that the FAP is not the entire story. For a journalist, that is difficult to explain to the public. It’s yet another example of how tricky it can be to communicate probability and uncertainty—both from scientists to journalists, and from journalists to the public. That False Alarm Probability might be alluringly small, but we better make sure we know what it means.

Now, this may seem like an esoteric case. Alpha Centauri Bb winking out of existence would be a big disappointment, but not, say, hazardous to anyone’s health. But it’s not hard to see how the latter case is problematic. Perhaps the biggest shift wrought by our era of Big Data isn’t the sheer amount of data but that the nature of reality and our predictions of the future are increasingly described in probabilistic terms—in everything from election results to climate change. When we communicate this, we all have to work hard to get it right.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s