Origin of “5$\sigma$” threshold for accepting evidence in particle physics?

News reports say that CERN will announce tomorrow that the Higgs boson has been experimentally detected with 5$\sigma$ evidence. According to that article:

5$\sigma$ equates to a 99.99994% chance that the data the CMS and ATLAS detectors are seeing aren’t just random noise — and a 0.00006% chance that they’ve been hoodwinked; 5$\sigma$ is the necessary certainty for something to be officially labeled a scientific “discovery.”

This isn't super rigorous, but it seems to say that physicists use standard "hypothesis testing" statistical methodology, setting $\alpha$ to $0.0000006$, which corresponds to $z=5$ (two-tailed)? Or is there some other meaning?

In much of science, of course, setting alpha to 0.05 is done routinely. This would be equivalent to "two-$\sigma$" evidence, although I've never heard of it being called that. Are there other fields (besides particle physics) where a much stricter definition of alpha is standard? Anyone know a reference for how the five-$\sigma$ rule got accepted by particle physics?

Update: I'm asking this question for a simple reason. My book Intuitive Biostatistics (like most stats books) has a section that explains how arbitrary the usual "P<0.05" rule is. I'd like to add this example of a scientific field where a much (much!) smaller value of $\alpha$ is considered necessary. But if the example is actually more complicated, with use of Bayesian methods (as some comments below suggest), then it wouldn't be quite apt or would require a lot more explanation.

-
Ever heard of "Six Sigma"? –  Daniel R Hicks Jul 3 '12 at 19:56
In quality control six sigma is considered as Daniel suggests with his question/remark. These rejection probabilities all do assume sampling from a normal distribution and the tail probabilities could be larger for other distributions. Using such extremes like 5 or 6 sigma can only be useful in special circumstances. In practice the sample size and variability in the data make inference beyond 2 or 3 sigma infeasible. –  Michael Chernick Jul 3 '12 at 20:41
Basically, most particle physicists are more comfortable with bayesian ideas when calculating parameters, so they are actually "$X\%$ sure, given the data and the priors, that the signal of the Higgs is not zero", which is certainly different from saying that there is only "0.01 percent chance of the signal being random noise" (there are non-random fluctuations arising from systematics too!). [1]: physics.stackexchange.com/questions/8752/… –  Néstor Jul 4 '12 at 2:33
@Néstor: I'm watching the live broadcast of the Higgs press conference now, and no one is mentioning Bayesian interpretations. "p-values" and "significance level" are used, but only horribly misinformed Bayesian would interpret those as probabilities that the signal is random noise. I think that the text in the quote in the OP's question simply is a misinterpretation of what a p-value really are. –  MånsT Jul 4 '12 at 7:40
BTW I did a blog post on my blog about this issue: randomastronomy.wordpress.com. –  Néstor Jul 5 '12 at 2:18
show 6 more comments

In most applications of statistics there is that old chestnut about 'all models are wrong, some are useful'. This being the case, we would only expected a model to perform at a given level since we are describing some incredibly complicated process using some simple model.

Physics is very different, so intuition developed from statistical models isn't so appropriate. In Physics, in particular particle physics which deals directly with fundamental physical laws, the model really is supposed to be an exact description of reality. Any departure from what the model predicts must be completely explained by experimental noise, not a limitation of the model. This means that if the model is good and correct and the experimental apparatus understood the statistical significance should be very high, hence the high bar that is set.

The other reason is historical, the particle physics community has been burned in the past by 'discoveries' at lower significance levels being later retracted, hence they are generally more cautious now.

-
Do you agree that physics uses standard statistical hypothesis testing with a very low alpha (in this case, anyway). Or do they use some kind of Bayesian approach as Nestor said in a comment above? –  Harvey Motulsky Jul 4 '12 at 20:21
My understanding from talking to some of the people I know who work on ATLAS is that the analysis is all very Bayesian. However they are lower level guys (i.e. the ones who actually do the work). It wouldn't surprise me if some of the talking heads higher up the chain had a poorer grasp of the interpretation. That being said, the presentation of the LHC results was pretty poor, and didn't really come across as very Bayesian, as others have noted. –  Bogdanovist Jul 4 '12 at 23:04
I've always thought that particle physics in particular also dealt with billions of events, so you have to set the bar very high. –  Wayne Jul 9 '12 at 1:43

For a reason entirely different from that of physics, there are other fields with much more strict alphas when they engage in hypothesis testing. Genetic Epidemiology is among them, especially when they use "GWAS" (Genome-Wide Association Study) to look at various genetic markers for disease.

Because a GWAS study is a massive exercise in multiple hypothesis testing, the state-of-the-art analysis techniques are all built around much more strict alphas than 0.05. Other such "candidate screening" study techniques that follow in the wake of the genomics studies will likely do the same.

-