10
$\begingroup$

I work in the analysis of biological data, and am familiar with the use of maths to determine a test statistic, and the comparison of this test statistic with a threshold to evaluate a hypothesis. Using frequentist frameworks with $p$ values and using Bayesian frameworks with posterior probabilities are both common. These test statistics can be very small, in the order of magnitude of $10^{-6}$, such that talking about the exponent is most convenient. This is particularly the case in situations where multiple testing occurs, and one is discussing nominal rather than corrected test statistics.

In physics, in the popular-science media at least, the significance threshold is usually talked about in terms of $σ$ with experimental results that reach the five or six $σ$ threshold being generally accepted. My understanding is this is how many observed standard deviations the test statistics is from the value predicted by the null hypothesis.

Under a frequentist framework, assuming a normally distributed test statistic, the $p$ value and $σ$ are directly related. In R this would be p_value = pnorm(-sigma) with five $σ$ corresponding roughly to $3·10^{-7}$ and six to $10^{-9}$.

What is the reason for this difference?

The biggest downside of this method would seem to be the problem of skewed distributions, where a test statistic is more likely to be within a certain number of standard deviations on one side of the mean than the other, but I am sure they have thought of that. There could be a similar issue with heavy tailed distributions.

It also makes me wonder how Bayesian maths could be used, and it is worth nothing that the physics papers a quick google brings up do not talk about sigma, and prominently include the intersection with biology.

It is worth noting that the generally accepted thresholds are very different, with 5% being common in many biological fields compared with the 0.00003% of five sigma. This may well be related, but does not seem to answer the question, as one can just talk about the exponent and the numbers are similar. As to why it is so strict is covered in this question.

As an illustration of what I mean, take the "Higgs Bump", that showed the existence of the Higgs Bosun. Obviously the analysis is complex, but taking the simple case of a single bin this is the number of events per GeV with a mean of around 1000. At this number of events this distribution is well modelled by the normal approximation, and observed counts would be expected to by symmetrically distributed around the mean, and sigma would behave very similar to a p value based on say the poisson distribution.

If these expected values were closer to 10 this would not be true. Using a sigma based threshold under the null one would be more likely to get a false rejection due to high counts than low counts (I think).

This seems a negative feature of the analysis method, that a p value or posterior probability based threshold does not have. As the methods are equivalent when the assumptions are met, it would seem the sigma based method has only disadvantages.

The Higgs Bump

$\endgroup$
11
  • 3
    $\begingroup$ Regarding skewed distributions, note that experimental particle physics (the field in which you're probably most likely to hear about 5-sigma in popular treatments) tend to have very large data sets, so the law of large numbers and central limit theorems are often assumed to apply. Systematic errors is typically a bigger concern (of physicists, anyway). $\endgroup$
    – Anyon
    May 15 at 12:42
  • $\begingroup$ @Anyon "particle physics tend to have very large data sets, so the ... central limit theorems are often assumed to apply" it is not clear why a large number of individual observations of particles could be assumed to follow the CLT. For that to apply each particle's value would have to be the sum of a large number of individual variables, with is not my understanding of physics. $\endgroup$
    – User65535
    May 15 at 12:46
  • 8
    $\begingroup$ @User65535: but test statistics are usually computed not on the level of individual observations, but based on large aggregates of observations, and that is where we hope the CLT to kick in. $\endgroup$ May 15 at 12:49
  • 1
    $\begingroup$ The difference in thresholds is due to multiple testing. The physicist says you need 5 sigma due to the "look elsewhere effect". The biologist says the corrected p-value has to be less than 0.05 - the physicist changes the threshold, the biologist adjusts the p-value, but they amount to the same thing. $\endgroup$ May 15 at 12:57
  • 3
    $\begingroup$ My suspicion is that while physics experiments are costly to set up, measuring more data does not cost a lot, because there are a lot of particles floating around. In biology, medicine etc., test subjects are more expensive, or we may have ethical concerns about killing lots of mice. So achieving high precision and high power is comparatively cheaper in physics than in the life sciences. But that is all POOMA from someone whose physics were a long time ago. $\endgroup$ May 15 at 13:24

2 Answers 2

5
$\begingroup$

With physics (and also chemistry) one often has some measurements with many data points that follow a Poisson distribution. E.g. mass spectra, photography in low light conditions, and time series (time series may also appear in many other fields).

This gives rise to graphs like below where some observations are compared with a baseline or with a theoretical model, And either error bars or some region for standard error are used to allow an easy graphical evaluation of the relative effect size.

I hope that the below image shows that it is very natural to express observed features, like peaks, in terms of the noise level. This is the signal to noise ratio and can be expressed in terms of the standard deviation of the signal, the $\sigma$.

The idea of signal to noise level is practical because it is easy to compute (just measure the peak size and divide by the noise level), and it is easy to see how it will change as we increase the integration time (exposure time). Below is an example by having the second image with 4 times more counts, which makes the noise level reduce by two.

So computations with these noise levels are easy and you can quickly express the accuracy of measurements and make calculations like power of the experiment and requirements for the experiment.

some toy example of a measurement with noise

The actual level, $3, 4, 5$ or $6\,\sigma$ that is in the end a rule of thumb based on experience with practical levels. The $5\sigma$ has evolved over time as explained here: Origin of "5$\sigma$" threshold for accepting evidence in particle physics?

The same is true for the 0.05 p-value that is often used. This was also a practical value and is actually used because it relates to $2\sigma$ What is the history of $p < 0.05$ or 95% confidence?

$\endgroup$
4
$\begingroup$

Firstly, as the Q. already points out, there is very straightforward relation between the p-values (for data that can be treated as normally distributed) and the standard deviation. Up to three sigma this is known as 68–95–99.7 rule:
enter image description here

In this sense, speaking of the number of sigmas or citing the p-value are two equivalent ways of stating the same thing - using one or the other is determined by tradition.

Regarding the size of the typical p-values: in biology and many other fields of science the significance thresholds are usually is often taken to be 0.05 (more rarely 0.01, see, e.g., here), so that there sigma is quite sufficient. In particle physics one deals with extremely rare events, which make sit necessary to consider much higher levels of precision. Here is a quote from an article at CERN site, with telling title Why do physicists mention “five sigma” in their results?:

In most areas of science that use statistical analysis, the five-sigma threshold seems overkill. In a population study, such as polls for how people will vote, usually a result with three sigma statistical significance would suffice. However, when discussing the very fabric of the Universe, scientists aim to be as precise as possible. The results of the fundamental nature of matter are high impact and have significant repercussions if they are wrong.

Remarks

Update
Following the discussion in the comments: the OP seems to put the principal emphasis on why particle physicists rely heavily on the normal distribution (which is inherent 5 sigma language): the reason is that experiments in particle physics usually involve very large numbers of particles (billions or trillions of protons/electrons/neutrons/etc.), which means that the conditions for the central limit theorem are fulfilled with high precision.

$\endgroup$
17
  • 1
    $\begingroup$ This does not seem to answer the question. I point out that they are the same under the frequentist model and the normal null hypothesis, that goes nowhere to explaining why one choice is made in physics and another in biology. I could also point out that it may take a physicist to think there answers are more important. Obligatory XKCD. $\endgroup$
    – User65535
    May 15 at 13:09
  • $\begingroup$ From your linked question, i guess "the chapter on statistical testing published by the Particle Data Group" probably has the answer, but it is not short. $\endgroup$
    – User65535
    May 15 at 13:16
  • 12
    $\begingroup$ "The results of the fundamental nature of matter are high impact and have significant repercussions if they are wrong." - only a physicist could have written that ;-) Results from economics or medical studies can have repercussions that the normal person might argue could be even stronger than those from physics... $\endgroup$ May 15 at 13:21
  • 8
    $\begingroup$ Yes, and we can afford to monitor a billion protons, so we get that five sigma event when one of them does something against the null hypothesis. We can't afford to monitor a billion humans for a similarly rare occurrence. Plus: protons are simple, humans (or mice) are complex, so such a rare occurrence does not give us as much information in humans as it does in protons. $\endgroup$ May 15 at 13:40
  • 4
    $\begingroup$ The reason for $5\sigma$ or other high standards in (particle) physics is not really 'high impact and have significant repercussions if they are wrong'. There are more considerations discussed here: Origin of "5 σ 𝜎 " threshold for accepting evidence in particle physics?. The reasons are: 1) historic/experience (what one found to be a practical cutoff), 2) the look elsewhere effect, 3) systematic uncertainty in $\sigma$, 4) Extraordinary claims require extraordinary evidence. The latter one is subtly different from high impact. $\endgroup$ May 15 at 13:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.