Why do physicists use sigma while biologists use p values/posterior probabilities?

Question

I work in the analysis of biological data, and am familiar with the use of maths to determine a test statistic, and the comparison of this test statistic with a threshold to evaluate a hypothesis. Using frequentist frameworks with $p$ values and using Bayesian frameworks with posterior probabilities are both common. These test statistics can be very small, in the order of magnitude of $10^{-6}$, such that talking about the exponent is most convenient. This is particularly the case in situations where multiple testing occurs, and one is discussing nominal rather than corrected test statistics.

In physics, in the popular-science media at least, the significance threshold is usually talked about in terms of $σ$ with experimental results that reach the five or six $σ$ threshold being generally accepted. My understanding is this is how many observed standard deviations the test statistics is from the value predicted by the null hypothesis.

Under a frequentist framework, assuming a normally distributed test statistic, the $p$ value and $σ$ are directly related. In R this would be p_value = pnorm(-sigma) with five $σ$ corresponding roughly to $3·10^{-7}$ and six to $10^{-9}$.

What is the reason for this difference?

The biggest downside of this method seems to be the problem of skewed distributions, where a test statistic is more likely to be within a certain number of standard deviations on one side of the mean than the other, but I am sure they have thought of that. There could be a similar issue with heavy tailed distributions.

It also makes me wonder how Bayesian maths could be used, and it is worth nothing that the physics papers a quick google brings up do not talk about sigma, and prominently include the intersection with biology.

It is worth noting that the generally accepted thresholds are very different, with 5% being common in many biological fields compared with the 0.00003% of five sigma. This may well be related, but does not seem to answer the question, as one can just talk about the exponent and the numbers are similar. As to why it is so strict is covered in this question.

As an illustration of what I mean, take the "Higgs Bump", that showed the existence of the Higgs boson. Obviously the analysis is complex, but taking the simple case of a single bin this is the number of events per GeV with a mean of around 1000. At this number of events this distribution is well modelled by the normal approximation, and observed counts would be expected to by symmetrically distributed around the mean, and sigma would behave very similar to a p value based on say the Poisson distribution.

If these expected values were closer to ten, this would not be true. Using a sigma-based threshold under the null, one would be more likely to get a false rejection due to high counts than low counts (I think).

This seems a negative feature of the analysis method, that a p value or posterior probability based threshold does not have. As the methods are equivalent when the assumptions are met, it would seem the sigma based method has only disadvantages.

Answer 1

With physics (and also chemistry) one often has some measurements with many data points that follow a Poisson distribution. E.g. mass spectra, photography in low light conditions, and time series (time series may also appear in many other fields).

This gives rise to graphs like below where some observations are compared with a baseline or with a theoretical model, And either error bars or some region for standard error are used to allow an easy graphical evaluation of the relative effect size.

I hope that the below image shows that it is very natural to express observed features, like peaks, in terms of the noise level. This is the signal to noise ratio and can be expressed in terms of the standard deviation of the signal, the $\sigma$.

The idea of signal to noise level is practical because it is easy to compute (just measure the peak size and divide by the noise level), and it is easy to see how it will change as we increase the integration time (exposure time). Below is an example by having the second image with 4 times more counts, which makes the noise level reduce by two.

So computations with these noise levels are easy and you can quickly express the accuracy of measurements and make calculations like power of the experiment and requirements for the experiment.

The actual level, $3, 4, 5$ or $6\,\sigma$ that is in the end a rule of thumb based on experience with practical levels. The $5\sigma$ has evolved over time as explained here: Origin of "5$\sigma$" threshold for accepting evidence in particle physics?

The same is true for the 0.05 p-value that is often used. This was also a practical value and is actually used because it relates to $2\sigma$ What is the history of $p < 0.05$ or 95% confidence?

Answer 2

Firstly, as the Q. already points out, there is very straightforward relation between the p-values (for data that can be treated as normally distributed) and the standard deviation. Up to three sigma this is known as 68–95–99.7 rule:

In this sense, speaking of the number of sigmas or citing the p-value are two equivalent ways of stating the same thing - using one or the other is determined by tradition.

Regarding the size of the typical p-values: in biology and many other fields of science the significance thresholds are usually is often taken to be 0.05 (more rarely 0.01, see, e.g., here), so that there sigma is quite sufficient. In particle physics one deals with extremely rare events, which make sit necessary to consider much higher levels of precision. Here is a quote from an article at CERN site, with telling title Why do physicists mention “five sigma” in their results?:

In most areas of science that use statistical analysis, the five-sigma threshold seems overkill. In a population study, such as polls for how people will vote, usually a result with three sigma statistical significance would suffice. However, when discussing the very fabric of the Universe, scientists aim to be as precise as possible. The results of the fundamental nature of matter are high impact and have significant repercussions if they are wrong.

Remarks

• Related discussion in Physics SE community: Help interpreting "five sigma" standard?
• six sigma is also the popular standard in industrial production quality control

Update
Following the discussion in the comments: the OP seems to put the principal emphasis on why particle physicists rely heavily on the normal distribution (which is inherent 5 sigma language): the reason is that experiments in particle physics usually involve very large numbers of particles (billions or trillions of protons/electrons/neutrons/etc.), which means that the conditions for the central limit theorem are fulfilled with high precision.

Answer 3

Background, for reference: I am physicist by training, specialised on complex systems, and now work on biological physics and data analysis.

I will use two cases for illustration:

• A biologist studies the effect of an antibiotic on some bacterium under some condition. He roughly determines the order of magnitude of the concentration at which the antibiotic stops the bacteria from growing. He checks that the bacteria grow significantly less in presence of the antibiotic than without.

• A physicist studies the temperature dependence of the electrical resistance of some metal. She provides a detailed curve of measured resistances in a certain temperature interval. From these, she computes the temperature coefficient with three significant digits and a relative confidence interval of about one percent. (If you look at the curve, there is no doubt that there is a significant dependence of resistance on temperature, but this is not investigated.)

These examples illustrate some fundamental differences of the fields (and how they relate to your question):

• Physics can be and is considerably more quantitative than biology. The typical result in experimental physics is a number with a confidence interval. The typical biology result is a statement with a p value.

  Now, if you are used to thinking of error margins, σ is only a small extension of this, whereas p values and similar are a completely new world.

• Physics experiments can easily provide a lot of data. Tools and techniques to precisely measure temperature and resistance have existed for a long time. Once the experiment is set up, taking a considerable amount of data is easy. The main challenge is to set up the experiment in the first place (think of the LHC) and to avoid systematic biases, e.g., heating our lump of metal homogeneously and measuring its temperature and not something else’s. By contrast, in most of biology, each data point reflects a considerable amount of work.

  As a consequence, physicists rarely face statistical challenges, i.e., cases where there is reason to doubt whether some effect is just due to statistical fluctuations. Therefore, there is no incentive for them to engage with more sophisticated statistics.

• Physics tends to produce broadly applicable but also basic results. If our physicist in the example works diligently, she will obtain values that hold everywhere in the universe at every time. There are some constraints regarding the purity of the material, but that’s about it. The application of the result however is rather indirect. There is a tiny chance that some engineering application will require the resistance of that particular metal in that particular temperature range. But most of the relevance is rather achieved from an interaction with theory yielding a general understanding of how metals, heat resistance, etc. work, or confirming quantitative predictions (which is hardly done in biology, if ever).

  Results in biology, on the other hand, are almost inevitably specific. For example, our biologist’s finding only holds for a particular strain under particular conditions. To make a general statement about the effectiveness of the antibiotic, he would have to study several strains, conditions, etc. On top, all this research may only lay the foundation for in-vivo experiments and eventually clinical trials. As a result, for many biological experiments there is little point to conduct them to the point where they become as quantitative as your typical physics experiment and statistics becomes simple. Nobody cares about knowing three significant digits of the growth rate of some bacterial strain under some conditions, even if we could obtain it.

  Another consequence is that the handling of false positives is different: If our biologist’s results turns out to be of interest in application, further studies will be conducted that can uncover false positives. Since that study will likely be conducted in different settings, it will automatically avoid certain systematic mistakes. In physics, this is rather unlikely due to the aforementioned indirectness of application. Moreover, for things like the Higgs boson, we cannot easily build another LHC to double-check the result. This explains different standards of significance to some extent (the other part being to address multiple testing as detailed e.g., by this answer).

• Statistical fluctuations in physics are mostly benign, not only in magnitude but also in statistical properties. For example, when measuring temperature and resistance, the main sources of statistical fluctuations are things like thermal noise. These can be reasonably assumed as normally distributed on account of being the sum of a lot of tiny effects. Their temporal correlation is also very low, such that pseudocorrelation and similar are no issues. On top, we can often easily measure the fluctuations themselves or even model them (see the example from Sextus Empiricus’ answer). For example, we can repeat our temperature and resistance measurement under fixed conditions umpteen times and see what happens.

  Biological fluctuations, however, are much more nasty. For example, in a bacterial growth experiment, if an individual bacterium has just the right metabolic setting for the current conditions right after the start, it may grow faster, dominate the population and considerably change the result. All of this may be caused in a highly non-trivial manner by a single mRNA molecule being produced. (And then there’s mutations.)

  At the end of the day, this is due to physics investigating simple phenomena from the bottom up. Usually, this means single molecules, particles, etc. or homogeneous systems (like our lump of metal). If something cannot be studied in such a way, but instead a more phenomenological, summarising approach is needed, it usually is subject to its own discipline, such as chemistry, biology, engineering, etc.

This is also reflected in education: The only statistical education most physicists receive is on how to determine errors (confidence intervals) and how they propagate, assuming normality. Most physicists are bad at statistics (but they are usually good at learning it), while biologists usually receive a training in basic statistics (but usually struggle with it).

So, to summarise: Most physicists use σ and similar because it suffices for their needs. They do not require more sophisticated statistics and are not used to its concepts and language. Thus even physicists performing thorough statistics may rather speak in terms of σ, because that is more likely to reach their physics audience.

Finally note that for some subsets of physics like astronomy, geophysics, biological physics (surprise), or the physics of complex systems, much of the above does not apply. However, there you are also much more likely to encounter p values and other more sophisticated statistics.