27
$\begingroup$

I think this is a fascinating topic and I do not fully understand it. What law of physics makes so that so many natural phenomena have normal distribution? It would seem more intuitive that they would have uniform distribution.

It is so hard for me to understand this and I feel I am missing some information. Can somebody help me with a good explanation or link me to a book/video/article?

$\endgroup$
  • $\begingroup$ Check this. $\endgroup$ – Antoni Parellada Mar 30 '16 at 4:49
  • 7
    $\begingroup$ Do you have a solid reason to think that your premise is actually the case? $\endgroup$ – Glen_b Mar 30 '16 at 5:27
  • 3
    $\begingroup$ Actually, the normal distribution may not be the "dominant" distribution in nature. There are many phenomena and behaviors which are extreme valued, heavy-tailed or describe power law functions. Gabaix documents many of the economic and financial variants of this distributional class in his paper Power Laws in Economics: An Introduction, ungated here...pages.stern.nyu.edu/~xgabaix/papers/pl-jep.pdf Shalizi, et al. discuss their empirical estimation in this paper Power-Law Distributions in Empirical Data, ungated here... santafe.edu/media/workingpapers/07-12-049.pdf $\endgroup$ – DJohnson Apr 1 '16 at 13:03
  • $\begingroup$ Who told you the normal distribution was dominant anyway? $\endgroup$ – shadowtalker Apr 1 '16 at 13:19
  • 1
    $\begingroup$ @DJohnson +1 for the links, but it's important to point out that a key finding in the Clauset et al. paper is that there aren't too many empirical power law distributions which are strongly supported! Quoting the findings, "In just one case—the distribution of the frequencies of occurrence of words in English text—the power law appears to be truly convincing in the sense that it is an excellent fit to the data and none of the alternatives carries any weight." $\endgroup$ – Sycorax Apr 1 '16 at 20:25
29
$\begingroup$

Let me start by denying the premise. Robert Geary probably didn't overstate the case when he said (in 1947) "...normality is a myth; there never was, and never will be, a normal distribution." --
the normal distribution is a model*, an approximation that is sometimes more-or-less useful.

$\:$*(about which, see George Box, though I prefer the version on my profile).

That some phenomena are approximately normal may be no vast surprise, since sums of independent [or even not-too-strongly-correlated effects] should, if there a lot of them and none has a variance that is substantial compared to the variance of the sum of the rest that we might see the distribution tend to look more normal.

The central limit theorem (which is about the convergence to a normal distribution of a standardized sample mean as $n$ goes to infinity under some mild conditions) at least suggests that we might see a tendency toward that normality with sufficiently large but finite sample sizes.

Of course if standardized means are approximately normal, standardized sums will be; this is the reason for the "sum of many effects" reasoning. So if there are a lot of little contributions to the variation, and they're not highly correlated, you might tend to see it.

The Berry-Esseen theorem gives us a statement about it (convergence toward normal distributions) actually happening with standardized sample means for iid data (under slightly more stringent conditions than for the CLT, since it requires that the third absolute moment be finite), as well as telling us about how rapidly it happens. Subsequent versions of the theorem deal with non-identically distributed components in the sum, though the upper bounds on the deviation from normality are less tight.

Less formally, the behavior of convolutions with reasonably nice distributions gives us additional (though closely related) reasons to suspect it might tend to be a fair approximation in finite samples in many cases. Convolution acts as a kind of "smearing" operator that people who use kernel density estimation across a variety of kernels will be familiar with; once you standardize the result (so the variance remains constant each time you do such an operation), there's clear a progression toward increasingly symmetric hill shapes as you repeatedly smooth (and it doesn't much matter if you change the kernel each time).

Terry Tao gives some nice discussion of versions of the Central limit theorem and the Berry-Esseen theorem here, and along the way mentions an approach to a non-independent version of Berry-Esseen.

So there's at least one class of situations where we might expect to see it, and formal reasons to think it really will tend to happen in those situations. However, at best any sense that the result of "sums of many effects" will be normal is an approximation. In many cases it's quite a reasonable approximation (and in additional cases even though the approximation of the distribution isn't close, some procedures that assume normality aren't especially sensitive to the distribution of the individual values, at least in large samples).

There are many other circumstances where effects don't "add" and there we may expect other things to happen; for example, in a lot of financial data effects tend to be multiplicative (effects will move amounts in percentage terms, like interest and inflation and exchange rates for example). There we don't expect normality, but we might sometimes observe a rough approximation to normality on the log scale. In other situations neither can be appropriate, even in a rough sense. For example, inter-event times are generally not going to be well approximated by either normality or normality of logs; there's no "sums" nor "products" of effects to argue for here. There are numerous other phenomena that we can make some argument for a particular kind of "law" in particular circumstances.

$\endgroup$
  • 10
    $\begingroup$ +1. Your argument begins to suggest--quite plausibly, in my view--that there may be a psychological answer to the question, such as groupthink: when everybody in your field sees normal distributions, who are you to say otherwise? This would go especially for fields of inquiry where statistical procedures are viewed as pedestrian tools, necessary perhaps to sanctify a paper for publication, but otherwise of little inherent value or interest. $\endgroup$ – whuber Mar 30 '16 at 21:06
  • 1
    $\begingroup$ To give a specific example, when Quetelet invented BMI (Body Mass Index), he explicitly did so in a way that yielded a normally-distributed quantity. We talked about that here: stats.stackexchange.com/questions/64171/… $\endgroup$ – Matt Krause Mar 31 '16 at 0:21
  • $\begingroup$ It seems to me that everyone is trying to sidestep this question rather than answering it. $\endgroup$ – Digio Jan 8 at 9:45
  • $\begingroup$ Geary was a statistician, so it's no wonder he thought normality is a myth. If he were a physicist he'd see it differently. $\endgroup$ – Aksakal Apr 16 at 18:03
  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$ – Glen_b Jun 7 at 11:04
15
$\begingroup$

There is a famous saying by Gabriel Lippmann (physicist, Nobel laureate), as told by Poincaré:

[The normal distribution] cannot be obtained by rigorous deductions. Several of its putative proofs are awful [...]. Nonetheless, everyone believes it, as M. Lippmann told me one day, because experimenters imagine it to be a mathematical theorem, while mathematicians imagine it to be an experimental fact.

-- Henri Poincaré, Le calcul des Probabilités. 1896

[Cette loi] ne s’obtient pas par des déductions rigoureuses; plus d’une démonstration qu’on a voulu en donner est grossière [...]. Tout le monde y croit cependant, me disait un jour M. Lippmann, car les expérimentateurs s’imaginent que c’est un théorème de mathématiques, et les mathématiciens que c’est un fait expérimental.

It seems that we don't have this quote in our List of Statistical Quotes thread, that's why I thought it would be good to post it here.

$\endgroup$
  • $\begingroup$ A downvote? Somebody here secretly hating Poincare? $\endgroup$ – amoeba Apr 1 '16 at 12:15
  • $\begingroup$ My physics knowledge ends with what I learned in high school, but didn't Gauss originally study the distribution in the context of the normal equations from physics? It was my Wikipedia-given impression that Gaussian errors fall naturally out of some classical physics model $\endgroup$ – shadowtalker Apr 1 '16 at 13:18
  • 2
    $\begingroup$ We should honour Lippmann more as the author of this bon mot. Gabriel Lippmann was a Nobel Prize winner in Physics. (M. here just means Monsieur, naturellement.) $\endgroup$ – Nick Cox Apr 1 '16 at 13:18
  • 3
    $\begingroup$ @ssdecontrol As I recall, Gauss was interested in normal errors of observation, particularly in astronomy and geodesy, but smart enough to know that the assumption was questionable. (He was also, e.g., using the median absolute deviation from the median as a resistant measure of spread in 1816.) $\endgroup$ – Nick Cox Apr 1 '16 at 13:20
  • $\begingroup$ Fair enough, @Nick. I edited to clarify. $\endgroup$ – amoeba Apr 1 '16 at 16:56
6
$\begingroup$

What law of physics makes so that so many natural phenomena have normal distribution? It would seem more intuitive that they would have uniform distribution.

The normal distribution is a common place in natural sciences. The usual explanation is why it happens in measurement errors is through some form of large numbers or central limit theorem (CLT) reasoning, which usually goes like this: "since the experiment outcomes are impacted by infinitely large number of disturbances coming from unrelated sources CLT suggests that the errors would normally distributed". For instance, here's an excerpt from Statistical Methods in Data Analysis by W. J. Metzger:

Most of what we measure is in fact the sum of many r.v.’s. For example, you measure the length of a table with a ruler. The length you measure depends on a lot of small effects: optical parallax, calibration of the ruler, temperature, your shaking hand, etc. A digital meter has electronic noise at various places in its circuitry. Thus, what you measure is not only what you want to measure, but added to it a large number of (hopefully) small contributions. If this number of small contributions is large the C.L.T. tells us that their total sum is Gaussian distributed. This is often the case and is the reason resolution functions are usually Gaussian.

However, as you must know this doesn't mean that every distribution will be normal, of course. For instance, Poisson distribution is as common in physics when dealing with counting processes. In spectroscopy Cauchy (aka Breit Wigner) distribution is used to describe the shape of radiation spectra and so on.

I realized this after writing: all three distributions mentioned so far (Gaussian, Poisson, Cauchy) are stable distributions, with Poisson being discrete stable. Now that I thought about this, it seems an important quality of a distribution that will make it survive aggregations: if you add a bunch of numbers from Poisson, the sum is a Poisson. This may "explain" (in some sense) why it's so ubiquitous.

In unnatural sciences you have to be very careful with applying normal (or any other) distribution for a variety of reasons. Particularly the correlations and dependencies are an issue, because they may break the assumptions of CLT. For instance, in finance it's well known that many series look like normal but have much heavier tails, which is a big issue in risk management.

Finally, there are more solid reasons in natural sciences for having normal distribution than sort of "hand waving" reasoning that I cited earlier. Consider, Brownian motion. If the shocks are truly independent and infinitesimal, then inevitably the distribution of an observable path will have normal distribution due to CLT, see e.g. Eq.(10) in Einstein's famous work "INVESTIGATIONS ON THE THEORY OF THE BROWNIAN MOVEMENT". He didn't even bother to call it by its today's name "Gaussian" or "normal".

Another example is quantum mechanics. It so happens that if the uncertainty of a coordinate $\Delta x$ and moment $\Delta p$ are from normal distributions, then the total uncertainty $\Delta x \Delta p$ reaches the minimum, Heisenberg's uncertainty threshold, see Eq.235-237 here.

Hence, don't be surprised to get very different reactions to Gaussian distribution use from researchers in different fields. In some fields such as physics, certain phenomena are expected to be linked naturally to Gaussian distribution based on very solid theory backed by enormous amount of observations. In other fields, Normal distribution is used for its technical convenience, handy mathematical properties or other questionable reasons.

$\endgroup$
  • 1
    $\begingroup$ +1. The quote is reasonable, however one can note that the measured length cannot be negative (i.e. is bounded) so cannot really follow a normal distribution. It is always an approximation. $\endgroup$ – amoeba Apr 1 '16 at 19:24
  • $\begingroup$ Unnatural sciences? You mean like Dr. Frankenstein's unseemly experiments? ;-) $\endgroup$ – Sycorax Apr 1 '16 at 19:38
  • 1
    $\begingroup$ @user777, it's Nobel laureate Landau's joke: "sciences can be divided into three types: natural, unnatural and antinatural" $\endgroup$ – Aksakal Apr 1 '16 at 19:46
  • $\begingroup$ @Aksakal: I think this particular link has it wrong; Landau said that sciences are divided into "естественные, неестественные и противоестественные" (instead of "сверхъестественные"). No idea how to translate it though. $\endgroup$ – amoeba Apr 1 '16 at 20:44
  • $\begingroup$ @amoeba, I'm translating "неестественные" as "unnatural". "сверхъестественные" is "supernatural", me thinks. Maybe Russians can correct me. $\endgroup$ – Aksakal Apr 1 '16 at 20:52
1
$\begingroup$

What law of physics makes so that so many natural phenomena have normal distribution?

No idea. On the other hand I've also no idea whether it's true, or indeed what 'so many' means.

However, rearranging the problem a little, there is good reason to assume (that is, to model) a continuous quantity that you believe to have a fixed mean and variance with a Normal distribution. That's because the Normal distribution is the result of maximizing entropy subject to those moment constraints. Since, roughly speaking, entropy is a measure of uncertainty, that makes the Normal the most non-commital or maximally uncertain choice of distributional form.

Now, the idea that one should choose a distribution by maximizing its entropy subject to known constraints really does have some physics backing in terms of the number of possible ways to fulfill them. Jaynes on statistical mechanics is the standard reference here.

Note that while maximum entropy motivates Normal distributions in this case, different sorts of constraints can be shown to lead to different distributional families, e.g. the familiar exponential, poisson, binomial, etc.

Sivia and Skilling 2005 ch.5 has an intuitive discussion.

$\endgroup$
1
$\begingroup$

there is an awful lot of overly complicated explanations here...

A good way it was related to me is the following:

  1. Roll a single die, and you have an equal likelihood of rolling each number (1-6), and hence, the PDF is constant.

  2. Roll two dice and sum the results together, and the PDF is no longer constant. This is because there are 36 combinations, and the summative range is 2 to 12. The likelihood of a 2 is unique singular combination of 1 + 1 . The likelihood of a 12, is also unique in that it can only occur in a single combination of a 6 + 6. Now, looking at 7, there are multiple combinations, i.e. 3 + 4, 5 + 2, and 6 + 1 (and their reverse permutations). As you work away from the mid-value (i.e. 7), there are lesser combinations for 6 & 8 etc until you arrive at the singular combinations of 2 and 12. This example does not result in a clear normal distribution, but the more die you add, and the more samples you take, then the result will tend towards a normal distribution.

  3. Therefore, as you sum a range of independent variables subject to random variation (which each can have their own PDFs), the more the resulting output will tend to normality. This in Six Sigma terms give us what we call the 'Voice of the Process'. This is what we call the result of 'common-cause variation' of a system, and hence, if the output is tending towards normality, then we call this system 'in statistical process control'. Where the output is non-normal (skewed or shifted), then we say the system is subject to 'special cause variation' in which there has been some 'signal' that has biased the outcome in some way.

Hope that helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.