# Who first used/invented p-values?

I am attempting to write a series of blog posts on p-values and I thought it would be interesting to go back to where it all started - which appears to be Pearson's 1900 paper. If you are familiar with that paper, you'll remember that this covers goodness-of-fit testing.

Pearson is a bit loose with his language when it comes to p-values. He repeatedly uses the "odds" when describing how to interpret his p-value. For example, on p.168 when talking about the results of repeat rolls of 12 dice, he says "...which leads us to P=.0000016, or the odds are 62,499 to 1 against such a system of deviation on a random selection. With such odds it would be reasonable to conclude that the dice exhibit bias towards the higher points."

In this article, he refers to earlier work, including an 1891 book on least squares by Merriman.

But Pearson does lay out the calculus for p-values (w.r.t. chi-squared goodness of fit testing).

Was Pearson the first person to conceive of p-values? When I do a search on p-values, Fisher is mentioned - and his work was in the 1920s.

Edited: and a thank you for the mention of Laplace - he did not seem to address the null hypothesis (Pearson appears to do so implicitly, although he never used that term in his 1900 paper). Pearson looked at goodness of fit testing from: assuming the counts are derived from an unbiased process, what is the probability that the observed counts (and counts more deviant) arise from the assumed distribution?

His treatment of the probabilities/odds (he converts the probabilities to odds) suggests he is working with an implicit idea of the null hypothesis. Crucially, he also mentions that the probability arising from the x^2 value shows the odds "against a system of deviations as improbable or more improbable than this one" - language we recognise now - with respect to his calculated p-values.

Did Arbuthnot go that far?

Feel free to put your comments in as answers. It would be nice to see a discussion.

• This article suggests it was used by Laplace, which would set a lower bound: en.wikipedia.org/wiki/…
– user145002
Apr 28, 2018 at 9:54
• One might argue that Arbuthnot (1710) in An Argument for Divine Providence, taken from the Constant Regularity observed in the Births of both Sexes could perhaps count. He uses a coin model ('cross and pile'), and first calculates the probabilitiy of getting exactly as many heads as tails before pointing out that the "chances will take in some of the Terms next the middle one, and will lean to one side or the other. But it is very improbable (if mere Chance govern’d) that they would never reach as far as the Extremities"; we can see him coming close to a notion of a p-value there. Apr 29, 2018 at 7:34
• Curiously David in his list jstor.org/stable/2685564?seq=1#page_scan_tab_contents suggests that the term P-value was first used in 1943 by Deming in his book "Statistical adjustment of data". I know you are after the concept not the term but I found it interesting to see when the term finally appeared. Apr 29, 2018 at 14:36
• Who invented might be hard to find out. But who’s to blame for current use of p-values is certainly Fisher. Apr 30, 2018 at 5:45
• We define p-value as 'the probability to obtain a result within a certain range (often some value or more extreme for a calculated statistic such as used by Pearson in 1900) given that a certain hypothesis is right'? May 2, 2018 at 21:52

# Jacob Bernoulli (~1700) - John Arbuthnot (1710) - Nicolaus Bernoulli (1710s) - Abraham de Moivre (1718)

The case of Arbuthnot1 see explanation in note below, can also be read about in de Moivre's Doctrine of Chance (1718) from page 251-254 who extends this line of thinking further.

1. The normal approximation of a Bernoulli distribution, which helps to easily calculate probabilities for results being within or out a certain range. In the section before the example about Arbuthnot's case, de Moivre writes about his approximation (now called the Gaussian/normal distribution) for the Bernoulli distribution. This approximation allows to easily calculate a p-value (which Arbuthnot could not do).

2. Generalization of Arbuthnot's argument. He mentions that "this method of reasoning may also be usefully applied in some other very interesting inquiries". (which may give partial credit to de Moivre for seeing the general applicability of the argument)

• According to de Moivre, Jacob Bernoulli wrote about this problem in his Ars Conjectandi. De Moivre names this in English 'Assigning the limits within which, by the repetition of experiments, the probability of an event may approach indefinitely to a probability given', but the original text by Bernouilli is in Latin. I do not know sufficient Latin to be able to figure out if Bernoulli was writing about a concept like the p-value or more like the law of large numbers. Interesting to note is that Bernouilli claims to have had these ideas for 20 years (and also the work 1713 was published after his death 1705 so it seems to precede the date 1710 mentioned in the comments by @Glen_b for Arbuthnot).

• One source of inspiration for de Moivre was Nicolaus Bernouilli, who in 1712/1713 made the calculations for the probability of the number of boys being born is not less than 7037 and not bigger than 7363, when 14000 is the number of total born kids and the probability for a boy is 18/35.

(The numbers for this problem were based on 80 years of statistics for London. He wrote about this in letters to Pierre Raymond de Montmort published in the second edition (1713) of Montmort's Essay d'analyse sur les jeux de hazard.)

The calculations, which I did not quite follow, turned out a probability of 43.58 to 1. (Using a computer summing all terms probability of a binomial from 7037 up to 7363 I get 175:1 so I may have misinterpreted his work/calculation.)

1: John Arbuthnot wrote about this case in An argument for divine providence, taken from the constant regularity observed in the births of both sexes (1710).

Explanation of Arbuthnot's argument: the boy:girl birth ratio is remarkably different from the middle. He does not calculate exactly the p-value (which is not his goal), but uses the probability to get boys>girls 82 times in a row $$\frac{1}{2}^{82} \sim \frac{1}{4 \,8360\,0000\,0000\,0000\,0000\,0000}$$ arguing that this number would be even more small when you would consider that one could take a smaller range and that it happened more than in just London and 82 years, he ends up at the conclusion that it is very unlikely and that this must be some (divine) providence to counter the greater mortality among men to finally end up with equal men and women.

Arbuthnot: then A’s Chance will be near an infinitely small Quantity, at least less than any assignable Fraction. From whence it follows that it is Art, not Chance that governs.

• Possibly A History of Inverse Probability by Andrew Dale may help out more. (In search for translation of Bernoulli I found he did translate the relevant passage mentioned by de Moivre). In the early days this inverse probability, now considered more a Bayesian technique, might have been quite a frequentist tool in interpretation and use. May 3, 2018 at 0:39

I have three supporting links/arguments that support the date ~1600-1650 for formally developed statistics and much earlier for simply the usage of probabilities.

If you accept hypothesis testing as the basis, predating probability, then the Online Etymology Dictionary offers this:

"hypothesis (n.)

1590s, "a particular statement;" 1650s, "a proposition, assumed and taken for granted, used as a premise," from Middle French hypothese and directly from Late Latin hypothesis, from Greek hypothesis "base, groundwork, foundation," hence in extended use "basis of an argument, supposition," literally "a placing under," from hypo- "under" (see hypo-) + thesis "a placing, proposition" (from reduplicated form of PIE root *dhe- "to set, put"). A term in logic; narrower scientific sense is from 1640s.".

Wiktionary offers:

"Recorded since 1596, from Middle French hypothese, from Late Latin hypothesis, from Ancient Greek ὑπόθεσις (hupóthesis, “base, basis of an argument, supposition”), literally “a placing under”, itself from ὑποτίθημι (hupotíthēmi, “I set before, suggest”), from ὑπό (hupó, “below”) + τίθημι (títhēmi, “I put, place”).

Noun hypothesis (plural hypotheses)

(sciences) Used loosely, a tentative conjecture explaining an observation, phenomenon or scientific problem that can be tested by further observation, investigation and/or experimentation. As a scientific term of art, see the attached quotation. Compare to theory, and quotation given there. quotations ▲

• 2005, Ronald H. Pine, http://www.csicop.org/specialarticles/show/intelligent_design_or_no_model_creationism, 15 October 2005:

Far too many of us have been taught in school that a scientist, in the course of trying to figure something out, will first come up with a "hypothesis" (a guess or surmise—not necessarily even an "educated" guess). ... [But t]he word "hypothesis" should be used, in science, exclusively for a reasoned, sensible, knowledge-informed explanation for why some phenomenon exists or occurs. An hypothesis can be as yet untested; can have already been tested; may have been falsified; may have not yet been falsified, although tested; or may have been tested in a myriad of ways countless times without being falsified; and it may come to be universally accepted by the scientific community. An understanding of the word "hypothesis," as used in science, requires a grasp of the principles underlying Occam's Razor and Karl Popper's thought in regard to "falsifiability" — including the notion that any respectable scientific hypothesis must, in principle, be "capable of" being proven wrong (if it should, in fact, just happen to be wrong), but none can ever be proved to be true. One aspect of a proper understanding of the word "hypothesis," as used in science, is that only a vanishingly small percentage of hypotheses could ever potentially become a theory.".

On probability and statistics Wikipedia offers:

"Data collection

Sampling

When full census data cannot be collected, statisticians collect sample data by developing specific experiment designs and survey samples. Statistics itself also provides tools for prediction and forecasting through statistical models. The idea of making inferences based on sampled data began around the mid-1600's in connection with estimating populations and developing precursors of life insurance. (Reference: Wolfram, Stephen (2002). A New Kind of Science. Wolfram Media, Inc. p. 1082. ISBN 1-57955-008-8).

To use a sample as a guide to an entire population, it is important that it truly represents the overall population. Representative sampling assures that inferences and conclusions can safely extend from the sample to the population as a whole. A major problem lies in determining the extent that the sample chosen is actually representative. Statistics offers methods to estimate and correct for any bias within the sample and data collection procedures. There are also methods of experimental design for experiments that can lessen these issues at the outset of a study, strengthening its capability to discern truths about the population.

Sampling theory is part of the mathematical discipline of probability theory. Probability is used in mathematical statistics to study the sampling distributions of sample statistics and, more generally, the properties of statistical procedures. The use of any statistical method is valid when the system or population under consideration satisfies the assumptions of the method. The difference in point of view between classic probability theory and sampling theory is, roughly, that probability theory starts from the given parameters of a total population to deduce probabilities that pertain to samples. Statistical inference, however, moves in the opposite direction — inductively inferring from samples to the parameters of a larger or total population.

From "Wolfram, Stephen (2002). A New Kind of Science. Wolfram Media, Inc. p. 1082.":

"Statistical Analysis

• History. Some computations of odds for games of chance were already made in antiquity. Beginning around the 1200s increasingly elaborate results based on the combinatorial enumeration of probabilities were obtained by mystics and mathematicians, with systematically correct methods being developed in the mid-1600s and early 1700s. The idea of making inferences from sampled data arose in the mid-1600s in connection with estimating populations and developing precursors of life insurance. The method of averaging to correct for what were assumed to be random errors of observation began to be used, primarily in astronomy, in the mid-1700s, while least squares fitting and the notion of probability distributions became established around 1800. Probabilistic models based on random variations between individuals began to be used in biology in the mid-1800s, and many of the classical methods now used for statistical analysis were developed in the late 1800s and early 1900s in the context of agricultural research. In physics fundamentally probabilistic models were central to the introduction of statistical mechanics in the late 1800s and quantum mechanics in the early 1900s. Beginning as early as the 1700s, the foundations of statistical analysis have been vigorously debated, with a succession of fairly specific approaches being claimed as the only ones capable of drawing unbiased conclusions from data.".

Other sources:

"This report, in mainly non-mathematical terms, defines the p value, summarizes the historical origins of the p value approach to hypothesis testing, describes various applications of p≤0.05 in the context of clinical research, and discusses the emergence of p≤5×10−8 and other values as thresholds for genomic statistical analyses."

The section "Historical origins" states:

"Published work on using concepts of probability for comparing data to a scientific hypothesis can be traced back for centuries. In the early 1700s, for example, the physician John Arbuthnot analyzed data on christenings in London during the years 1629–1710 and observed that the number of male births exceeded female births in each of the years studied. He reported$^{}$ that if one assumes a balance of male and female births is based on chance, then the probability of observing an excess of males over 82 consecutive years is 0.582=2×10−25, or less than a one in a septillion (ie, one in a trillion-trillion) chance.

. Arbuthnott J. An argument for divine Providence, taken from the constant regularity observ'd in the births of both sexes. Phil Trans 1710;27:186–90. doi:10.1098/rstl.1710.0011 published 1 January 1710

"P-values have long linked medicine and statistics. John Arbuthnot and Daniel Bernoulli were both physicians, in addition to being mathematicians, and their analyses of sex ratios at birth (Arbuthnot) and inclination of the planets’ orbits (Bernoulli) provide the two most famous early examples of significance tests $^{1–4}$. If their ubiquity in medical journals is the standard by which they are judged, P-values are also extremely popular with the medical profession. On the other hand, they are subject to regular criticism from statisticians $^{5–7}$ and only reluctantly defended $^8$. For example, a dozen years ago, the prominent biostatisticians, the late Martin Gardner and Doug Altman $^9$, together with other colleagues, mounted a successful campaign to persuade the British Medical Journal to place less emphasis on P-values and more on confidence intervals. The journal Epidemiology has banned them altogether. Recently, attacks have even appeared in the popular press $^{10,11}$. P-values thus seem to be an appropriate subject for the Journal of Epidemiology and Biostatistics. This essay represents a personal view of what, if anything, may be said to defend them.

I shall offer a limited defence of P-values only. ...".

References

1 Hald A. A history of probability and statistics and their appli- cations before 1750. New York: Wiley, 1990.
2 Shoesmith E, Arbuthnot, J. In: Johnson, NL, Kotz, S, editors. Leading personalities in statistical sciences. New York: Wiley, 1997:7–10.
3 Bernoulli, D. Sur le probleme propose pour la seconde fois par l’Acadamie Royale des Sciences de Paris. In: Speiser D,
editor. Die Werke von Daniel Bernoulli, Band 3, Basle:
Birkhauser Verlag, 1987:303–26.
4 Arbuthnot J. An argument for divine providence taken from
the constant regularity observ’d in the births of both sexes. Phil Trans R Soc 1710;27:186–90.
5 Freeman P. The role of P-values in analysing trial results. Statist Med 1993;12:1443 –52.
6 Anscombe FJ. The summarizing of clinical experiments by
significance levels. Statist Med 1990;9:703 –8.
7 Royall R. The effect of sample size on the meaning of signifi- cance tests. Am Stat 1986;40:313 –5.
8 Senn SJ. Discussion of Freeman’s paper. Statist Med
1993;12:1453 –8.
9 Gardner M, Altman D. Statistics with confidence. Br Med J
1989.
10 Matthews R. The great health hoax. Sunday Telegraph 13
September, 1998.
11 Matthews R. Flukes and flaws. Prospect 20–24, November 1998.


@Martijn Weterings: "Was Pearson in 1900 the revival or did this (frequentist) concept appear earlier? How did Jacob Bernoulli think about his 'golden theorem' in a frequentist sense or in a Bayesian sense (what does the Ars Conjectandi tell and are there more sources)?

The American Statistical Association has a webpage on the History of Statistics which, along with this information, has a poster (reproduced in part below) titled "Timeline of statistics".

• AD 2: Evidence of a census completed during the Han Dynasty survives.

• 1500s: Girolamo Cardano calculates probabilities of different dice rolls.

• 1600s: Edmund Halley relates death rate to age and develops mortality tables.

• 1700s: Thomas Jefferson directs the first U.S. Census.

• 1839: The American Statistical Association is formed.

• 1894: The term “standard deviation” is introduced by Karl Pearson.

• 1935: R.A. Fisher publishes Design of Experiments. In the "History" section of Wikipedia's webpage "Law of large numbers" it explains:

"The Italian mathematician Gerolamo Cardano (1501–1576) stated without proof that the accuracies of empirical statistics tend to improve with the number of trials. This was then formalized as a law of large numbers. A special form of the LLN (for a binary random variable) was first proved by Jacob Bernoulli. It took him over 20 years to develop a sufficiently rigorous mathematical proof which was published in his Ars Conjectandi (The Art of Conjecturing) in 1713. He named this his "Golden Theorem" but it became generally known as "Bernoulli's Theorem". This should not be confused with Bernoulli's principle, named after Jacob Bernoulli's nephew Daniel Bernoulli. In 1837, S.D. Poisson further described it under the name "la loi des grands nombres" ("The law of large numbers"). Thereafter, it was known under both names, but the "Law of large numbers" is most frequently used.

After Bernoulli and Poisson published their efforts, other mathematicians also contributed to refinement of the law, including Chebyshev, Markov, Borel, Cantelli and Kolmogorov and Khinchin.".

Question: "Was Pearson the first person to conceive of p-values?"

No, probably not.

In "The ASA's Statement on p-Values: Context, Process, and Purpose" (09 Jun 2016) by Wasserstein and Lazar, doi: 10.1080/00031305.2016.1154108 there's an official statement on the definition of the p-value (which is no doubt not agreed upon by all disciplines utilizing, or rejecting, p-values) which reads:

". What is a p-Value?

Informally, a p-value is the probability under a specified statistical model that a statistical summary of the data (e.g., the sample mean difference between two compared groups) would be equal to or more extreme than its observed value.

3. Principles

...

6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.

Researchers should recognize that a p-value without context or other evidence provides limited information. For example, a p-value near 0.05 taken by itself offers only weak evidence against the null hypothesis. Likewise, a relatively large p-value does not imply evidence in favor of the null hypothesis; many other hypotheses may be equally or more consistent with the observed data. For these reasons, data analysis should not end with the calculation of a p-value when other approaches are appropriate and feasible.".

Rejection of the null hypothesis likely occurred long before Pearson.

Wikipedia's page on early examples of null hypothesis testing states:

Early choices of null hypothesis

Paul Meehl has argued that the epistemological importance of the choice of null hypothesis has gone largely unacknowledged. When the null hypothesis is predicted by theory, a more precise experiment will be a more severe test of the underlying theory. When the null hypothesis defaults to "no difference" or "no effect", a more precise experiment is a less severe test of the theory that motivated performing the experiment. An examination of the origins of the latter practice may therefore be useful:

1778: Pierre Laplace compares the birthrates of boys and girls in multiple European cities. He states: "it is natural to conclude that these possibilities are very nearly in the same ratio". Thus Laplace's null hypothesis that the birthrates of boys and girls should be equal given "conventional wisdom".

1900: Karl Pearson develops the chi squared test to determine "whether a given form of frequency curve will effectively describe the samples drawn from a given population." Thus the null hypothesis is that a population is described by some distribution predicted by theory. He uses as an example the numbers of five and sixes in the Weldon dice throw data.

1904: Karl Pearson develops the concept of "contingency" in order to determine whether outcomes are independent of a given categorical factor. Here the null hypothesis is by default that two things are unrelated (e.g. scar formation and death rates from smallpox). The null hypothesis in this case is no longer predicted by theory or conventional wisdom, but is instead the principle of indifference that lead Fisher and others to dismiss the use of "inverse probabilities".

Despite any one person being credited for rejecting a null hypothesis I don't think it's reasonable to label them the "discover of skepticism based on weak mathematical standing".

• I have been looking up the work from Daniel Bernouilli (second edition is 1808 but the work itself is from 1734) "RECHERCHES PHYSYQUES ET ASTRONOMIQUES, sur le problème proposé pour la seconde fois par l'Académie Royale des Sciences de Paris : Quelle est la cause physique de l'inclinaison des plans des Orbites des Planètes par rapport au plan de l'Equateur de la révolution du Soleil autour de son axe; et d'où vient que les inclinaisons de ces Orbites sont différentes entre elles. e-rara.ch/zut/wihibe/content/titleinfo/13426461 May 15, 2018 at 15:33
• It mentions something like the probability that the inclination of all the six planets occur in the same of 17 sectors, which is $1:17^5$. It is the calculation of a probability but not so much a p-value. May 15, 2018 at 15:44