To evaluate a statistical test or means of generating frequentist confidence intervals, it makes sense to repeatedly simulate data for which the null is true and then compute the nominal p-value, and generate a histogram of these p-values. For a valid test, this distribution is basically flat (uniform). There is good discussion of this point here. Within either the frequentist or Bayesian literature, is there a name for this type of histogram/ method for assessing nominal p-values? A citation for when this approach was first proposed/ implemented?
$\begingroup$
$\endgroup$
0
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
6
This idea of the uniform distribution for P-values is fairly new in statistics education and practice. I don't know if anyone has yet made up a name for the related histograms that has come into general use. Below I just call them "Null P-value" histograms.
It is important to note that this uniform distribution for P-values holds only if the null hypothesis is true, the test statistic is continuous, and all of the assumptions for the test are met.
Ordinarily, the test statistic must be exact and continuous, as in a one-sample t test or ANOVA. Tests involving discrete distributions and certain approximations have useful P-values for hypothesis testing, but often the P-values are not uniformly distributed across the interval $(0,1).$
Below are a few examples. All
tests shown are standard tests in R, with P-values 'extracted' using $ notation.
Code for the histogram is shown only in the first example; except for the header the code is the same in all examples.
Shapiro-Wilk test for normality: $H_0$ true because data are normal. Too many P-values near 1.
set.seed(1212)
pv = replicate(10^5, shapiro.test(rnorm(20))$p.val)
mean(pv < .05)
[1] 0.04924
hist(pv, prob=T, col="skyblue2", main="Shapiro-Wilk Null P-values")
curve(dunif(x), add=T, col="red", n=10001)
One-sample Wilcoxon test: $H_0$ is true because population sampled has median 0. Discrete rank-based test statistic.
set.seed(1212)
pv = replicate(10^5, wilcox.test(rnorm(20), mu=0)$p.val)
mean(pv < .05)
[1] 0.04905
Binomial test: Discrete test statistic, $H_0$ true because $p = 1/2.$ Because of discreteness a test at exactly the 5% level is not available.
set.seed(1213)
pv = replicate(10^5,
binom.test(rbinom(1,20,.5), 20, p=.5, alt="two")$p.val)
mean(pv < .05)
[1] 0.04169
Pooled 2-sample t test: Assumptions not met because variances unequal. $H_0$ true because means equal. This test rejects more often than 5% of the time. (Note: In R, the default two-sample t.test is the Welch test; the parameter var.eq=T invokes a pooled test.)
set.seed(1213)
pv = replicate(10^5,
t.test(rnorm(20,100,2), rnorm(10,100,20), var.eq=T)$p.val)
mean(pv < .05)
[1] 0.18476
Welch 2-sample t test: P-value has uniform distribution on $(0,1).$ Continuous test statistic. Assumptions met. $H_0$ true. Technically an approximate test, but very nearly exact.
set.seed(1214)
pv = replicate(10^5, t.test(rnorm(20,100,2),
rnorm(10,100,20))$p.val)
mean(pv < .05)
[1] 0.04939
Reference: Murcoch DJ, Tsai Y-L, Adcock J: P-values are random variables (2008), The American Statistician, 242-245, has several histograms similar to those shown here. This paper contains an early emphasis, if not the first, on regarding P-values as random variables, using Monte Carlo simulation to obtain their distributions in various cases, and the standard uniform distribution of P-values from continuous test statistics under the null hypothesis. The caption of Figure 2 in that paper refers to "Histograms of p-values under the null hypothesis."
An earlier paper in the same journal, Sackrowitz H & Samuel-Cahn E (1999), P-values as random variables---Expected P-values, 326-333, does not contain such histograms.
-
1$\begingroup$ Thanks for the examples. Do you have a guess of how new "fairly new" is? Any clues regarding the intellectual history of this pretty straightforward idea? $\endgroup$ Commented Apr 29, 2019 at 13:23
-
$\begingroup$ A decade or so ago there was a paper in The American Statistician that seemed to get the this discussion started. At least the first I was aware of it. Can't find the ref immediately; maybe someone on this site knows it. Will ask around. // I'm in my early 80's and have been teaching stat courses for over 50 yrs, so 'fairly new may mean something different to me than it does to others. $\endgroup$– BruceETCommented Apr 29, 2019 at 14:53
-
$\begingroup$ Thanks. I saw in the CV post I cited this paper: [Duncan J Murdoch, Yu-Ling Tsai & James Adcock (2008) P-Values are Random Variables, The American Statistician, 62:3, 242-245, DOI: 10.1198/000313008X332421] (doi.org/10.1198/000313008X332421); certainly people were doing this before though? $\endgroup$ Commented Apr 29, 2019 at 16:46
-
$\begingroup$ I added Murdoch et al. and one earlier paper to my Answ as references. I just scanned Wikipedia on P-values, which gives the impression early uses were connected to discrete test statistics. // Recently, various uses of P-values have been challenged, with ASA-endorsed statements deprecating some uses. IMHO, the research literature in psychology and other social sciences is full of blatant misuses, resulting in the inability to replicate some studies once touted as 'landmark', so some change clearly warranted. Anyhow, if you search recent articles on P-values you will see a lot of controversy. $\endgroup$– BruceETCommented Apr 29, 2019 at 19:02
-
2$\begingroup$ As early as 1982 this was well known. See folk.uio.no/tores/Publications_files/… $\endgroup$ Commented Aug 6, 2019 at 1:18
$\begingroup$
$\endgroup$
6
...is there a name for this type of histogram/method for assessing nominal p-values?
The true distribution of a quantity under a (simple) null hypothesis is called the null distribution of that quantity. There is no specific name for the histogram of a Monte-Carlo simulation of the distribution of the p-value. It would usually be named by description: the histogram of a Monte Carlo simulation of the null distribution of the p-value.
-
$\begingroup$ Seems to me Monte Carlo need not always be part of the solution, so need not be part of a generic 'name'. I used Monte Carlo methods mainly to demonstrate some messy cases in which the null dist'n of the P-value is not uniform. But for a t-test on one normal sample, MC would not be necessary to show P-val is UNIF. Also some simple discrete cases can be shown to be non-uniform without MC. $\endgroup$– BruceETCommented Apr 29, 2019 at 14:35
-
$\begingroup$ The question describes that OP "repeatedly simulate[s] data", so it is part of the question. As you can see from my answer, I have not mentioned the uniform distribution. $\endgroup$– BenCommented Apr 29, 2019 at 22:07
-
$\begingroup$ Clear enough. One reason why all my examples are MC. But I'm pondering general terminology that might be brief, generally descriptive, and obvious enough to catch on. Because the Murdoch paper finds pedagogical value in histograms for P-values for other than $H_0,$ maybe not restricted to 'null' either. $\endgroup$– BruceETCommented Apr 30, 2019 at 1:26
-
$\begingroup$ I am eager to learn of terminology that exists, is general, etc. FWIW, a collaborator of mine has been using these histograms since long before the Murdoch paper and calls them "Checkplots." We are hoping to submit a paper this summer introducing them as such (in an ecology journal) and have been keen to learn whether others had a name already. $\endgroup$ Commented Apr 30, 2019 at 12:13
-
$\begingroup$ And to follow up more directly, as @Ben noted, we have been thinking about this using MC simulations, so while overly verbose for our purposes, the description in Ben's answer matches my goal. $\endgroup$ Commented Apr 30, 2019 at 12:14