0
$\begingroup$

I was wondering why the expected value (i.e., long-run average) of the p-values from a simulation of a binomial experiment in the below experiment becomes roughly about $.61$, regardless of $p$; the probability of success?

Here is my simulation code in R:

simulation <- function(n, p, n.sim){
    fun = function(){
      x = rbinom(1, n, p)
     pe = x/n
p.value = binom.test(x, n, p)[[3]]
    c(p.value, pe)
}
sim <- t(replicate(n.sim, fun()))
par(mfcol = c(2, 2))
plot(sim[, 1], 1:n.sim, xlim = c(0, 1), pch = 19, col = 2, main = "p.value")
plot(sim[, 2], 1:n.sim, xlim = c(0, 1), pch = 19, col = 4, main = "proportion")
abline(v = p, lty = 2, col = 2)
hist(sim[, 1], main = "p.value")
hist(sim[, 2], main = "proportion")
list(p.value = mean(sim[, 1]), proportion = mean(sim[, 2]))
}
simulation(n = 15, p = .1, n.sim = 1e3) ## Change p to whatever and p.value remains around .61

enter image description here

Improve this question
$\endgroup$
9
  • 1
    $\begingroup$ Since p-values are uniformly distributed under the null (at least approximately, depending on the circumstances and the test), and uniform variables have a mean of $0.5$, it sounds like your computer experiments are in line with theory. $\endgroup$
    – whuber ♦
    Aug 18, 2017 at 20:40
  • $\begingroup$ @whuber, but change the sample size (I should say n to 15) and then what happens? $\endgroup$
    – rnorouzian
    Aug 18, 2017 at 20:45
  • $\begingroup$ It will make no material difference. Hypothesis tests are constructed so that p-values will have uniform distributions under the null hypothesis, or as close to uniform as is feasible, regardless of sample size and regardless of the alternative. (A "composite" null creates a complication that I am glossing over here, because it's not a conceptual one.) $\endgroup$
    – whuber ♦
    Aug 18, 2017 at 20:54
  • 1
    $\begingroup$ Not at all: it has been explained many times on this site. $\endgroup$
    – whuber ♦
    Aug 18, 2017 at 21:26
  • 1
    $\begingroup$ The case of a simple null but discrete test statistic is also discussed here $\endgroup$
    – Glen_b
    Aug 19, 2017 at 1:06

0

Reset to default