I heard that:

  • Under the null hypothesis, the distribution of p-values is flat over $[0,1]$.

  • Under the alternative hypothesis, the distribution of p-values is skewed towards $0$.

Is there a situation under which the distribution of p-values is skewed towards $1$?

  • $\begingroup$ What would you consider an error for the test rule? $\endgroup$ – Maarten Buis Jul 9 '13 at 15:22
  • $\begingroup$ 0-1 errors, i.e. FP error, FN error. $\endgroup$ – Tim Jul 9 '13 at 15:26
  • $\begingroup$ I have no idea what those abbreviations mean $\endgroup$ – Maarten Buis Jul 9 '13 at 15:33
  • $\begingroup$ False positive/negative. $\endgroup$ – Tim Jul 9 '13 at 15:35
  • $\begingroup$ A false positive or negative requires a choice (significant or not significant). The $p$-value feeds into that choice, not the other way around. So problems with $p$-values may influence the rate of false positives or negatives, but not the other way around. $\endgroup$ – Maarten Buis Jul 9 '13 at 15:40

It is also possible to have the effect in practice, when the null hypothesis is true but not all the assumption for the test are given. For example, the classical (non Welch) t-Test assumes equal variance in both groups. In the case that both groups are equally sized a violation is usually not that bad, otherwise the null distribution gets skewed.

If the smaller group has a higher variance than the larger one the null distribution is skewed towards 0 and if it has a smaller variance it is skewed towards 1.

Some R Code for experimentation:

p.vals <- vector("numeric", 1e5)
for (i in 1:1e5) {
  x <- rnorm(5, 0, 1)
  y <- rnorm(50, 0, 10)
  p.vals[i] <- t.test(x,y, var.equal = TRUE)$p.value

Distribution of Pvalues

The example shown is the case where the larger group has higher variance. Note that a skewing of the null distribution towards 1 indicated the test is too conservative so results in more Type II Errors and skewing towards 0 gives too many false positives (Type I error).

| cite | improve this answer | |
  • $\begingroup$ @COOLSerdash Thanks for adding the picture. I would have done so in the first place but I unable to upload picture from where I was when I wrote the post. $\endgroup$ – Erik Jul 10 '13 at 21:33
  • $\begingroup$ (+1) I just thought it would be nice if one can see the distribution. Sorry, I didn't want to mess with your answer ;) $\endgroup$ – COOLSerdash Jul 10 '13 at 21:35
  • $\begingroup$ The t test you used assume the variances of the groups are equal, which is not correct for the data. Can this mismatch between data and assumption be found from the distribution of the p-value under a null or alternative? $\endgroup$ – Tim Jul 10 '13 at 23:15
  • $\begingroup$ @Tim I am not quite sure what you are asking. Are you wondering whether this property can be used to detect violations of your assumption on actual sample data? $\endgroup$ – Erik Jul 11 '13 at 12:24
  • 1
    $\begingroup$ Ok. The answer is not directly since you just have one data set and therefore just one p-value. You are also not sure about whether the null hypothesis or the alternative distribution is true and what you would expect. However, bootstrapping can and is often used to do something similar though you usually do not look at the empirical p-value distribution but at the underlying statistic. In the example above that would be either the t-statistic or more commonly the actual difference of means. $\endgroup$ – Erik Jul 11 '13 at 13:27

That can happen in a one-sided test when your "true" parameter is inside the region of the null hypothesis but not on the boundary. Consider the following example in Stata where the "true" parameter (in this case mean) is 1:

clear all

program define sim, rclass
    drop _all
    set obs 100
    gen x = rnormal(1,1)
    ttest x = 0.75
    return scalar p1 = r(p_l)
    ttest x = 1
    return scalar p2 = r(p_l)

simulate p1=r(p1) p2=r(p2) , reps(20000) : sim
simpplot p1 p2, scheme(s2color) ylabel(,angle(horizontal)) ///
legend(order( 2 "H0: {&mu} {&ge} .75" 3 "H0: {&mu} {&ge} 1"))

enter image description here

I like this representation of the $p$-values. It shows on the y-axis the difference between the empirical estimate of the Cumulative Distribution Function (CDF) and the theoretical (continuous standard uniform) distribution. On the x-axis is the nominal $p$-value. The logic behind this graph is that for $p$-values in a simulation study in which the null hypothesis is true, the empirical CDF is an empirical estimate of the $p$-value. The empirical CDF gives for each nominal $p$-value an estimate of the probability of drawing a sample which deviates at least as much from the null hypothesis as the current sample (i.e. has a nominal $p$-value less than or equal to the current nominal $p$-value) if the null hypothesis is true. So negative values on the y-axis means that the emprical estimates of the $p$-value are less than the nominal $p$-values and positive values on the y-axis say that the empirical estimates of the $p$-values are larger than the nominal $p$-values. So the blue points correspond to a cumulative density function which bluges below the diagonal line which one would expect for a continuous standard uniform distribution. The corresponding histogram is shown below:

enter image description here

| cite | improve this answer | |
  • $\begingroup$ Thanks!So it is skewed towards both 0 and 1? How about only towards 1? $\endgroup$ – Tim Jul 9 '13 at 15:25
  • $\begingroup$ No, the blue points represent a curve that is just skewed towards 1. $\endgroup$ – Maarten Buis Jul 9 '13 at 15:29
  • $\begingroup$ I have added the corresponding histogram, maybe that is easier to interpret. $\endgroup$ – Maarten Buis Jul 9 '13 at 15:32
  • 1
    $\begingroup$ What do the x and y axes in the plot mean? $\endgroup$ – Tim Jul 9 '13 at 16:04
  • $\begingroup$ the first plot. $\endgroup$ – Tim Jul 9 '13 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.