7
$\begingroup$

I heard that:

  • Under the null hypothesis, the distribution of p-values is flat over $[0,1]$.

  • Under the alternative hypothesis, the distribution of p-values is skewed towards $0$.

Is there a situation under which the distribution of p-values is skewed towards $1$?

$\endgroup$
7
  • $\begingroup$ What would you consider an error for the test rule? $\endgroup$ Jul 9, 2013 at 15:22
  • $\begingroup$ 0-1 errors, i.e. FP error, FN error. $\endgroup$
    – Tim
    Jul 9, 2013 at 15:26
  • $\begingroup$ I have no idea what those abbreviations mean $\endgroup$ Jul 9, 2013 at 15:33
  • $\begingroup$ False positive/negative. $\endgroup$
    – Tim
    Jul 9, 2013 at 15:35
  • $\begingroup$ A false positive or negative requires a choice (significant or not significant). The $p$-value feeds into that choice, not the other way around. So problems with $p$-values may influence the rate of false positives or negatives, but not the other way around. $\endgroup$ Jul 9, 2013 at 15:40

2 Answers 2

7
$\begingroup$

It is also possible to have the effect in practice, when the null hypothesis is true but not all the assumption for the test are given. For example, the classical (non Welch) t-Test assumes equal variance in both groups. In the case that both groups are equally sized a violation is usually not that bad, otherwise the null distribution gets skewed.

If the smaller group has a higher variance than the larger one the null distribution is skewed towards 0 and if it has a smaller variance it is skewed towards 1.

Some R Code for experimentation:

p.vals <- vector("numeric", 1e5)
for (i in 1:1e5) {
  x <- rnorm(5, 0, 1)
  y <- rnorm(50, 0, 10)
  p.vals[i] <- t.test(x,y, var.equal = TRUE)$p.value
}
hist(p.vals)

Distribution of Pvalues

The example shown is the case where the larger group has higher variance. Note that a skewing of the null distribution towards 1 indicated the test is too conservative so results in more Type II Errors and skewing towards 0 gives too many false positives (Type I error).

$\endgroup$
6
  • $\begingroup$ @COOLSerdash Thanks for adding the picture. I would have done so in the first place but I unable to upload picture from where I was when I wrote the post. $\endgroup$
    – Erik
    Jul 10, 2013 at 21:33
  • $\begingroup$ (+1) I just thought it would be nice if one can see the distribution. Sorry, I didn't want to mess with your answer ;) $\endgroup$ Jul 10, 2013 at 21:35
  • $\begingroup$ The t test you used assume the variances of the groups are equal, which is not correct for the data. Can this mismatch between data and assumption be found from the distribution of the p-value under a null or alternative? $\endgroup$
    – Tim
    Jul 10, 2013 at 23:15
  • $\begingroup$ @Tim I am not quite sure what you are asking. Are you wondering whether this property can be used to detect violations of your assumption on actual sample data? $\endgroup$
    – Erik
    Jul 11, 2013 at 12:24
  • 1
    $\begingroup$ Ok. The answer is not directly since you just have one data set and therefore just one p-value. You are also not sure about whether the null hypothesis or the alternative distribution is true and what you would expect. However, bootstrapping can and is often used to do something similar though you usually do not look at the empirical p-value distribution but at the underlying statistic. In the example above that would be either the t-statistic or more commonly the actual difference of means. $\endgroup$
    – Erik
    Jul 11, 2013 at 13:27
6
$\begingroup$

That can happen in a one-sided test when your "true" parameter is inside the region of the null hypothesis but not on the boundary. Consider the following example in Stata where the "true" parameter (in this case mean) is 1:

clear all

program define sim, rclass
    drop _all
    set obs 100
    gen x = rnormal(1,1)
    ttest x = 0.75
    return scalar p1 = r(p_l)
    ttest x = 1
    return scalar p2 = r(p_l)
end

simulate p1=r(p1) p2=r(p2) , reps(20000) : sim
simpplot p1 p2, scheme(s2color) ylabel(,angle(horizontal)) ///
legend(order( 2 "H0: {&mu} {&ge} .75" 3 "H0: {&mu} {&ge} 1"))

enter image description here

I like this representation of the $p$-values. It shows on the y-axis the difference between the empirical estimate of the Cumulative Distribution Function (CDF) and the theoretical (continuous standard uniform) distribution. On the x-axis is the nominal $p$-value. The logic behind this graph is that for $p$-values in a simulation study in which the null hypothesis is true, the empirical CDF is an empirical estimate of the $p$-value. The empirical CDF gives for each nominal $p$-value an estimate of the probability of drawing a sample which deviates at least as much from the null hypothesis as the current sample (i.e. has a nominal $p$-value less than or equal to the current nominal $p$-value) if the null hypothesis is true. So negative values on the y-axis means that the emprical estimates of the $p$-value are less than the nominal $p$-values and positive values on the y-axis say that the empirical estimates of the $p$-values are larger than the nominal $p$-values. So the blue points correspond to a cumulative density function which bluges below the diagonal line which one would expect for a continuous standard uniform distribution. The corresponding histogram is shown below:

enter image description here

$\endgroup$
6
  • $\begingroup$ Thanks!So it is skewed towards both 0 and 1? How about only towards 1? $\endgroup$
    – Tim
    Jul 9, 2013 at 15:25
  • $\begingroup$ No, the blue points represent a curve that is just skewed towards 1. $\endgroup$ Jul 9, 2013 at 15:29
  • $\begingroup$ I have added the corresponding histogram, maybe that is easier to interpret. $\endgroup$ Jul 9, 2013 at 15:32
  • 1
    $\begingroup$ What do the x and y axes in the plot mean? $\endgroup$
    – Tim
    Jul 9, 2013 at 16:04
  • $\begingroup$ the first plot. $\endgroup$
    – Tim
    Jul 9, 2013 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.