6
$\begingroup$

During a lesson at university, we ran this simulation to assess the fact that p-value distribution under alternative hypothesis is stochastically smaller than the uniform distribution.

So suppose we want to make an F-test for a linear regression (joint nullity of parameters)

n=10
p=3
beta=c(1,2,0) #beta_2=2,null hypothesis of test f is false
sim<-function(n,p)
{
x<-cbind(1,matrix(runif(n*(p-1)),ncol=p-1))
y<-x%*%beta+rnorm(n)
X<-as.data.frame(x)
anova(lm(y~1),lm(y~.,X))
#prendo il p value
pval<-anova(lm(y~1),lm(y~.,X))$'Pr(>F)'[2]
return (pval)
}

res<-replicate(100,sim(10,3))
hist(res)
plot(ecdf(res))
curve(punif,0,1,add=T,col="red")

and I obtain this graph

enter image description here

Could anyone explain (and maybe provide a little proof) of the above statement:

p-value distribution under alternative hypothesis is stochastically smaller than the uniform distribution

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
2
$\begingroup$

I assume that this is a self-study question, so I will not give a full explanation, but rather some hints

Assuming you know

  • What "stochastically smaller" means (see Wikipedia)
  • And you are able to interpret the difference of two cumulative distribution such as in the figure above above (note that the red line is the uniform, maybe it helps to look at the histogram as well)

Then the answer should be obvious.

Side note: excellent that you do these kind of simulations in class, I think this is very instructive.

Improve this answer
$\endgroup$
11
  • 1
    $\begingroup$ Ok i realize that it was a dumb question. Anyway is there any proof for that fact without using any simulation? Is that a general fact that holds always? thanks $\endgroup$
    – Marco Fumagalli
    Commented Dec 29, 2016 at 16:10
  • $\begingroup$ You want to proof that ANY alternative hypothesis is stochastically smaller than a uniform under a correctly specified hypothesis test? $\endgroup$
    – Florian Hartig
    Commented Dec 29, 2016 at 21:00
  • 1
    $\begingroup$ Is it something impossible? $\endgroup$
    – Marco Fumagalli
    Commented Dec 30, 2016 at 15:19
  • $\begingroup$ Honestly, I don't know. My intuition would be that it should be possible to construct a test statistic that produces a counterexample, but then we wouldn't call it a valid hypothesis test. Would be a good question to ask. $\endgroup$
    – Florian Hartig
    Commented Jan 2, 2017 at 9:33
  • 2
    $\begingroup$ @Florian No; "improperly calibrated" type I error would be called conservative tests or anti-conservative tests. Bias in hypothesis testing is when you have lower rejection rates under some alternative than under the null. These are really common in practice, however. $\endgroup$
    – Glen_b
    Commented Jan 16, 2017 at 10:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.