5
$\begingroup$

During a lesson at university, we ran this simulation to assess the fact that p-value distribution under alternative hypothesis is stochastically smaller than the uniform distribution.

So suppose we want to make an F-test for a linear regression (joint nullity of parameters)

n=10
p=3
beta=c(1,2,0) #beta_2=2,null hypothesis of test f is false
sim<-function(n,p)
{
x<-cbind(1,matrix(runif(n*(p-1)),ncol=p-1))
y<-x%*%beta+rnorm(n)
X<-as.data.frame(x)
anova(lm(y~1),lm(y~.,X))
#prendo il p value
pval<-anova(lm(y~1),lm(y~.,X))$'Pr(>F)'[2]
return (pval)
}

res<-replicate(100,sim(10,3))
hist(res)
plot(ecdf(res))
curve(punif,0,1,add=T,col="red")

and I obtain this graph

enter image description here

Could anyone explain (and maybe provide a little proof) of the above statement:

p-value distribution under alternative hypothesis is stochastically smaller than the uniform distribution

$\endgroup$
1
$\begingroup$

I assume that this is a self-study question, so I will not give a full explanation, but rather some hints

Assuming you know

  • What "stochastically smaller" means (see Wikipedia)
  • And you are able to interpret the difference of two cumulative distribution such as in the figure above above (note that the red line is the uniform, maybe it helps to look at the histogram as well)

Then the answer should be obvious.

Side note: excellent that you do these kind of simulations in class, I think this is very instructive.

$\endgroup$
  • 1
    $\begingroup$ Ok i realize that it was a dumb question. Anyway is there any proof for that fact without using any simulation? Is that a general fact that holds always? thanks $\endgroup$ – Marco Fumagalli Dec 29 '16 at 16:10
  • $\begingroup$ You want to proof that ANY alternative hypothesis is stochastically smaller than a uniform under a correctly specified hypothesis test? $\endgroup$ – Florian Hartig Dec 29 '16 at 21:00
  • 1
    $\begingroup$ Is it something impossible? $\endgroup$ – Marco Fumagalli Dec 30 '16 at 15:19
  • $\begingroup$ Honestly, I don't know. My intuition would be that it should be possible to construct a test statistic that produces a counterexample, but then we wouldn't call it a valid hypothesis test. Would be a good question to ask. $\endgroup$ – Florian Hartig Jan 2 '17 at 9:33
  • 1
    $\begingroup$ @Florian No; "improperly calibrated" type I error would be called conservative tests or anti-conservative tests. Bias in hypothesis testing is when you have lower rejection rates under some alternative than under the null. These are really common in practice, however. $\endgroup$ – Glen_b Jan 16 '17 at 10:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.