I am new to hypothesis testing so forgive me if this has already been asked. Suppose that I've already decided on a null hypothesis with a significance level $\alpha = 0.05$, then obtained my representative sample, then computed the statistic of concern, and then computed a $p$-value that leads me to reject my null hypothesis.
1) Does it follow that I'll reject the null hypothesis again when I perform the experiment on another representative sample?
I think not necessarily but I would like a more mathematically formal answer. But one practical consideration is that performing the experiment again may not be possible or may be costly, so that generalizing the result from one representative sample may seem like a good idea.
One interpretation told to me, with $\alpha=0.05$, is that if I were to perform the experiment $100$ times with $100$ representative samples, then $5$ times, the statistic will fall in the rejection region, leading me to reject the null hypothesis.
2) Can I think of these $5$ times as obtaining $p$ values less than $\alpha$?
Thanks for any help or useful links.
-
1$\begingroup$ Can you clarify what you want when you say 'mathematically formal'? I could stick symbols all over my answer but it would only render it less readable, not any more correct. $\endgroup$– Glen_bCommented Apr 24, 2015 at 1:56
1 Answer
Does rejection of null hypothesis (p<α) for one representative sample imply rejection in a different sample?
Definitely not.
Consider a hypothesis test carried out at level $\alpha$. When $H_0$ is true, you reject a fraction of the time that is $\alpha$ (or for composite null, no more than $\alpha$).
So imagine $H_0$ is true and you reject. You then get a new sample at random from the population. The probability that the next sample would result in rejection would still only be $\alpha$.
Now consider a situation where the null is false, but where the power is 50%. Then the probability that the next sample will reject given the current sample rejected is a toss-up.
In particular, for two random samples taken independently from the same population, the probability of rejection for each is normally independent of a rejection in the other (leaving aside some potential issues with small populations, and so on).
Can I think of these 5 times as obtaining p values less than α?
Yes, the test statistic falling into the rejection region corresponds to a p-value $\leq \alpha$
