1
$\begingroup$

Say I plan an experiment to test the null hypothesis of $H_0:\mu_1-\mu_2 = 0$ and, with a power analysis for a Student's $t$-test for independent samples, I find that samples of $n=10$ (per group) will give me power of 99.5% with the alternative hypothesis of $H_a: \mu_1-\mu_2=3$ with $\alpha=0.005$.

(Assume the statistical model is a good enough match to reality for the purposes of the analysis.)

Now I run the planned experiment and reject the null hypothesis. What is my inference regarding $H_a$?

It seems to me that any time I reject $H_0$ I implicitly accept the negation of $H_0$, but should I accept the planned point value of $H_a$?

A test where I accept the planning alternative whenever I reject the null would allow me to accept an alternative when the evidence do not actually favour it.

(Yes, there are several other questions an answers that are similar, but I have not found any that is explicit on this point, and I think clarity here will be useful.)

$\endgroup$
1
$\begingroup$

It seems you understand it just fine: evidence against null cannot be immediately translated into evidence for any particular $H_a$ (unless $H_a$ is just the negation of $H_0$). For an obvious example, a significant p-value from a two-tailed t-test tells you nothing about the effect direction.

If you need inference beyond null hypothesis testing, a small step forward is looking into confidence intervals, but the final answer is Bayesian analysis: then your posterior support for $H_a$ will be determined from prior support for $H_a$ and new evidence for it. In practice, this is often replaced with a qualitative discussion of probable effect sizes, power, and evidence ("we had good power to detect any effects larger than $c$..."). IMHO, this requires just as much effort and provides less answers than the Bayesian framework.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.