My professor asked the following problem to drill in the ideas of significance levels and $p$-values during lectures and it's really doing my head in. Suppose I want to test a particular hypothesis $H_1$. Is there a difference between the two hypothesis testing scenarios?

  1. I pose 99 additional hypotheses $\{H\}_{2\leq i\leq 100}$, collect data, fail to reject null hypothesis on $\{H\}_{1\leq i\leq 99}$  at a 0.05 significance level, then get a $p$-value of 0.01 for $H_{100}$.

  2. I collect data, fail to reject null hypothesis on $H_1$ at a 0.05 significance level, pose a new hypothesis $G$ calculated on the collected data, immediately obtain a $p$-value of 0.01.

What can I claim about $H_{100}$ and $G$ at a 0.05 significance level or a 0.001 significance level? I'm comfortable with my understanding of when to reject for a single test, but when it comes to combining multiple hypothesis tests together I lose track of what we can and cannot do. Could someone please provide some insight into this? Will be eternally grateful!

  • 1
    $\begingroup$ It depends strongly on whether you want to evaluate the evidence concerning H1 with a significance test or whether you want to decide whether to reject H1. See this for a little more essential background: stats.stackexchange.com/questions/16218/… $\endgroup$ – Michael Lew Nov 7 '20 at 20:27
  • $\begingroup$ If you are going to perform a Neyman-Pearsonian hypothesis test then you must set the critical value (i.e. the "significance level") in advance of performing the test. You really need read a little more about hypothesis tests and significance tests before attempting to sort out anything as complicated as your question. $\endgroup$ – Michael Lew Nov 8 '20 at 4:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.