2
$\begingroup$

When reading about cumulation if type-1 Error, the sentence "for independent statistical tests" occures alot, now I was wondering what this is actually means.

Since tests are also random variables on the sample space, the definition of independence can be applied, mathematically. Now lets look at two tests: $t_1,t_2$, the definition of independence could be:

$P(t_1 \in A, t_2 \in B) = P(t_1 \in A) \cap P(t_2 \in B)$ for alle sets $A,B$ in $\mathcal{B}(\mathbb{R})$. Since tests can only assume discrete values, the sets $A,B$ can in reality only be discrete sets consiting of $0,1$ and sometimes one more values.

Now so far so good, but recently I read a paper in which the following sitation occured: A test rejected the $0$ hypothesis, then the test was repeated on a different sample. The author then stated, that the probability for a type-1 Error in this "repeat test setup" was $\alpha^2$, because of the independence of the tests. I understand why this is true if I acknowledge the "independent" part, but this is the part I have my problems with.

To complete my post, here is the paper the statement is from, it is a download link. The statement is made on page 2 in the last paragraph.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

This is really talking about conditional independence, with conditioning on the hypothesis.

So if the null hypothesis is correct, then the probability the test is significant should be $\alpha$. If you then collect new data and test the new data, the probability the second test is significant should also be $\alpha$, and the conditional independence means the probability of both tests being significant is $\alpha^2$.

Similarly, if the alternative point hypothesis is correct, and the test has power $1-\beta$, i.e. that is the probability of a significant result, then the probability of two significant results is $(1-\beta)^2$ for the same reason.

But if you do not know which hypothesis is correct then you do not get this independence. If in a quasi-Bayesian approach you think there is a probability $p$ that the null hypothesis is correct and a $1-p$ probability that the alternative point hypothesis is correct, then the probability the first test is significant is $p\alpha + (1-p)(1-\beta)$, and that both are significant is $p\alpha^2 + (1-p)(1-\beta)^2$ which is not the square of the previous expression when $0 < p<1$. This is the concern you express in your question.

You also need to consider what you would do if the first test was significant and the second not, which happens with probability $\alpha(1-\alpha)$ or $\beta(1-\beta)$ depending on which hypothesis is correct.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Can you elaborate on the part, why they are conditionally independent? $\endgroup$
    – QED
    Commented Jan 27, 2023 at 12:08
  • $\begingroup$ Is what is meant here by independence really the definition I described above, is the definition even appliable, since sample space could change. $\endgroup$
    – QED
    Commented Jan 27, 2023 at 12:16
  • $\begingroup$ Yes - if you want to think about it that way, and you started with $(\Omega_1, \mathcal F_1, \mathbb P_1)$ for the first test and $(\Omega_2, \mathcal F_2, \mathbb P_2)$ for the second with the same underlying distribution then combined this might be $(\Omega_{both}, \mathcal F_{both}, \mathbb P_{both})$ where $\Omega_{both}=\Omega_1 \times \Omega_2$ and $\mathbb P_{both}) = \mathbb P_{1}(A)\mathbb P_{2}(B)$ for $A \in \mathcal F_1$ and for $B \in \mathcal F_2$ $\endgroup$
    – Henry
    Commented Jan 27, 2023 at 12:16
  • $\begingroup$ Well but doing that automatically makes them independet right? In this setting, there are no "dependent" tests, when the experiment is repeated. $\endgroup$
    – QED
    Commented Jan 27, 2023 at 12:17
  • $\begingroup$ You need to include the hypotheses somewhere, presumably affecting $\Omega$ and/or $\mathbb P$ if you do not take account of the uncertainty, or using conditional probability methods if you do. $\endgroup$
    – Henry
    Commented Jan 27, 2023 at 12:19
0
$\begingroup$

After thinking alot about this now, I think the definition above does not apply here at all. The problem is, that the two tests are part of 2 independent experiments, therefore should get their own sample spaces etc. Doing that, independence does however not mean anything anymore, since we used the product space measure, which automatically makes them independent. This also makes sense, why would test2 be influenced by test1, when caring about a type1/type2 error. There is no correspondence there.

I think what the author meant by independence was more that a new sample was chosen.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Right, tests on the same data set are usually dependent to one degree or another. Even tests of orthogonal contrasts in ANOVA are dependent because the test statistics share a common MSE in their denominators. BTW, when they speak of "dependent tests" in multiple comparisons, they are usually referring to the dependence of the test statistics (eg, the T-statistics). $\endgroup$
    – BigBendRegion
    Commented Jan 29, 2023 at 15:21
  • $\begingroup$ Thanks for that info, that was really useful. $\endgroup$
    – QED
    Commented Jan 30, 2023 at 10:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.