1
$\begingroup$

I have written a script that computes a list of values (based on some parameters). Now, I executed the script for different parameters1. And for each value, I did it 5 times.

5 times param1 = 10; 5 times param1 = 100; ....

On each of these datasets, I performed a one sample t-test (so H0: mean = X, H1: mean < X). Thus, I have results like this: For param1 = 10: 5 out of 5 H0 accepted For param1 = 100: 1 out of 5 H0 accepted ...

Now I would want for each parameter1 only 1 value: accepted or rejected. Can I say that I accept the null hypothesis if for example more than half of H0's are accepted (So in this example, for param1 = 10 it would be accepted, while it would be rejected for param1 = 100), or is there another way to do it?

$\endgroup$

1 Answer 1

4
$\begingroup$

Suppose you only care about each value of the parameter separately; that is to say, you want to make separate inferences for each instance of the parameter, and that the overall Type I error control across the families of hypotheses corresponding to each instance of the parameter does not need to be maintained. Rather, you want some Type I error control over each family comprising the five tests at a given value of the parameter.

You have already intuitively recognized that if each $t$-test of one of the five samples is conducted at, say, level $\alpha$, then the probability that at least one of these null hypotheses is rejected erroneously (i.e. the test rejects $H_0$ when $H_0$ is true), is substantially greater than $\alpha$--this is the multiplicity problem. Specifically, for five independent samples, the familywise Type I error is as large as $1 - (1-\alpha)^5$. For $\alpha = 0.05$ for a single test, this would inflate the familywise error to over $0.22$.

However, what has not been discussed above is that the overall conclusion need not be that $H_a$ is true if only one hypothesis in the family is rejected--if, for example, you decide that to reject the familywise null hypothesis one must reject $H_0$ for each of the five tests, then clearly you have no multiplicity issue, and your type I error is actually tiny: the chance of rejecting $H_0$ (familywise) when $H_0$ is true becomes $\alpha^5$, and for $\alpha = 0.05$ this is $3.125 \times 10^{-7}$.

In between these two extremes, one might use the intuitive criterion that at at least 3 out of 5 tests must reject $H_0$. What would be the familywise Type I error in this case? I leave this question to the reader as an exercise.

That said, none of the above addresses the power under these scenarios, or even the more fundamental question of the choice of $\alpha$ for a single test to begin with; neither of which were mentioned in your question.

$\endgroup$
1
  • $\begingroup$ As an aside, I just want to state for the record that the phrase "$H_0$ accepted" is abhorrent and must never be used. The notion that a frequentist test could ever provide statistical evidence that the null hypothesis is true, when the distribution of the test statistic is conditional on the null being true is an affront to statistical practice. $\endgroup$
    – heropup
    Commented Dec 7, 2018 at 18:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.