Overall type I error with dependent tests

Imagine we test 2 dependent samples: For example, we first test an overall sample (treatment vs. control) and then a subset of this overall sample (treatment sbg vs. control sbg). In this case, we are performing 2 tests. But, they are not independent. Hence, $\alpha^*$ is not equal to $1-(1-\alpha)^2$. Do you know how to calculate theoretically this new $\alpha^*$ when the tests are not independent?

Edit: In this case, I am not interested in the adjusted critical value. For example, when we perform 2 independent tests at 5% level. The probability of rejecting at least one hypothesis under the null is not 5% but 9.75%. But when I perform a first test in an overall population and then in a subgroup, there is no independence. Let's take an example:

Treatment arm: 250 patients (125 males, 125 females) Control arm: 250 patients (125 males, 125 females)

So, I first compare the 2 treatments (t-test) and then, I only compare the 2 treatments into a pre-specifie subgroup (let's say male). But this second test is not independent from the first one. Hence, the overall type I error is not 9.75% but... That is what I am looking for. I think I need to use the variance-covariance matrix. But I do not know how to do the calculation and I think it was already done in the past. But I do not find any papers which explain that.

• Is $\alpha^*$ supposed to be the appropriate critical value, or just the actual long run type I error rate when everything was done conventionally? – gung - Reinstate Monica Oct 9 '12 at 12:39
• You can now merge your two unregistered accounts yourself, Momo. Please, refer to the following instructions on our help page. – chl Mar 10 '13 at 20:06

When statistical issues get sufficiently complex, sometimes I prefer to do a simulation to get my answer. Now I won't vouch for this solutions being 100% correct. It has been many months since I worked with simulations of this kind, so I would probably talk it over with a few other people to be sure, but since we are here on CV I think the comments and voting should take care of that.

... at any rate, on with the show...

In your case I think it makes sense to:

1. Run both tests and record your inferential statistic
2. Set aside your good data
3. Shuffle across treatment and control labels, but keep males as males and females as females. For each new shuffle, calculate and record new inferential statistics along with their p-values. These are your empirical reference distributions.

Now, I think you are saying you want to control Familywise $\alpha = .05$.

If so, what you will do is identify the proportion of the shuffles you conducted in which (EDIT) any p value was $\leq$ .05. Then for each reference distribution you look up what the value is at that proportion, those are your new dependency adjusted critical values.