Imagine we test 2 dependent samples: For example, we first test an overall sample (treatment vs. control) and then a subset of this overall sample (treatment sbg vs. control sbg). In this case, we are performing 2 tests. But, they are not independent. Hence, $\alpha^*$ is not equal to $1-(1-\alpha)^2$. Do you know how to calculate theoretically this new $\alpha^*$ when the tests are not independent?
Edit: In this case, I am not interested in the adjusted critical value. For example, when we perform 2 independent tests at 5% level. The probability of rejecting at least one hypothesis under the null is not 5% but 9.75%. But when I perform a first test in an overall population and then in a subgroup, there is no independence. Let's take an example:
Treatment arm: 250 patients (125 males, 125 females) Control arm: 250 patients (125 males, 125 females)
So, I first compare the 2 treatments (t-test) and then, I only compare the 2 treatments into a pre-specifie subgroup (let's say male). But this second test is not independent from the first one. Hence, the overall type I error is not 9.75% but... That is what I am looking for. I think I need to use the variance-covariance matrix. But I do not know how to do the calculation and I think it was already done in the past. But I do not find any papers which explain that.