I am thinking about p-values correction in case of multiple test.

In general if the complex test 'contain' several simple test then probability of first type error increases. In this case I conclude that complex null hypothesis is rejected if at least one of the simple hyphothesis is rejected. To reduce 1-type error of complex hypotheisis testing due to cumulation of propability of 1-type error several method vere proposed : https://en.wikipedia.org/wiki/Multiple_comparisons_problem

The my question is what if we want to reject complex null hypothesis if all simple hyphothesis can be rejected.

In first case first type error probability can be calculated as: $$ \alpha_{complex} = 1 - (1-\alpha)^k $$ So to have the same $ \alpha $ value corrected p-value by Sidak correction is: $$ p_{corrected,i} = 1 - (1-p_i)^{k} $$

I assume that in case when we want reject null hypothesis if all $ k $ hypothesis are rejected then first type error can be calculated as folow:

$$ \alpha_{complex} = \alpha^k $$

So as well

$$ p_{corrected,i} = p_i^{k} $$

Is my assumption correct?

What do you think about other correction factor :

$$ p_{corrected,i} = \dfrac{p_{i}}{k} $$

I think that this factor can be better in some cases, but this just my 'personal feeling'. I do not have any evidence for that :)

PS. please correct me if I am completly wrong, I've just started my adventure with statistic:)

  • 1
    $\begingroup$ I think by complex hypothesis you want to simultaneously test several hypotheses and by simple hypothesis you mean the individual hypotheses. You want to make a statement about rejecting all hypotheses. For that you want to assign a p-value to the simultaneous inference. This means that you need to require smaller p-values for each hypothesis in order to safely reject them all. $\endgroup$ Dec 21 '16 at 18:44
  • 1
    $\begingroup$ This is a strong requirement. Now there is a theory allowing out of a large number of hypothesis tests allowing a few false rejections. This is called false discovery rate (FDR). In the strict formulation there are several ways to adjust the p-value. Bonferroni takes the original choice for significance level and divides it by n where n is the number of hypotheses. This is fine if n is small. But it is overly conservative so there are several other ways of doing the correction. You should look into these. $\endgroup$ Dec 21 '16 at 18:56
  • 1
    $\begingroup$ Peter Westfall's book shows how bootstrap and permutation methods can be used and he compares them to the standard ones. $\endgroup$ Dec 21 '16 at 18:58
  • $\begingroup$ I think that for FDR analysis I have to have some knowledge about distribution of alternative hypothesis. If I do not have such a knowledge then i think that I shoud use lets say 'safety' mechanism for p-value correction (or $ \alpha $ correction). I think that the safest mechanism is in exponential equation. $\endgroup$
    – Staszko90
    Dec 22 '16 at 7:39
  • 1
    $\begingroup$ You might be interested in this Q&A stats.stackexchange.com/questions/243003/… which discusses a related issue. $\endgroup$
    – mdewey
    Dec 23 '16 at 9:55

This site is temporarily in read only mode and not accepting new answers.

Browse other questions tagged .