0
$\begingroup$

I have two types of items in an experiment (A and B), and each type has two subtypes (1 and 2), so there are four classes of items (A1, A2, B1, B2). The same treatment is applied to all items.

I want to test whether the treatment causes a difference in the response, for each class of item. My hypothesis going into the experiment (ie this is an attempted replication of a previous finding) was that the treatment should have one effect for the A items, and a different effect for the B items.

However, as my analysis is now, it looks like the treatment only had an effect on A1 and B1 (and the effects are in opposite directions, which is fine). I didn't expect a different effect of treatment by subtype (1 and 2). What's the right correction to apply for multiple comparisons in this case?

|cite|improve this question
$\endgroup$
1
$\begingroup$

Multiple correction is all about the inference, so it depends what you want to show with your data. There are two general classes of multiple comparison corrections, and the inference that you can pull from them are different.

Family wise correction

Firstly, you can adjust the Family wise error rate (FWER) of your tests to $\alpha$ level. This means that instead of there being a 0.05 probability of incorrectly rejecting the null for each test, there is now a 0.05 probability of incorrectly rejecting the null in any of your tests. More formally, FWER adjusts such that the probability of $V$ (false positives) being equal to or larger than 1 is controlled at $\alpha$ under arbitrary dependence.

$$P(V \geq 1) \leq \alpha $$

The most popular, though also perhaps the most ill advised, correction methodology is the Bonferroni correction. Contrary to popular belief, Bonferroni does not assume independence, though it will overly conservative in situations of dependence.

For a given test, the probability of incorrectly rejecting the null can be given by

$$ 1-(1-\alpha)$$

and for $M$ (independent) tests, the probability of falsely rejecting in at least one of them is

$$ 1 - (1-\alpha)^M$$

Bonferroni adjusts the significance level such that the threshold a test must cross to be called significant is now

$$ 1 - (1-\alpha)^{\frac{1}{M}}$$

which adjusts the probability of falsely rejecting at least 1 to $\alpha$.

An alternative and uniformly more powerful test is the Holm-Bonferonni, which you may want to look into. For this type of correction, you would specify in your paper that the change that one of your tests is false is adjusted to $\alpha$ level.

A way to get around the over conservative nature of Bonferroni / Holm would be to calculate the number of effective tests (look into this paper)

False discovery rate

As opposed to the more traditional FWER correction, you can also correct for the false discovery rate instead. This method was introduced in 1995 by Benjamini and Hochberg, two Israeli statisticians from Tel Aviv. Instead of adjusting for the probability of making at least one false test, they decided it was more intuitive and important to correct for the rate at which you make these false tests. For $V$ false positives and $S$ true positives:

$$ FDR \equiv \frac{V}{V+S} $$

FDR correction (implemented in any software package, or you can read their paper) orders the test statistics and adjusts based on this ordering. They do this such that for $i$ of $N$ tests, finding the largest $k$ that satisfies

$$ P_{(k)} \leq \frac{k}{N}\alpha$$

then subsequently rejecting $H_{i}$ for $i = 1 ... k$.

For this correction, you would report that among your results you would expect 5% of the positives to be false.

FDR correction is robust to a number of dependencies, and is generally regarded as "safe" correction in the general case. It sounds like you're doing some kind of linear regression or ANOVA testing, which would definitely satisfy the requirements. For a more complete explanation of these dependency situations, see my answer here.

The fact that there are two subgroups in your analysis is intersting, but doesn't pose a challenge to any of the above mentioned methodologies. At least, I haven't mentioned any that it would cause a problem to. If you would like, there is a particular subclass of FDR correction that deals with subgroups called stratified FDR (sFDR) introduced by Lei Sun et al in 2006. I don't think you need to worry about this, but if you ever need a larger more powerful test, you can look it up.

If you have any questions don't hesitate to comment.

References

[1] A farewell to Bonferroni: the problems of low statistical power and publication bias, Behavioral Ecology (2004) 15 (6): 1044-1045 first published online June 30, 2004 doi:10.1093/beheco/arh107

[2] A new measure of the effective number of tests, a practical tool for comparing families of non-independent significance tests. Galwey NW. Genet Epidemiol. 2009 Nov;33(7):559-68. doi: 10.1002/gepi.20408.

[3] Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Yoav Benjamini and Yosef Hochberg Journal of the Royal Statistical Society. Series B (Methodological) Vol. 57, No. 1 (1995), pp. 289-300

[4] The control of the false discovery rate in multiple testing under dependency. Yoav Benjamini and Daniel Yekutieli, Ann. Statist. Volume 29, Number 4 (2001), 1165-1188.

[5] Correction for Multiple comparisons using Mann-Whitney U test

|cite|improve this answer
$\endgroup$
1
$\begingroup$

If you did not specify the hypothesis up-front, clearly you can only claim it is an interesting finding worthy of further exploration. P-values (other than p=1) from any hypothesis testing would be invalid in terms of keeping any familywise type error rate or false discovery rate.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ This is the further exploration already. I ran an experiment with the hypothesis that the treatment should have the same effect on all four classes of items, but I found an interaction by item type instead. To confirm the interaction, I replicated the experiment with twice as many observations, but now there seems to be an interaction by type and also by subtype. I'm trying to figure out what, if anything, I can conclude from all this. The data are very expensive to collect and it's impractical to run another experiment. $\endgroup$ – user2034412 Sep 14 '15 at 1:18
  • $\begingroup$ One possible conclusion is that if one runs enough annalyses some are bound to look like there is something interesting, but that often such findings will turn out to be spurious and do not replicate. $\endgroup$ – Björn Sep 14 '15 at 5:25
  • $\begingroup$ I understand and appreciate your concern, but there is only one analysis (a full factorial linear model), and it has been applied in exactly the same way to both experiments. I only need to determine whether I can trust the interaction at the specified error level. $\endgroup$ – user2034412 Sep 14 '15 at 7:29
  • $\begingroup$ Multiplicity in terms of the familywise type I error rate does not arise from separate models, but rather from separate null hypotheses that could be rejected. Switching hypotheses (or adding additional hypotheses) after seeing the data will no control the familywise type I error rate unless you had a pre-specified plan for how to do this up-front in a manner that controls the type I error rate (e.g. with some pre-specified multiplicity adjustment). Unless that was the case, formally one cannot say the null hypotheses of no interaction has not been rejected at any type I error level. $\endgroup$ – Björn Sep 14 '15 at 11:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.