Is this an example of a multiple comparisons problem?

I am conducting a comprehensive research determining the effect of 3 independent variables on the metabolic rate of an organism. However, the set-up of my data analysis leads me to believe that I might be committing a multiple comparisons problem or some other violation.

Lets say that my dependent/numerical variable is rate, and the 3 independent variables are race (black and white), gender (male and female), and eye color (blue, green). All of my independent variables are dichotomous.

METHOD 1: performing TukeyHSD on all my samples

#####My entire compiled dataset consisting of 250 samples##########
    detach()
attach(mydata)
hist(rate)
compiled<-aov(rate~race*gender*eye)
summary(compiled)
TukeyHSD(compiled)


METHOD 2: Subsetting my data

#######Only Black males
    detach()
sub1<-subset(mydata,race!= "white")
sub2<-subset(sub1, gender!="females")
attach(sub2)
blackmales<-aov(rate~eye)
summary(blackmales)
cohen.d.formula(rate~eye,pooled=TRUE,paired=FALSE,na.rm=FALSE,
hedges.correction=FALSE,conf.level=0.95,noncentral=FALSE)

#######Only Black females
    detach()
sub3<-subset(mydata,race!= "whites")
sub4<-subset(sub3, gender!="males")
attach(sub4)
blackfemales<-aov(rate~eye)
summary(blackfemales)
cohen.d.formula(rate~eye,pooled=TRUE,paired=FALSE,na.rm=FALSE,
hedges.correction=FALSE,conf.level=0.95,noncentral=FALSE)

#######Only white males
    detach()
sub5<-subset(mydata,race!= "black")
sub6<-subset(sub5, gender!="females")
attach(sub6)
whitemales<-aov(rate~eye)
summary(whitemales)
cohen.d.formula(rate~eye,pooled=TRUE,paired=FALSE,na.rm=FALSE,
hedges.correction=FALSE,conf.level=0.95,noncentral=FALSE)

#######Only white females
    detach()
sub7<-subset(mydata,race!= "black")
sub8<-subset(sub7, gender!="males")
attach(sub8)
whitefemales<-aov(rate~eye)
summary(whitefemales)
cohen.d.formula(rate~eye,pooled=TRUE,paired=FALSE,na.rm=FALSE,
hedges.correction=FALSE,conf.level=0.95,noncentral=FALSE)

#######Only blue eyes males
    detach()
sub9<-subset(mydata,eye!= "green")
sub10<-subset(sub9, gender!="females")
attach(sub10)
bluemales<-aov(rate~race)
summary(bluemales)
cohen.d.formula(rate~race,pooled=TRUE,paired=FALSE,na.rm=FALSE,
hedges.correction=FALSE,conf.level=0.95,noncentral=FALSE)

#######Only blue eyes females
    detach()
sub11<-subset(mydata,eye!= "green")
sub12<-subset(sub11, gender!="males")
attach(sub12)
bluefemales<-aov(rate~race)
summary(bluefemales)
cohen.d.formula(rate~race,pooled=TRUE,paired=FALSE,na.rm=FALSE,
hedges.correction=FALSE,conf.level=0.95,noncentral=FALSE)

#######Only green eyes males
    detach()
sub13<-subset(mydata,eye!= "blue")
sub14<-subset(sub13, gender!="females")
attach(sub14)
greenmales<-aov(rate~race)
summary(greenmales)
cohen.d.formula(rate~race,pooled=TRUE,paired=FALSE,na.rm=FALSE,
hedges.correction=FALSE,conf.level=0.95,noncentral=FALSE)

#######Only green eyes females
    detach()
sub15<-subset(mydata,eye!= "blue")
sub16<-subset(sub15, gender!="males")
attach(sub16)
greenfemales<-aov(rate~race)
summary(greenfemales)
cohen.d.formula(rate~race,pooled=TRUE,paired=FALSE,na.rm=FALSE,
hedges.correction=FALSE,conf.level=0.95,noncentral=FALSE)


The last 4 subsets would follow this same format, but test rate~gender.

In most cases, the individual Anova among each subset varies from the pairwise analyses from the TukeyHSD. TukeyHSD on the entire dataset is not appropriate since race, gender, and eye color each are dichotomous variables. What analyses should I go for?Are the subsets a valid way of statistically analyzing my data, or is this a violation of multiple comparisons problem?

Thank you in advance, and I apologize for this confusing explanation.

some thoughts on this:

1. Ten years ago you would see the word "Bonferroni correction" in each and every observational study in general medical journals. You don't anymore. You rarely see anyone perform multiple comparison correction for observational studies. Please refer to Lancet, NEJM, JAMA (and other journals with in-house statisticians) to see for yourself that observational studies need not correct for multiple comparisons.

2. I interpret your study as an observational study. Observational studies can only be exploratory in nature. Drawing causal inferences can rarely be made, which is why the p-value has no great value. So correction for the multiple P-value is not really warranted. That's why you see this sentence very often " .. due to the exploratory nature of this study, correction for multiple comparisons were not made .. " these days..

3. Clinical trials is a whole different story. FDA and EMA requires correction for multiple comparison, and they're way of doing this is fairly cumbersome. Generally, those who are into clinical trials, they put enormous weight into their p-values and multiple correction. But they attempt att causal inference...

In other words, I wouldn't bother correcting for multiple comparisons. Put the effort into interpreting your findings, putting them into a context and discussing them instead. But thats just one opinion.

If by Tukey, you mean Tukey HSD, in its classic application it is designed for pairwise comparisons among a single factor (with 3 or more categories). This does not seem to be occurring in any of your comparisons. Further, it does not provide control of the family wise error rate across different factors or interactions.

Multiple comparison correction is, as the name of the concept suggests, a problem that relates to the number of comparisons that are performed. Your method 2 increases the number of comparisons, so suffers from even worse problems than method 1 based on your description (but, sharing a reproducible example of your problem would help avoid confusion).

The most straightforward approach for situations like these is often to compute all the p-values, and then use the False Discovery Rate correction.

• Hello Tim. Thank you for your reply. Yes, I did mean TukeyHSD, and now I see why it is not appropriate for my analysis. Each of my variables are binary. I guess my confusion is that I do not understand what constitutes as multiple comparisons. All the examples of multiple comparisons that I have seen have a single variable with more than 2 levels/groups being compared in different combinations. However, I am creating subsets so that a single combination is only tested once aagainst another single combination of variables. Can you please provide a code or function that can help me? – Ragnoraok Feb 21 '18 at 11:12

After much research on False Discovery Rates and Multiple Comparisons problems, I now see the flaws in my statistical approach. Here is my understanding on the topic.

Most parametric analyses, including ANOVA, work under the assumption that parametrics are performed simultaneously on a dataset. The corresponding p-value reveals the chance of committing a type I error, or falsely rejecting a null hypothesis. Most disciplines accept that a p-value under 0.05 indicates a statistically significant effect, or in other words there would be less than a 5 % chance that the interaction was random. For a given hypothesis, all analyses should be conducted simultaneously in order to adhere to the 0.05 chance of making a type I error, OR each the p-value for each pair-wise analysis should be adjusted to account for each statistical test.

Where I went wrong

By creating 12 separate subsets, and generating an isolated ANOVA in each, I increased the chances of acquiring false positives. This was an example of a multiple comparisons problem; I performed in 12 separate ANOVA, and erroneously used them to support my experimental hypothesis without adjusting for the number of tests. My experimental hypothesis was that race:gender:eye color had a significant impact on rate; for this hypothesis, the p-value cutoff was 0.05. However, each individual subset was performed under the assumption that it was its own independent analysis for separate hypotheses/conclusions. Thus, each ANOVA was run under the flawed premise that the probability of a false rejection was 0.05 for each.

In reality, my flawed approach introduced many false discoveries, because the p-value corresponding to my hypothesis was 0.05 multiplied by the number of ANOVA analyses pertaining to my hypothesis. In other words, I used each isolated ANOVA as a piece to complete the puzzle that was my experimental hypothesis, yet I did not adjust the data to account for that fact that there were 12 pieces. In essence, what I thought was a p-value cutoff of 0.05 was in actuality 0.05*12= 0.6. This meant that I had a 60 % chance to commit a type I error (falsely reject a null hypothesis).

There are 2 ways to solve this dilemma:

1) Perform a TukeyHSD analysis on the entire dataset. TukeyHSD corrects the p-value to account for each variable.

2) Perform pair-wise analyses for each condition and then adjust the p-value for the # of statistical tests. In this case, my subset method can still be used. However, the p-value should be corrected using Bonferroni or another p-value adjustment, because these corrections adjust a significant value to be proportionally more stringent to the number of analyses/tests.

    #######Only Black males
detach()
sub1<-subset(mydata,race!= "white")
sub2<-subset(sub1, gender!="females")
attach(sub2)
blackmales<-aov(rate~eye)
summary(blackmales)


The generated p-value is 0.04 for black:males. This only pertains to a single, isolated ANOVA and should NOT be used for pair-wise comparisons for rate~race:gender:color

p.adjust.M <- p.adjust.methods[p.adjust.methods != "fdr"]
12))