Orthogonal contrasts and the family-wise error rate

Question

I recently read about contrasts and how to code them. Contrasts can be orthogonal which means that they are independent and the statistical tests for them are not correlated. Normally, when doing multiple tests on the same data the family-wise error rate increases. It is possible to calculate this increase if the tests are not correlated (then the alpha-levels of each test can be multiplied to get the new probability of making a type 1 error). However, I read in Field (2018) that if contrasts are orthogonal then Type 1 error-rate is controlled. Does this mean that the family-wise error rate does not increase when you test the orthogonal contrasts with a statistical test? As I said before, I thought there was always an increase when multiple tests are done on the same data. So I am a little bit confused.

Reference

• Discovering Statistics using IBM SPSS Statistics, Field, 2018, 5th edition (see chapter 12 and specifically page 544-546)

Answer 1

I have not read the original text (Field), but I find that when there is disagreement, or just when I want to understand something better, then running some simulations can be very enlightening.

Here is some quick (quick to write, the actual running did take a couple of minutes) and simple R code to simulate when the Null for each contrast is TRUE (using Helmert contrasts since they are orthogonal):

tmpfun <- function() {
  y <- rnorm(10000, 0, 1)
  x <- factor(rep(1:10, each=1000))
  fit <- lm(y ~ C(x, contr.helmert))
  summary(fit)$coefficients[2:10, 4]
}

out1 <- replicate(10000, tmpfun())

# type I error rate at individual contrast level
mean(out1 <= 0.05)

# family type I error rate
mean(apply(out1, 2, function(x) any(x <= 0.05)))

# family type I error rate with Bonferroni correction
mean(apply(out1, 2, function(x) any(x <= 0.05/9)))

When I ran the above code (your results will likely differ a little due to different random seeds) it computed the individual error rate at 0.0499 (pretty close to 5% for $\alpha=0.05$. But the familywise error rate (at least one significant value out of the 9) was 0.365 (theory puts it at about 0.36975). I think that is fairly strong evidence that orthogonal contrasts do not control the family wise error rate (at least as I interpreted it).

But adjustments like the Bonferroni work well with orthogonal/independent contrasts and the adjusted familywise result was 0.0512 (again very close to what it should be). Perhaps this is what the text book author intended (and stated elsewhere).

A nice thing about simulations is that you can run the above code yourself, as well as changing many of the numbers that I used to try it under different conditions to see what changes and what does not.

Answer 2

It appears that advice on this differs by discipline. The replies to your answer see this as an error, yet, the author of a well-regarded textbook on experiment design in behavioral research, Roger Kirk, states that "If orthogonal contrasts have been planned in advance, contemporary practice favors adopting the contrast as the conceptual unit for a Type I error." He also cited, “Research by Norton and Bulgren, as cited by Games (1971), indicates that when the degrees of freedom for MSerror are moderately large, say 40, multiple t tests can, for all practical purposes, be regarded as independent.” (section 5.2).