1
$\begingroup$

I recently read about contrasts and how to code them. Contrasts can be orthogonal which means that they are independent and the statistical tests for them are not correlated. Normally, when doing multiple tests on the same data the family-wise error rate increases. It is possible to calculate this increase if the tests are not correlated (then the alpha-levels of each test can be multiplied to get the new probability of making a type 1 error). However, I read in Field (2018) that if contrasts are orthogonal then Type 1 error-rate is controlled. Does this mean that the family-wise error rate does not increase when you test the orthogonal contrasts with a statistical test? As I said before, I thought there was always an increase when multiple tests are done on the same data. So I am a little bit confused.

Reference

  • Discovering Statistics using IBM SPSS Statistics, Field, 2018, 5th edition (see chapter 12 and specifically page 544-546)
$\endgroup$
5
  • 1
    $\begingroup$ You are intuition is correct. Yes, we need to correct of multiple testing. Can you please point to the reference/paper "Field (2013)" you refer at so we can contextualise it better? $\endgroup$
    – usεr11852
    Oct 15, 2020 at 12:37
  • $\begingroup$ I added the reference, hopefully this helps. Since it is a book and not a paper, if you need screenshots of the pages then I can send them to you. $\endgroup$ Oct 15, 2020 at 13:13
  • 1
    $\begingroup$ This is an error of fact in Field. There are others. $\endgroup$ Oct 15, 2020 at 15:17
  • $\begingroup$ Are you 100% sure about this? Because he mentions it at least twice I think. $\endgroup$ Oct 15, 2020 at 16:05
  • $\begingroup$ @BigBendRegion makes a polite understatement. Field is the #1 source of erroneous information in questions here on CV. $\endgroup$
    – whuber
    Apr 22, 2022 at 22:04

2 Answers 2

2
$\begingroup$

I have not read the original text (Field), but I find that when there is disagreement, or just when I want to understand something better, then running some simulations can be very enlightening.

Here is some quick (quick to write, the actual running did take a couple of minutes) and simple R code to simulate when the Null for each contrast is TRUE (using Helmert contrasts since they are orthogonal):

tmpfun <- function() {
  y <- rnorm(10000, 0, 1)
  x <- factor(rep(1:10, each=1000))
  fit <- lm(y ~ C(x, contr.helmert))
  summary(fit)$coefficients[2:10, 4]
}

out1 <- replicate(10000, tmpfun())

# type I error rate at individual contrast level
mean(out1 <= 0.05)

# family type I error rate
mean(apply(out1, 2, function(x) any(x <= 0.05)))

# family type I error rate with Bonferroni correction
mean(apply(out1, 2, function(x) any(x <= 0.05/9)))

When I ran the above code (your results will likely differ a little due to different random seeds) it computed the individual error rate at 0.0499 (pretty close to 5% for $\alpha=0.05$. But the familywise error rate (at least one significant value out of the 9) was 0.365 (theory puts it at about 0.36975). I think that is fairly strong evidence that orthogonal contrasts do not control the family wise error rate (at least as I interpreted it).

But adjustments like the Bonferroni work well with orthogonal/independent contrasts and the adjusted familywise result was 0.0512 (again very close to what it should be). Perhaps this is what the text book author intended (and stated elsewhere).

A nice thing about simulations is that you can run the above code yourself, as well as changing many of the numbers that I used to try it under different conditions to see what changes and what does not.

$\endgroup$
0
$\begingroup$

It appears that advice on this differs by discipline. The replies to your answer see this as an error, yet, the author of a well-regarded textbook on experiment design in behavioral research, Roger Kirk, states that "If orthogonal contrasts have been planned in advance, contemporary practice favors adopting the contrast as the conceptual unit for a Type I error." He also cited, “Research by Norton and Bulgren, as cited by Games (1971), indicates that when the degrees of freedom for MSerror are moderately large, say 40, multiple t tests can, for all practical purposes, be regarded as independent.” (section 5.2).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.