1
$\begingroup$

I have recently come across the idea of a type III error through discussion with colleagues. The definition they gave me was something along these lines:

"Type III error occurs when you correctly conclude that the two groups are statistically different, but you are wrong about the direction of the difference. Say that a treatment increases some variable. But in your experiment, random sampling leads the value of that variable to be lower (on average) in the treated group, and enough lower that the difference is statistically significant."

To me, it sounds like

  • A specific Type II error in a one-sided test

  • Arguably, a specific Type I error in a two-sided test (a form of "False Positive", in the wrong direction)

Is type III considered a "real error", just like Type I or II, or just a subset of those two? I did not find literature on the topic.

$\endgroup$
1
  • 3
    $\begingroup$ That's odd; I thought that Type III error was conventionally "the right answer to the wrong question"/"using the wrong model", but it looks like there are multiple definitions; mine appears to have come from Kimball (perhaps itself broadening Mosteller's definition), perhaps with some Mitroff and Featheringham mixed in, while yours seems to be Kaiser's (itself apparently a narrowing of Mosteller's). See en.wikipedia.org/wiki/Type_III_error $\endgroup$
    – Glen_b
    Apr 28, 2019 at 8:28

1 Answer 1

2
$\begingroup$

While the phenomenon is distinct from a type I or II error, I think calling it a type III error is misleading, because it suggests some sort of similarity in interpretation.

This phenomenon arises only in null hypothesis significance testing. In the absence of a two-tailed null-hypothesis, (e.g., in binary classification) there exist only false positives and false negatives.

The other problem is that you can't do anything useful with this error. The rate at which it occurs directly depends on the chance of a type II error. Think of how you control error rates:

  • Type I error rate: Increase your stringency for significance;
  • Type II error rate: Increase sample size / use a more powerful test;
  • Type III error rate: Increase sample size / be less unlucky?

A procedure for controlling this phenomenon would simply be avoiding sampling bias, so its rate isn't useful in comparing methods. Comparing the type I or II error rates, on the other hand, is useful when deciding which method is more lenient or more powerful, respectively.

$\endgroup$
2
  • $\begingroup$ Andrew Gelman calls this a "type S error" (sign error) and has written much about it on his blog andrewgelman.com. $\endgroup$ Apr 28, 2019 at 12:15
  • $\begingroup$ Thank you for your answer! Does a more powerful test mechanically leads to more type III errors then? $\endgroup$
    – L. M.
    Apr 30, 2019 at 19:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.