Is a Type III statistical error just a subset of Type I or II error?

Question

I have recently come across the idea of a type III error through discussion with colleagues. The definition they gave me was something along these lines:

"Type III error occurs when you correctly conclude that the two groups are statistically different, but you are wrong about the direction of the difference. Say that a treatment increases some variable. But in your experiment, random sampling leads the value of that variable to be lower (on average) in the treated group, and enough lower that the difference is statistically significant."

To me, it sounds like

• A specific Type II error in a one-sided test

• Arguably, a specific Type I error in a two-sided test (a form of "False Positive", in the wrong direction)

Is type III considered a "real error", just like Type I or II, or just a subset of those two? I did not find literature on the topic.

Answer 1

While the phenomenon is distinct from a type I or II error, I think calling it a type III error is misleading, because it suggests some sort of similarity in interpretation.

This phenomenon arises only in null hypothesis significance testing. In the absence of a two-tailed null-hypothesis, (e.g., in binary classification) there exist only false positives and false negatives.

The other problem is that you can't do anything useful with this error. The rate at which it occurs directly depends on the chance of a type II error. Think of how you control error rates:

• Type I error rate: Increase your stringency for significance;
• Type II error rate: Increase sample size / use a more powerful test;
• Type III error rate: Increase sample size / be less unlucky?

A procedure for controlling this phenomenon would simply be avoiding sampling bias, so its rate isn't useful in comparing methods. Comparing the type I or II error rates, on the other hand, is useful when deciding which method is more lenient or more powerful, respectively.