What does it mean for a statistical test to be "robust"?

Is there an intuitive way of understanding what these two sentences mean and why they're true?:

"ANOVA is 'robust' to deviations from normality with large samples", and... "ANOVA is 'robust' to heteroscedasticity if the groups have similar sample sizes".

• Commented Dec 3, 2021 at 22:53
• I'm not so sure about that last statement: I suspect a limited sense of "heteroscedasticity" and a limited sense of "robust" must be applied for it to be true. ANOVA, after all, comprises a lot of different things, including estimating group means, analyzing components of variances, and testing differences among group means. The p-values for those latter tests will be rather wrong with sufficiently large discrepancies in group variances, regardless of group sample sizes.
– whuber
Commented Dec 3, 2021 at 23:01
• Although almost everyone follows almost everybody else in using the language of assumptions, I think it's a misleading word to use. I prefer to explain in terms of ideal condition. Commented Dec 5, 2021 at 13:28
• Another partly terminological problem: the mainstream statistical use refers to procedures where the context is worrying about outliers and long tails most of all. In economics and related subjects robust often refers to working well in the presence of heteroscedasticity etc. as in the use of robust to refer to Eicker-Huber-White standard errors (terminology is far from standardized). Commented Dec 5, 2021 at 13:39

Roughly speaking, a test or estimator is called 'robust' if it still works reasonably well, even if some assumptions required for its theoretical development are not met in practice. Comments:

• If you need to do one-factor ("one-way") ANOVA for data with different variances at each level of the factor, then it is best to use some variant of one-way ANOVA such as oneway.test in R that does not require equal variances. As you say, a 'pooled' t test or simple one-way ANOVA where the numbers of replications per factor differ greatly, may be problematic if variances also differ among levels of the factor.

• Some texts seem to say 2-sample t test and one-way ANOVA are OK for non-normal data whenever there are more than 30 replications per group. But this may not be true if data within groups are highly skewed.

• If levels of 2-sample t or one-factor ANOVA are far from normal, but differences between groups are mainly a 'shift' of location (with little change in shape or variance) then it may be best to use Welch t test or Kruskal-Wallis nonparametric test instead of t or ANOVA, respectively.

Note: I could show an example to illustrate, if you could say what test is of particular interest and what assumption you feel unsure of.

When we say that a procedure is "robust" or "robust to [a particular failure of assumption]" we mean that the procedure still works well when the underlying assumption is not met. So, in the present case, the quoted statement is telling you that, under the stipulated conditions, ANOVA still works well even when the normality or homoskedasticity conditions in a model are not a realistic reflection of the data.

• This answers the "robust" part of the question, but not the "why" question. Why is ANOVA robust to deviations from normality or heteroscedasticity under similar sample sizes? Commented Dec 3, 2021 at 22:58

We must be specific about what the claim is. It's not sufficient to wave our hands and say something vague like the test "works well" in those circumstances -- that is not what was examined in order to make the statement.

Both statements are specifically about accuracy of the significance level (a.k.a. "level-robustness").

That is to say, the type I error rate is claimed not to be too far from what you would calculate/choose under the (violated) assumption in those circumstances.

Even in that restricted sense, these sorts of general claims are too vague to be useful in practice, however. For example, you don't really know how large is sufficiently large for your purposes in the first case, because you don't know the population distribution (if you did, you wouldn't need to consider this issue at all!).

Of course, significance level is not the only consideration with tests. Certainly I'd hope that people care about power. Sadly, however, the direct evidence that the people who repeat these statements care much in practice is weak when common statements like these so rarely are accompanied by the merest mention of what happens with power.

In the first case, large samples don't save you when you're looking at relative efficiency (the relative sample sizes needed to achieve a given level of power) -- and relative efficiency can be arbitrarily poor in large samples -- so if your sample sizes were large because your anticipated effect size was small, you might have some potentially serious issues.

• This answer has some good points, but some bad points too. In particular, I don't agree with the view that every statistical concept must be specific --- surely there is a place for qualitative concepts like "robustness" where you are non-specific about exactly what it means for the thing to "work well". Similarly, saying that a statement is too vague to be useful if it does not specify how large is "sufficiently large" for a specific quantitative result also seems like a major exaggeration, and inconsistent with common use of limiting results.
– Ben
Commented Dec 6, 2021 at 20:36
• The objection is to saying something "works" while leaving half the task of working at all utterly unexamined, which is generally what is done when making the judgements given in the question. If we are not specific about what we mean, we are very likely to mislead and that is very often worse than nothing. On the second thing ... you're going to have to explain how a claim about large sample results is useful in practice if we don't mention the basis on which the naive user can tell if $n=5$, or $n=5000$ or $n=5$ billion is needed for reasonable performance. ... Commented Dec 7, 2021 at 2:25
• .... It's not a problem for a user who can address the issues and come up with some idea in a particular situation, but it can very much be a problem for a user who does not. The usual rules of thumb that are offered are (again) very often misleading (I have on hundreds of occasions see people avoid a perfectly reasonable analysis at say n=20 or n=25 (substituting one that doesn't address their original hypothesis) and many dozens of times seen a blithe application of a very unsuitable analysis at a slightly larger sample size. Commented Dec 7, 2021 at 2:32
• I'm very happy to be disagreed with, though, since it's often the case that in such discussions that something fruitful arises. Commented Dec 7, 2021 at 2:33
• Again, I agree with much of this (though even a novice wouldn't confuse $n=5$ for a "large sample size"). At an overarching level, I just think that there is a place for generic statements like, "ANOVA is robust to deviations from normality with large samples". That statement seem to me to capture useful information about ANOVA, even though it comes with no elaboration on what "robust" or "large samples" actually means in a quantitative sense.
– Ben
Commented Dec 7, 2021 at 5:13