# Why haven't robust (and resistant) statistics replaced classical techniques?

When solving business problems using data, it's common that at least one key assumption that under-pins classical statistics is invalid. Most of the time, no one bothers to check those assumptions so you never actually know.

For instance, that so many of the common web metrics are "long-tailed" (relative to the normal distribution) is, by now, so well documented that we take it for granted. Another example, online communities--even in communities with thousands of members, it's well-documented that by far the largest share of contribution to/participation in many of these community is attributable to a minuscule group of 'super-contributors.' (E.g., a few months ago, just after the SO API was made available in beta, a StackOverflow member published a brief analysis from data he collected through the API; his conclusion--less than one percent of the SO members account for most of the activity on SO (presumably asking questions, and answering them), another 1-2% accounted for the rest, and the overwhelming majority of the members do nothing).

Distributions of that sort--again more often the rule rather than the exception--are often best modeled with a power law density function. For these type of distributions, even the central limit theorem is problematic to apply.

So given the abundance of populations like this of interest to analysts, and given that classical models perform demonstrably poorly on these data, and given that robust and resistant methods have been around for a while (at least 20 years, i believe)--why are they not used more often? (I am also wondering why I don't use them more often, but that's not really a question for CrossValidated.)

Yes i know that there are textbook chapters devoted entirely to robust statistics and i know there are (a few) R Packages (robustbase is the one i am familiar with and use), etc.

And yet given the obvious advantages of these techniques, they are often clearly the better tools for the job--why are they not used much more often? Shouldn't we expect to see robust (and resistant) statistics used far more often (perhaps even presumptively) compared with the classical analogs?

The only substantive (i.e., technical) explanation i have heard is that robust techniques (likewise for resistant methods) lack the power/sensitivity of classical techniques. I don't know if this is indeed true in some cases, but i do know it is not true in many cases.

A final word of preemption: yes I know this question does not have a single demonstrably correct answer; very few questions on this Site do. Moreover, this question is a genuine inquiry; it's not a pretext to advance a point of view--i don't have a point of view here, just a question for which i am hoping for some insightful answers.

• The Black Swann by Nassim Nicholas Taleb explains why simple models have been used in the financial world and the dangers this has led to. A particular fault is equating very-low probabilities with zero and blindly applying the normal distribution in risk management! – James Aug 4 '10 at 13:38
• Tests relying on many assumptions are more powerful when those assumptions are satisfied. We can test for significance of deviation assuming that observations are IID Gaussian, which gives mean as statistic. A less restrictive set of assumptions tells us to use median. We can go further and assume that observations are correlated to get even more robustness. But each step reduces the power of our test, and if we make no assumptions at all, our test is useless. Robust tests implicitly make assumptions about data and are better than classical only when those assumptions match reality better – Yaroslav Bulatov Sep 19 '10 at 23:20

Researchers want small p-values, and you can get smaller p-values if you use methods that make stronger distributional assumptions. In other words, non-robust methods let you publish more papers. Of course more of these papers may be false positives, but a publication is a publication. That's a cynical explanation, but it's sometimes valid.

• "sometimes" is an understatement... the authors logic isn't often this direct but the stimulus / reward scenario is such that people will do this as a matter of conditioning – John Aug 3 '10 at 13:52
• I don't researchers are being dishonest so much as acting out of ignorance. They don't understand what statistics mean or what assumptions they require, but as you said they clearly understand the stimulus/reward: p > 0.05 => no publication. – John D. Cook Aug 3 '10 at 14:44
• You must also present something that those "in power" (decision makers, supervisors, reviewers) understand. Therefore it has to be in the common language which evolves quite slowly, as those people tend to be older and more resistant to change, largely as it may invalidate their careers hitherto! – James Aug 4 '10 at 13:29
• Good point. "I understand p-values. Just give me a p-value." Ironically, they probably do not understand p-values, but that's another matter. – John D. Cook Aug 5 '10 at 0:22
• I don't believe this is categorically true. At least, I've heard modern nonparametrics often sacrifice very little power, if any. AFAIK, power loss is most pronounced in tests involving rank transformations, which are hardly ubiquitous among robust methods. – Nick Stauner Jun 2 '14 at 23:19

So 'classical models' (whatever they are - I assume you mean something like simple models taught in textbooks and estimated by ML) fail on some, perhaps many, real world data sets.

If a model fails then there are two basic approaches to fixing it:

1. Make fewer assumptions (less model)
2. Make more assumptions (more model)

Robust statistics, quasi-likelihood, and GEE approaches take the first approach by changing the estimation strategy to one where the model does not hold for all data points (robust) or need not characterize all aspects of the data (QL and GEE).

The alternative is to try to build a model that explicitly models the source of contaminating data points, or the aspects of the original model that seems to be false, while keeping the estimation method the same as before.

Some intuitively prefer the former (it's particularly popular in economics), and some intuitively prefer the latter (it's particular popular among Bayesians, who tend to be happier with more complex models, particularly once they realize they're going to have use simulation tools for inference anyway).

Fat tailed distributional assumptions, e.g. using the negative binomial rather than poisson or t rather than normal, belong to the second strategy. Most things labelled 'robust statistics' belong to the first strategy.

As a practical matter, deriving estimators for the first strategy for realistically complex problems seems to be quite hard. Not that that's a reason for not doing so, but it is perhaps an explanation for why it isn't done very often.

• +1. Very good explanation. I also think that some "robust" methods are rather ad hoc (truncated means), and that "robust" is tied to a particular aspect of a method and is not a general quality but many people interpret "robust" to mean "I don't have to worry about my data, since my method is robust". – Wayne May 16 '11 at 17:54
• Great answer. It bothers me that so many answers focus on the difficulty of understanding robust statistics or on the incentives for ignoring the breaching of assumptions. They ignore the people out there who know that there are cases when robust statistics are needed and when they are not. – Kenji Oct 3 '13 at 15:07

I would suggest that it's a lag in teaching. Most people either learn statistics at college or University. If statistics is not your first degree and instead did a mathematics or computer science degree then you probably only cover the fundamental statistics modules:

1. Probability
2. Hypothesis testing
3. Regression

This means that when faced with a problem you try and use what you know to solve the problem.

• Data isn't Normal - take logs.
• Data has annoying outliers - remove them.

Unless you stumble across something else, then it's difficult to do something better. It's really hard using Google to find something if you don't know what it's called!

I think with all techniques it will take a while before the newer techniques filter down. How long did it take standard hypothesis tests to be part of a standard statistics curriculum?

BTW, with a statistics degree there will still be a lag in teaching - just a shorter one!

• But this raises an interesting pedagogical problem, at least in Psychology, because as far as I know most introductory statistics books being used in my field do not really discuss robust measures except as an aside. – russellpierce Aug 4 '10 at 0:43
• That is very true, and also in psychology, there's an annoying confusion between non-parametric and non-normal, which seems to hinder understanding. – richiemorrisroe Jan 4 '11 at 10:58
• Some of us psychologists are just confused about everything statistical! :) – Nick Stauner Jun 2 '14 at 23:22

Anyone trained in statistical data analysis at a reasonable level uses the concepts of robust statistics on a regular basis. Most researchers know enough to look for serious outliers and data recording errors; the policy of removing suspect data points goes back well into the 19th century with Lord Rayleigh, G.G. Stokes, and others of their age. If the question is:

Why don't researchers use the more modern methods for computing location, scale, regression, etc. estimates?

then the answer is given above -- the methods have largely been developed in the last 25 years, say 1985 - 2010. The lag for learning new methods factors in, as well as inertia compounded by the 'myth' that there is nothing wrong with blindly using classical methods. John Tukey comments that just which robust/resistant methods you use is not important—what is important is that you use some. It is perfectly proper to use both classical and robust/resistant methods routinely, and only worry when they differ enough to matter. But when they differ, you should think hard.

Why don't researchers stop and ask questions about their data, instead of blindly applying highly unstable estimates?

then the answer really comes down to training. There are far too many researchers who were never trained in statistics properly, summed up by the general reliance on p-values as the be-all and end-all of 'statistical significance'.

@Kwak: Huber's estimates from the 1970s are robust, in the classical sense of the word: they resist outliers. And redescending estimators actually date well before the 1980s: the Princeton robustness study (of 1971) included the bisquare estimate of location, a redescending estimate.

• projecteuclid.org/… Freely available document written by Peter Huber on John Tukey's contributions to robust statistics. Reasonably easy read, light on the formulae. – Wesley Burr Aug 6 '10 at 3:08

Statistics is a tool for non-statistical-minded researchers, and they just don't care.

I once tried to help with a Medicine article my ex-wife was co-authoring. I wrote several pages describing the data, what it suggested, why certain observations had been excluded from the study... and the lead researcher, a doctor, threw it all away and asked someone to compute a p-value, which is all she (and just about everyone who would read the article) cared about.

I Give an answer in two directions:

1. things that are robust are not necessarily labeled robust. If you believe robustness against everything exists then you are naive.
2. Statistical approaches that leave the problem of robustness appart are sometime not adapted to the real world but are often more valuable (as a concept) than an algorithm that looks like kitchening.

developpment

First, I think there are a lot of good approaches in statistic (you will find them in R packages not necessarily with robust mentionned somewhere) which are naturally robust and tested on real data and the fact that you don't find algorithm with "robust" mentionned somewhere does not mean it is not robust. Anyway if you think being robust means being universal then you'll never find any robust procedure (no free lunch) you need to have some knowledge/expertise on the data you analyse in order to use adapted tool or to create an adapted model.

On the other hand, some approaches in statistic are not robust because they are dedicated to one single type of model. I think it is good sometime to work in a laboratory to try to understand things. It is also good to treat problem separatly to understand to what problem our solution is... this is how mathematician work. The example of Gaussian model elocant: is so much criticised because the gaussian assumption is never fulfilled but has bring 75% of the ideas used practically in statistic today. Do you really think all this is about writting paper to follow the publish or perish rule (which I don't like, I agree) ?

As someone who has learned a little bit of statistics for my own research, I'll guess that the reasons are pedagogical and inertial.

I've observed within my own field that the order in which topics are taught reflects the history of the field. Those ideas which came first are taught first, and so on. For people who only dip into stats for cursory instruction, this means they'll learn classical stats first, and probably last. Then, even if they learn more, the classical stuff with stick with them better due to primacy effects.

Also, everyone knows what a two sample t-test is. Less than everyone knows what a Mann-Whitney or Wilcoxon Rank Sum test is. This means that I have to exert just a little bit of energy on explaining what my robust test is, versus not having to exert any with a classical test. Such conditions will obviously result in fewer people using robust methods than should.

Wooldridge "Introductory Econometrics - A Modern Approach" 2E p.261.

If Heteroskedasticity-robust standard errors are valid more often than the usual OLS standard errors, why do we bother we the usual standard errors at all?...One reason they are still used in cross sectional work is that, if the homoskedasticity assumption holds and the erros are normally distributed, then the usual t-statistics have exact t distributions, regardless of the sample size. The robust standard errors and robust t statistics are justified only as the sample size becomes large. With small sample sizes, the robust t statistics can have distributions that are not very close to the t distribution, and that could throw off our inference. In large sample sizes, we can make a case for always reporting only the Heteroskedasticity-robust standard errors in cross-sectional applications, and this practice is being followed more and more in applied work.

While they're not mutually exclusive, I think the growing popularity of Bayesian statistics is part of it. Bayesian statistics can achieve a lot of the same goals through priors and model averaging, and tend to be a bit more robust in practice.

I'm not statistician, my experience in statistics is fairly limited, I just use robust statistics in computer vision/3d reconstruction/pose estimation. Here is my take on the problem from the user point of view:

First, robust statistics used a lot in engineering and science without calling it "robust statistics". A lot of people use it intuitively, coming to it in the process of adjusting specific method to real-world problem. For example iterative reweighted least squares and trimmed means/trimmed least square used commonly, that just the user don't know they used robust statistics - they just make method workable for real, non-synthetic data.

Second, both "intuitive" and conscious robust statistics practically always used in the case where results are verifiable, or where exists clearly visible error metrics. If result obtained with normal distribution are obviously non-valid or wrong, people start tinkering with weights, trimming,sampling, read some paper and end up using robust estimators, whether they know term or not. On the other hand if end result of research just some graphics and diagrams, and there is no insensitive to verify results, or if normal statistic produce reults good enough - people just don't bother.

And last, about usefulness of robust statistics as a theory - while theory itself is very interesting it's not often give any practical advantages. Most of robust estimators are fairly trivial and intuitive, often people reinventing them without any statistical knowledge. Theory, like breakdown point estimation, asymptotics, data depth, heteroskedacity etc allow deeper understanding of data, but in most cases it's just unnecessary. One big exception is intersection of robust statistics and compressive sensing, which produce some new practical methods such as "cross-and-bouquet"

My knowledge of robust estimators is solely in regards to robust standard errors for regression parameters so my comment will only be in regards to those. I would suggest people read this article,

On The So-Called "Huber Sandwich Estimator" and "Robust Standard Errors" by: Freedman, A. David The American Statistician, Vol. 60, No. 4. (November 2006), pp. 299-302. doi:10.1198/000313006X152207 (PDF Version)

Particular what I am concerned about with these approaches is not that they are wrong, but they are simply distracting from bigger problems. Thus I entirely agree with Robin Girard's answer and his mention of "no free lunch".

The calculus and probability needed for robust statistics is (usually) harder, so (a) there is less theory and (b) it is harder to grasp.

I am surprised to see the Gauss-Markov theorem is not mentioned in this long list of answers, afaics:

In a linear model with spherical errors (which along the way includes an assumption of no outliers, via a finite error variance), OLS is efficient in a class of linear unbiased estimators - there are (restrictive, to be sure) conditions under which "you can't do better than OLS".

I am not arguing this should justify using OLS almost all of the time, but it sure contributes to why (especially since it is a good excuse to focus so much on OLS in teaching).

• Well, yes, but that assumes that minimizing variance is the relevant criterion, and with heavy tails, it might not be so! – kjetil b halvorsen Apr 13 '15 at 12:35
• Sure. I just wanted to add what I believe is maybe the most famous reason to think OLS is a useful technique to the list of understandable reasons why robust techniques haven't replaced it: there are cases where you shouldn't replace it. – Christoph Hanck Apr 13 '15 at 13:50

My guess would be that robust statistics are never sufficient i.e. to be robust these statistics skip some of the information about the distribution. And I suspect that it is not always a good thing. In other words there's a trade-off between robustness and loss of information.

E.g. the median is robust because (unlike the mean) it utilizes information only about half of the elements (in discrete case): $$median(\{1, 2, 3, 4, 5\})=3=median(\{0.1, 0.2, 3, 4000, 5000\})$$