Why haven't robust (and resistant) statistics replaced classical techniques?

Question

When solving business problems using data, it's common that at least one key assumption that under-pins classical statistics is invalid. Most of the time, no one bothers to check those assumptions so you never actually know.

For instance, that so many of the common web metrics are "long-tailed" (relative to the normal distribution) is, by now, so well documented that we take it for granted. Another example, online communities--even in communities with thousands of members, it's well-documented that by far the largest share of contribution to/participation in many of these community is attributable to a minuscule group of 'super-contributors.' (E.g., a few months ago, just after the SO API was made available in beta, a StackOverflow member published a brief analysis from data he collected through the API; his conclusion--less than one percent of the SO members account for most of the activity on SO (presumably asking questions, and answering them), another 1-2% accounted for the rest, and the overwhelming majority of the members do nothing).

Distributions of that sort--again more often the rule rather than the exception--are often best modeled with a power law density function. For these type of distributions, even the central limit theorem is problematic to apply.

So given the abundance of populations like this of interest to analysts, and given that classical models perform demonstrably poorly on these data, and given that robust and resistant methods have been around for a while (at least 20 years, I believe)--why are they not used more often? (I am also wondering why I don't use them more often, but that's not really a question for CrossValidated.)

Yes I know that there are textbook chapters devoted entirely to robust statistics and I know there are (a few) R Packages (robustbase is the one I am familiar with and use), etc.

And yet given the obvious advantages of these techniques, they are often clearly the better tools for the job--why are they not used much more often? Shouldn't we expect to see robust (and resistant) statistics used far more often (perhaps even presumptively) compared with the classical analogs?

The only substantive (i.e., technical) explanation I have heard is that robust techniques (likewise for resistant methods) lack the power/sensitivity of classical techniques. I don't know if this is indeed true in some cases, but I do know it is not true in many cases.

A final word of preemption: yes I know this question does not have a single demonstrably correct answer; very few questions on this Site do. Moreover, this question is a genuine inquiry; it's not a pretext to advance a point of view--I don't have a point of view here, just a question for which i am hoping for some insightful answers.

Answer 1

Researchers want small p-values, and you can get smaller p-values if you use methods that make stronger distributional assumptions. In other words, non-robust methods let you publish more papers. Of course more of these papers may be false positives, but a publication is a publication. That's a cynical explanation, but it's sometimes valid.

Answer 2

So 'classical models' (whatever they are - I assume you mean something like simple models taught in textbooks and estimated by ML) fail on some, perhaps many, real world data sets.

If a model fails then there are two basic approaches to fixing it:

1. Make fewer assumptions (less model)
2. Make more assumptions (more model)

Robust statistics, quasi-likelihood, and GEE approaches take the first approach by changing the estimation strategy to one where the model does not hold for all data points (robust) or need not characterize all aspects of the data (QL and GEE).

The alternative is to try to build a model that explicitly models the source of contaminating data points, or the aspects of the original model that seems to be false, while keeping the estimation method the same as before.

Some intuitively prefer the former (it's particularly popular in economics), and some intuitively prefer the latter (it's particular popular among Bayesians, who tend to be happier with more complex models, particularly once they realize they're going to have use simulation tools for inference anyway).

Fat tailed distributional assumptions, e.g. using the negative binomial rather than poisson or t rather than normal, belong to the second strategy. Most things labelled 'robust statistics' belong to the first strategy.

As a practical matter, deriving estimators for the first strategy for realistically complex problems seems to be quite hard. Not that that's a reason for not doing so, but it is perhaps an explanation for why it isn't done very often.

Answer 3

I would suggest that it's a lag in teaching. Most people either learn statistics at college or University. If statistics is not your first degree and instead did a mathematics or computer science degree then you probably only cover the fundamental statistics modules:

1. Probability
2. Hypothesis testing
3. Regression

This means that when faced with a problem you try and use what you know to solve the problem.

• Data isn't Normal - take logs.
• Data has annoying outliers - remove them.

Unless you stumble across something else, then it's difficult to do something better. It's really hard using Google to find something if you don't know what it's called!

I think with all techniques it will take a while before the newer techniques filter down. How long did it take standard hypothesis tests to be part of a standard statistics curriculum?

BTW, with a statistics degree there will still be a lag in teaching - just a shorter one!

Answer 4

Statistics is a tool for non-statistical-minded researchers, and they just don't care.

I once tried to help with a Medicine article my ex-wife was co-authoring. I wrote several pages describing the data, what it suggested, why certain observations had been excluded from the study... and the lead researcher, a doctor, threw it all away and asked someone to compute a p-value, which is all she (and just about everyone who would read the article) cared about.

Answer 5

Anyone trained in statistical data analysis at a reasonable level uses the concepts of robust statistics on a regular basis. Most researchers know enough to look for serious outliers and data recording errors; the policy of removing suspect data points goes back well into the 19th century with Lord Rayleigh, G.G. Stokes, and others of their age. If the question is:

Why don't researchers use the more modern methods for computing location, scale, regression, etc. estimates?

then the answer is given above -- the methods have largely been developed in the last 25 years, say 1985 - 2010. The lag for learning new methods factors in, as well as inertia compounded by the 'myth' that there is nothing wrong with blindly using classical methods. John Tukey comments that just which robust/resistant methods you use is not important—what is important is that you use some. It is perfectly proper to use both classical and robust/resistant methods routinely, and only worry when they differ enough to matter. But when they differ, you should think hard.

If instead, the question is:

Why don't researchers stop and ask questions about their data, instead of blindly applying highly unstable estimates?

then the answer really comes down to training. There are far too many researchers who were never trained in statistics properly, summed up by the general reliance on p-values as the be-all and end-all of 'statistical significance'.

@Kwak: Huber's estimates from the 1970s are robust, in the classical sense of the word: they resist outliers. And redescending estimators actually date well before the 1980s: the Princeton robustness study (of 1971) included the bisquare estimate of location, a redescending estimate.

Answer 6

I Give an answer in two directions:

1. things that are robust are not necessarily labeled robust. If you believe robustness against everything exists then you are naive.
2. Statistical approaches that leave the problem of robustness appart are sometime not adapted to the real world but are often more valuable (as a concept) than an algorithm that looks like kitchening.

developpment

First, I think there are a lot of good approaches in statistic (you will find them in R packages not necessarily with robust mentionned somewhere) which are naturally robust and tested on real data and the fact that you don't find algorithm with "robust" mentionned somewhere does not mean it is not robust. Anyway if you think being robust means being universal then you'll never find any robust procedure (no free lunch) you need to have some knowledge/expertise on the data you analyse in order to use adapted tool or to create an adapted model.

On the other hand, some approaches in statistic are not robust because they are dedicated to one single type of model. I think it is good sometime to work in a laboratory to try to understand things. It is also good to treat problem separatly to understand to what problem our solution is... this is how mathematician work. The example of Gaussian model elocant: is so much criticised because the gaussian assumption is never fulfilled but has bring 75% of the ideas used practically in statistic today. Do you really think all this is about writting paper to follow the publish or perish rule (which I don't like, I agree) ?

Answer 7

As someone who has learned a little bit of statistics for my own research, I'll guess that the reasons are pedagogical and inertial.

I've observed within my own field that the order in which topics are taught reflects the history of the field. Those ideas which came first are taught first, and so on. For people who only dip into stats for cursory instruction, this means they'll learn classical stats first, and probably last. Then, even if they learn more, the classical stuff with stick with them better due to primacy effects.

Also, everyone knows what a two sample t-test is. Less than everyone knows what a Mann-Whitney or Wilcoxon Rank Sum test is. This means that I have to exert just a little bit of energy on explaining what my robust test is, versus not having to exert any with a classical test. Such conditions will obviously result in fewer people using robust methods than should.

Answer 8

Wooldridge "Introductory Econometrics - A Modern Approach" 2E p.261.

If Heteroskedasticity-robust standard errors are valid more often than the usual OLS standard errors, why do we bother we the usual standard errors at all?...One reason they are still used in cross sectional work is that, if the homoskedasticity assumption holds and the erros are normally distributed, then the usual t-statistics have exact t distributions, regardless of the sample size. The robust standard errors and robust t statistics are justified only as the sample size becomes large. With small sample sizes, the robust t statistics can have distributions that are not very close to the t distribution, and that could throw off our inference. In large sample sizes, we can make a case for always reporting only the Heteroskedasticity-robust standard errors in cross-sectional applications, and this practice is being followed more and more in applied work.

Answer 9

While they're not mutually exclusive, I think the growing popularity of Bayesian statistics is part of it. Bayesian statistics can achieve a lot of the same goals through priors and model averaging, and tend to be a bit more robust in practice.

Answer 10

I'm not statistician, my experience in statistics is fairly limited, I just use robust statistics in computer vision/3d reconstruction/pose estimation. Here is my take on the problem from the user point of view:

First, robust statistics used a lot in engineering and science without calling it "robust statistics". A lot of people use it intuitively, coming to it in the process of adjusting specific method to real-world problem. For example iterative reweighted least squares and trimmed means/trimmed least square used commonly, that just the user don't know they used robust statistics - they just make method workable for real, non-synthetic data.

Second, both "intuitive" and conscious robust statistics practically always used in the case where results are verifiable, or where exists clearly visible error metrics. If result obtained with normal distribution are obviously non-valid or wrong, people start tinkering with weights, trimming,sampling, read some paper and end up using robust estimators, whether they know term or not. On the other hand if end result of research just some graphics and diagrams, and there is no insensitive to verify results, or if normal statistic produce reults good enough - people just don't bother.

And last, about usefulness of robust statistics as a theory - while theory itself is very interesting it's not often give any practical advantages. Most of robust estimators are fairly trivial and intuitive, often people reinventing them without any statistical knowledge. Theory, like breakdown point estimation, asymptotics, data depth, heteroskedacity etc allow deeper understanding of data, but in most cases it's just unnecessary. One big exception is intersection of robust statistics and compressive sensing, which produce some new practical methods such as "cross-and-bouquet"

Answer 11

My knowledge of robust estimators is solely in regards to robust standard errors for regression parameters so my comment will only be in regards to those. I would suggest people read this article,

On The So-Called "Huber Sandwich Estimator" and "Robust Standard Errors" by: Freedman, A. David The American Statistician, Vol. 60, No. 4. (November 2006), pp. 299-302. doi:10.1198/000313006X152207 (PDF Version)

Particular what I am concerned about with these approaches is not that they are wrong, but they are simply distracting from bigger problems. Thus I entirely agree with Robin Girard's answer and his mention of "no free lunch".

Answer 12

The calculus and probability needed for robust statistics is (usually) harder, so (a) there is less theory and (b) it is harder to grasp.

Answer 13

I am surprised to see the Gauss-Markov theorem is not mentioned in this long list of answers, afaics:

In a linear model with spherical errors (which along the way includes an assumption of no outliers, via a finite error variance), OLS is efficient in a class of linear unbiased estimators - there are (restrictive, to be sure) conditions under which "you can't do better than OLS".

I am not arguing this should justify using OLS almost all of the time, but it sure contributes to why (especially since it is a good excuse to focus so much on OLS in teaching).

Answer 14

My guess would be that robust statistics are never sufficient i.e. to be robust these statistics skip some of the information about the distribution. And I suspect that it is not always a good thing. In other words there's a trade-off between robustness and loss of information.

E.g. the median is robust because (unlike the mean) it utilizes information only about half of the elements (in discrete case): $$median(\{1, 2, 3, 4, 5\})=3=median(\{0.1, 0.2, 3, 4000, 5000\})$$