If we are taught to not remove outliers without investigation, how do robust methods (median, trimmed mean) can be even suggested?

Question

I just saw an article, which taught to nor remove outliers without investigation, because it may be a unusual but valid observation or naturally skewed data, for example in chemistry or medicine. Only true errors were advised to remove. But how does this relate to using quartile based methods, like quantile regression, median or trimmed means based methods?

Trimmed mean is nothing but a mean calculated on data with removed XX% of observations from both sides. Typically 10% - 20%. Together it makes 20% - 40% of observations removed. And, usually, it removes most of the skewness, making symmetric, or even Gaussian data from, say, log-normal. So it changes everything!

Median is even "worse" - it removes 99% of the data, as only the mid point, or the average of two mids points, is returned. 99% of the data are totally ignored. From the other side, the median is equal to the geometric mean in log-normal distribution, so maybe it is justified?

Quantile regression is based on quantiles, including the median, so it share the same behavior.

My question is how is using those methods different from just automatically removing everything that is beyond some threshold (like 3 times standard deviations, or certain quantile)? We "don't delete outliers", using robust methods, which removes much more, even 40% (trimmed) or 99% (median) of the data! Isn't this all cheating?

Answer 1

Removal of 'outliers' without a good reason is different from systematic trimming, which does actually take all the data into account.

For illustration, I use a random sample of data from a gamma distribution with mean $\mu = 50$ and a median $\eta \approx 46.7.$

Here is a random sample of 100 observations:

set.seed(515)
x = rgamma(100, 5, .1)
mean(x); mean(x, trim=.05)
[1] 48.47809   # regular mean
[1] 47.52384   # 5% trimmed mean
median(x)
[1] 46.76814

Trimming has disregarded exact magnitudes of about ten observations. 5 from each end of the distribution.

The mean $48.5$ and the trimmed mean $47.5$ are not much different. The median $46.8$ (which you might consider as a 50% trimmed mean) is a little smaller than the mean (because the distribution is mildly skewed), but not very much smaller. And it is good estimate of the population median.

boxplot(x, horizontal=T)

Now suppose I decide on a whim that it's 'best' to remove the values above 70.

y = x[x <= 70]
mean(y)
[1] 44.48163
length(y)
[1] 91

I have removed only nine observations, but in a destructive way: all from the right tail. Now the mean $44.5$ of the remaining 91 observations is considerably lower than the original mean.

The point is that the various robust measures, which you say do not use all of the data, are actually influenced in a balanced way by the 'unused' part of the data. (The median would not be itself without half of the observations on either side of it.)

Answer 2

Taking the median is not cheating. Taking the median and calling it the arithmetic mean is cheating. Removing outliers without strong justification is more like the latter. It's also not quite right to say that 99% of the data is "ignored". The data is used to form the median. What is ignored is how far from the median the data are.

Quantile regression changes the question from regular regression. In many cases, median regression will give very similar results to regular OLS regression, but two questions can get the same answer. Like some other robust methods, it has somewhat lower power if all the assumptions of OLS regression are met, but, these days, we often have so much data that power is not a problem. Of course, with quantile regression you can also look at other quantiles.

Robust regression methods attempt to answer the same question as OLS regression, but, at least most of the methods do not remove data, they downweight some data.