# Difference between removing outliers and using Least Trimmed Squares?

In most cases we would be reluctant to remove outliers from the dataset just to get a better fit. Robust estimators such as Least Trimmed Squares are sometimes recommended in order to fit a regression line without the influence of outliers (or at least weighing them down).

I see that we are keeping the full dataset, so the outlier points will be present in summary statistics, plots, etc. But besides that, is there any other substantial difference between the two approaches? The usual criticisms on not considering datapoints that might be legitimate and correctly reflect the population seem to not be addressed, just circumvented with a formalized method that automates the process.

• Omitting outliers in effect gives those observations zero weight, regardless. If with LTS or similar method, some observations end with in effect zero weight that is a result contingent on the data and the method. It's not that different from say calculating the mean after omitting outliers and calculating a median. The median is contingent on the values of all the data even if in practice you would get the same result from many perturbed versions of the same dataset. Aug 13, 2020 at 13:01
• Yes, indeed I am aware that omitting them is equivalent to giving them zero weight. My question is then pointing more towards why is omitting outliers a generally criticized practice, while an automated method that does in practice the same is seen as a valid approach?
– Kuku
Aug 13, 2020 at 17:08
• You could say that taking a median is equivalent to omitting every value except one or two in the middle. So, why don't you? There is no deep problem in omitting outliers if they are all impossible values. But otherwise the motive for omitting outliers is often just that they are awkward for certain methods. It's better to change the methods than to change the data. Aug 13, 2020 at 17:38
• The median example is very telling. The way I see it the main criticism on outlier removal in studies would then be a matter of disclosure? With adjusted methods such as LTS making explicit the decisions and thresholds being used.
– Kuku
Aug 19, 2020 at 20:19

The reason is largely cultural, in my opinion. Well defined statistical methods are favored in science because they give a transparent analysis of the data. This is probably one of the reason that p-values are so popular.

When an outlier is excluded by a practitioner manually, there may be many factors that might lead to this judgement. A reader of the practitioner's research may need detailed and non-leading explanation before they understand the justification for exclusion of a data point.

In constrast, a method like LTS excludes points based on a clear algorithm. Once the tuning parameters, like the alpha level, are set, it is generally transparent as to why points are excluded. Full disclosure - to some extent the can is being kicked here - there are those selected values for the tuning parameters which still need to be justified. That is similar to the way that the 5% p-value level should be justified.

Besides a an algorithm that can be deconstructed to see why some points are excluded, there are some additional advantages to algorithms. Since substantial work has gone into the development of methods like LTS, some properties about it are already proven (like breakdown value, etc). There is no proof about properties of a person's justification for removing points.

In short, the substantial difference between algorithmic and manual outlier selection exists.

Let $$(X_i,Y_i),\dots,(X_n,Y_n)$$ be an sample. Let $$r_i^2(f)=(f(X_i)-Y_i)^2$$ Least Trimmed Squared can be written like that: $$\widehat f= \arg\min_{f \in \mathcal{F}} \sum_{i=1}^k r_{(i)}(f)^2$$ where the parenthesis means that we sorted the data $$r_{(1)}(f)\le \dots\le r_{(n)}(f)$$. It is adaptative to the data, we don't threshold at a given value we use the data to know which points are to be excluded and this exclusion depends on $$f$$ which is not the case when you do outlier removal. Here the outlier removal procedure is kind of embedded in the method and you can't decompose the procedure into two parts outliers removal and then estimation. In some non-complicated cases indeed this would give you the same value but when $$\mathcal{F}$$ is complicated, when the data are high-dimensional... this is not obvious that you would get the same thing.

Other more involved reasons is that an outlier will not have the same influence (as in influence function, if you are interested you can search this keyword). Suppose we are in a very simple case where $$f(x)$$ is a constant and call $$T(y_1,\dots,y_n)$$ the value of $$f(x)$$ for a given sample $$Y_i=y_i$$, it means that in fact you are searching for the mean of the distribution $$Y$$ and $$T(Y_1,\dots,Y_n)$$ is a (robust) estimator of the mean. Then, define for $$y\in \mathbb{R}$$ $$S(y)=|T(Y_1,\dots,Y_n)- T(Y_1,\dots,Y_{n-1},y)|$$ call this the sensitivity of $$T$$ it correspond to the change of value when changing $$Y_n$$ for an outlier situated in $$y$$. For least trimmed square estimator, $$S(\infty)$$ is not zero if, say $$r_{n}(f)=r_{(i)}(f)$$ for some $$i\le k$$.

In a few words, an outlier placed in a very big value will pull the estimator $$\widehat f$$ towards infinity, not a lot but a little and this means that the outlier has been taken into account and this is not true when using outlier removal techniques in which case you ignore outliers.