When to remove outliers?

I am currently doing a linear regression model. At the suggestion of my professor, we have looked at Cook’s distance to identify outliers. Here is the Cook’s distance plot using R. From what I understand, this shows that points 6 and 24 are influential.

But how should that affect our analysis? Does this mean we should eliminate these points? According to our datasets background, the data is reliable. I read somewhere else that unless you have a specific reason to remove an outlier you should always keep it. Is this true?

• Yes, you should keep outliers, unless you have a very compelling reason not to keep them. If you have confidence that the data is good, you have to leave them in. Commented Apr 29, 2020 at 22:32
• I find it interesting that your apparent outliers are all too big instead of some being too small.
– Dave
Commented May 1, 2020 at 16:49

Outliers are not always a bad thing.

• Sometimes they reflect the stochastic nature of the data (e.g. data in finance tend to have heavy tails, and it is common to observe "outliers"),

• in other instances, they may be explained by covariates.

For example,

set.seed(1)
x = c(21,22,23,24,25,50)
y = 5 + 2*x + rnorm(length(x))
> y
[1]  46.37355  49.18364  50.16437  54.59528  55.32951 104.17953


One could think that the largest observation is an outlier, but it is clearly explained by the covariate $$x$$, and the residual errors are of course normal.

• In other cases the presence of outliers might be related to data quality (e.g. a typo).

• Among other possible reasons.

Thus, in general, it is better to reflect about potential reasons for having outliers, rather than automatically and blindly applying methods to detect outliers.

A nice quote from Andrew Gelman:

Stepwise regression is one of these things, like outlier detection and pie charts, which appear to be popular among non-statisticans but are considered by statisticians to be a bit of a joke.

Reference for the quote: https://statmodeling.stat.columbia.edu/2014/06/02/hate-stepwise-regression/

• +1 Nice first post! That R simulation is quite telling. I do wonder if that $x=50$ would be an outlier, but at least the further analysis of looking at $x=50$ instead of $y=104$ would provide different insights that do a better job of describing the data.
– Dave
Commented May 1, 2020 at 17:05
• @Dave Indeed, looking at the responses, one might think that 104.17 is an outlier, but then one can see that it is related to a large covariate value, which one could try to reflect about their presence. For instance, if $x$ is age, then it is possible that the covariates (age groups) were misclassified/mixed up. However, for fitting the data, it does not affect estimation. In general, there are a number of potential explanations for outliers, and finding out the reasons for their presence is often a task for Batman or Sherlock Holmes. Commented May 1, 2020 at 17:10

Just to expand on Liar Lier's answer, you first have to think about what is the goal of the data analysis / data modeling at hand - how do outliers play into the problem? Are they important? or are they a distraction?

For example, if the dataset you have is about earthquake magnitudes, then we don't actually care all that much about the non-outliers (frequent small magnitude earthquakes). In the case of earth quake detection, we are tasked specifically with predicting the outliers as the big earthquakes causes the most damage; therefore, removing outliers would be a mistake.

More generally, you should think about if the outliers is a important part of the underlying data generating process.

Building on the earthquake example, all the earth quake data is generated from the same data generating process, frequent small earth quakes and infrequent large earthquakes alike. In contrast, if the outliers were generated from a different process, say someone accidentally bumping into the detector, then the outliers are generated by a different process. This means if we are tasked with forecasting earthquakes, we don't want to include the "bumping into the machine" data into the training set as it is not the process of interest to model.

Consider robust Least Absolute Deviation regression, discussed here.

In the univariate case, the answer is the sample median. In a two-parameter regression model, the answer is a line passing through two points, which nearly always completely avoids outliers.

It can be computed by iteratively employing weighted least-squares. The weight for point 'i' is 1/(absolute value for the residue of point 'i' at the prior iteration).

Computing confidence intervals can be computed by bootstrap sampling from the observed regression model errors (see this source).

[EDIT] Yes, elaboration is likely needed. My opinion is that the examination and targeting of 'outliers' in a linear regression model beginning with two predictor variables and higher dimensions is, not only visually hard to discern, but generally requires some knowledge of the parent distribution generating the noise. If this could be analyzed with, say a Box-Cox Analysis of transformation, I will greatly favor a data transformation approach, that preserves the integrity of the data set.

In my answer, I assumed that this cannot be performed (or undesirable, as a data transformation may introduce interpretation issues). The data set itself may (or may not) be a mixture of distributions.

As such, I would suggest a robust methodology. LAD, in the univariate case, reduces to the median. I would not prefer to remove 'points' based on some criteria, on data for which I have little to no knowledge as to source, generation,... and proceed with great faith in the adjusted mean (or mean-based methodology) that then would be employed (which are known not to be robust to outliers).

Here are comments from Wikipedia on Cook's distance measure (which apparently preferential selects a median point from the F distribution for decision purposes):

Detecting highly influential observations

There are different opinions regarding what cut-off values to use for spotting highly influential points. Since Cook's distance is in the metric of an F distribution with p and n-p (as defined for the design matrix X above) degrees of freedom, the median point (i.e., $${\ F_{0.5}(p,n-p)}$$) can be used as a cut-off.[7] Since this value is close to 1 for large n, a simple operational guideline of $${D_i > 1}$$ has been suggested.[8] Note that the Cook's distance measure does not always correctly identify influential observations.[9]

Interestingly, as the Cook's distance measure is defined as a function of the sum of all the changes in the regression model point forecasts when a particular observation is removed, and as in LAD regression, two points (excluding degenerate cases) are selected to solely determine the regression line, this is could be viewed as operationally related to Cook's methodology. This follows as LAD operationally excludes exterior points with large non-zero absolute errors, as a criterion (since the latter are not included in the determination of the line). Viewing from the perspective of the univariate case, removing suspect large values, could bring a mean (the Least-Squares estimate), which was swayed by an extreme value, closer to the population's median (the LAD estimate).

• The questions asks how influential points might affect a linear regression model--and the use of Cook's distance implies it is a least squares fit. It takes a great leap to connect that question with your answer, indicating you ought to consider elaborating on it to explain how it's relevant.
– whuber
Commented Apr 29, 2020 at 23:23