Whether to delete cases that are flagged as outliers by statistical software when performing multiple regression?

Question

I am performing multiple regression analyses and I am not sure whether outliers in my data should be deleted. The data I am concerned about appear as "circles" on the SPSS boxplots, however there are no asterisks (which makes me think they are not 'that bad'). The cases I am concerned about do appear under the "casewise diagnostics" table in the output - therefore should I delete these cases?

Answer 1

Flagging outliers is not a judgement call (or in any case need not be one). Given a statistical model, outliers have a precise, objective definition: they are observations that do not follow the pattern of the majority of the data. Such observations need to be set apart at the onset of any analysis simply because their distance from the bulk of the data ensures that they will exert a disproportionate pull on any multivariable model fitted by maximum likelihood (or indeed any other convex loss function).

It is important to point out that multivariable outliers can simply not be reliably detected using residuals from a least square fit (or any other model estimated by ML, or any other convex loss function). Simply put, multivariable outliers can only be reliably detected using their residuals from a model fitted using an estimation procedure not susceptible to be swayed by them.

The belief that outliers will necessary stand out in the residuals of a classical fit ranks somewhere up there with other hard to debunk statistical no-no's such as interpreting p-values as measure of evidence or drawing inference on a population from a biased sample. Except perhaps that this one may well be much older: Gauss himself recommended the use of robust estimator such as the median and the mad (instead of the classical mean and standard deviations) to estimate the parameters of a normal distribution from noisy observations (even going so far as deriving the consistency factor of the mad(1)).

To give a simple visual example based on real data, consider the infamous CYG star data. The red line here depicts the least square fit, the blue line the fit obtained using a robust linear regression fit. The robust fit here is namely the FastLTS (2) fit, an alternative to the LS fit which can be used to detect outliers (because it uses an estimation procedure that ensures that the influence of any observation on the estimated coefficient is bounded). The R code to reproduce it is:

library(robustbase)
data(starsCYG)
plot(starsCYG)
lm.stars <- lm(log.light ~ log.Te, data = starsCYG)
abline(lm.stars$coef,col="red",lwd=2)
lts.stars <- ltsReg(log.light ~ log.Te, data = starsCYG)
abline(lts.stars$coef,col="blue",lwd=2)

Interestingly, the 4 outlying observations on the left do not even have the largest residuals with respect to the LS fit and the QQ plot of the residuals of the LS fit (or any of the diagnostic tools derived from them such as the Cook's distance or the dfbeta) fail to show any of them as problematic. This is actually the norm: no more than two outliers are needed (regardless of the sample size) to pull the LS estimates in such a way that the outliers would not stand out in a residual plot. This is called the masking effect and it is well documented. Perhaps the only thing remarkable about the CYGstars dataset is that it is bivariate (hence we can use visual inspection to confirm the result of the robust fit) and that there actually is a good explanation for why these four observations on the left are so abnormal.

This is, btw, the exception more than the rule: except in small pilot studies involving small samples and few variables and where the person doing the statistical analysis was also involved in the data collection process, I have never experienced a case where prior beliefs about the identity of the outliers were actually true. This is by the way quiet easy to verify. Regardless of whether outliers have been identified using an outlier detection algorithm or the researcher's gut feeling, outliers are by definition observations that have an abnormal leverage (or 'pull') over the coefficients obtained from an LS fit. In other words, outliers are observations whose removal from the sample should severely impact the LS fit.

Although I have never personally experienced this either, there are some well documented cases in the literature where observations flagged as outliers by an outlier detection algorithm were latter found to have been gross errors or generated by a different process. In any case, it is neither scientifically warranted nor wise to only remove outliers if they can somehow be understood or explained. If a small cabal of observations is so far removed from the main body of the data that it can single handedly pull the results of a statistical procedure all by itself it is wise (and i might add natural) to treat it apart regardless of whether or not these data points happens to be also suspect on other grounds.

(1): see Stephen M. Stigler, The History of Statistics: The Measurement of Uncertainty before 1900.

(2): Computing LTS Regression for Large Data Sets (2006) P. J. Rousseeuw, K. van Driessen.

(3): High-Breakdown Robust Multivariate Methods (2008). Hubert M., Rousseeuw P. J. and Van Aelst S. Source: Statist. Sci. Volume 23, 92-119.

Answer 2

In general, I am wary of removing "outliers." Regression analysis can be correctly applied in the presence of non-normally distributed errors, errors that exhibit heteroskedasticity, or values of the predictors/independent variables that are "far" from the rest. The true problem with outliers is that they don't follow the linear model that every other data point follows. How do you know whether this is the case? You don't.

If anything, you don't want to look for values of your variables that are outliers; instead, you want to look for values of your residuals that are outliers. Look at these data points. Are their variables recorded correctly? Is there any reason that they wouldn't follow the same model as the rest of your data?

Of course, the reason why these observations may appear as outliers (according to the residual diagnostic) could be because your model is wrong. I have a professor that liked to say that, if we threw away outliers, we'd still believe that the planets revolve around the sun in perfect circles. Kepler could have thrown away Mars and the circular orbit story would have looked pretty good. Mars provided the key insight that this model was incorrect and he would have missed this result if he ignored that planet.

You mentioned that removing the outliers doesn't change your results very much. Either this is because you only have a very small number of observations that you removed relative to your sample or they are reasonably consistent with your model. This might suggest that, while the variables themselves may look different from the rest, that their residuals are not that outstanding. I would leave them in and not try to justify my decision to remove some points to my critics.

Answer 3

+1 to @Charlie and @PeterFlom; you're getting good information there. Perhaps I can make a small contribution here by challenging the premise of the question. A boxplot will typically (software can vary, and I don't know for sure what SPSS is doing) label points more than 1.5 times the Inter-Quartile Range above (below) the third (first) quartile as 'outliers'. However, we can ask how often should we expect to find at least one such point when we know for a fact that all points come from the same distribution? A simple simulation can help us answer this question:

set.seed(999)                                     # this makes the sim reproducable

outVector = vector(length=10000)                  # to store the results
N = 100                                           # amount of data per sample

for(i in 1:10000){                                # repeating 10k times
  X = rnorm(N)                                    # draw normal sample
  bp = boxplot(X, plot=FALSE)                     # make boxplot
  outVector[i] = ifelse(length(bp$out)!=0, 1, 0)  # if there are 'outliers', 1, else 0
}

mean(outVector)                                   # the % of cases w/ >0 'outliers'
[1] 0.5209

What this demonstrates is that such points can be expected to occur commonly (>50% of the time) with samples of size 100, even when nothing is amiss. As that last sentence hints, the probability of finding a faux 'outlier' via the boxplot strategy will depend on the sample size:

   N    probability
  10    [1] 0.2030
  50    [1] 0.3639
 100    [1] 0.5209
 500    [1] 0.9526
1000    [1] 0.9974

There are other strategies for automatically identifying outliers, but any such method will sometimes misidentify valid points as 'outliers', and sometimes misidentify true outliers as 'valid points'. (You can think of these as type I and type II errors.) My thinking on this issue (for what it's worth) is to focus on the effects of including / excluding the points in question. If your goal is prediction, you can use cross validation to determine whether / how much including the points in question increase the root mean squared error of prediction. If your goal is explanation, you can look at dfBeta (i.e., look at how much the beta estimates of your model change depending on whether the points in question are included or not). Another perspective (arguably the best) is to avoid having to choose whether aberrant points should be thrown out, and just use robust analyses instead.

Answer 4

You should first look at plots of the residuals: Do they follow (roughly) a normal distribution? Do they show signs of heteroskedasticity? Look at other plots as well (I do not use SPSS, so cannot say exactly how to do this in that program, nor what boxplots you are looking at; however, it's hard to imagine that asterisks mean "not that bad" they probably mean that these are highly unusual points by some criterion).

Then, if you have outliers, look at them and try to figure out why.

Then you can try the regression with and without the outliers. If the results are similar, life is good. Report the full results with a footnote. If not similar, then you should explain both regressions.

Answer 5

(late add: this topic has some whiskers on it. But I'm just seeing it for the first time.)

I disagree strongly with some of User603's premises. Some of this reinforces what Charlie said above.

In the abstract, there is no OBJECTIVE definition of an outlier, UNTIL you mentally or formally propose a data-generating model. I view outliers as the gods possibly trying to tell me I'm not as smart as I think I am, unless I can attribute those outliers, with very high certainty, to measurement or data collection errors.

Tossing data that inconveniently does not fit your model is a very dubious practice, because it makes the model look stronger and more certain than it really is.

Even when explanation is the goal, if you suspect outliers may be present, then re-sampling 500 times is a good technique for demonstrating how certain/uncertain you really are with your model.

In a production mode for predictive models, I am opposed to tossing even KNOWN BAD DATA from the training dataset ... UNLESS the cause of the bad data has been fixed, so that similar bad data will not be showing up in the future, in the data to be scored. Real world model performance tracking has to include how well your model does on bad data, as long as similar bad data is still flowing in as part of the data stream.