Whether to delete cases that are flagged as outliers by statistical software when performing multiple regression?

I am performing multiple regression analyses and I am not sure whether outliers in my data should be deleted. The data I am concerned about appear as "circles" on the SPSS boxplots, however there are no asterisks (which makes me think they are not 'that bad'). The cases I am concerned about do appear under the "casewise diagnostics" table in the output - therefore should I delete these cases?

• Thank you very much Charlie and Epigrad. Could you please suggest which graph in SPSS I look at to assess whether there are outliers in the residuals? The scatterplot looks quite messy! I don't there is any problem with the data as such (as in they have not been entered incorrectly) I just think that some of my participants had much higher scores on some of my scales, i.e. because they were much more socially anxious that the rest of the sample. – Anon Sep 12 '11 at 2:39
• You should plot the predicted value of y (the one given according the model that you estimate) on the x axis and the residuals on the y axis. Instead of the predicted value of y, you could put one of your predictors/independent variables on the x axis. You could create several plots, each with a different predictor on the x axis to see which x value is leading to the outlier behavior. Again, I would caution against outlier removal; instead, analyze why the outlier is occurring. – Charlie Sep 12 '11 at 2:47
• Echoing Charlie's statement, its the "why" that matters, rather than the "if", and I too would caution against their removal. I'm not familiar with SPSS, but whatever features you used to run the regression should be able to also give you a plot of residuals, or at least the value of them which you can use to make the plot Charlie suggests. – Fomite Sep 12 '11 at 4:14
• @Anon I've merged your two accounts. Please register so that you can update and/or comment your question. – chl Sep 12 '11 at 6:32
• @user603 No, you don't read me correctly. "Outlier" doesn't mean anything - especially when flagged by an automatic procedure in statistical software. There are just as many examples of the important findings of a study being in the "outliers". Whenever you have data you're deleting, it should be for a reason. "They're inconvenient" isn't a reason. – Fomite Feb 25 '13 at 1:21

5 Answers

Flagging outliers is not a judgement call (or in any case need not be one). Given a statistical model, outliers have a precise, objective definition: they are observations that do not follow the pattern of the majority of the data. Such observations need to be set apart at the onset of any analysis simply because their distance from the bulk of the data ensures that they will exert a disproportionate pull on any multivariable model fitted by maximum likelihood (or indeed any other convex loss function).

It is important to point out that multivariable outliers can simply not be reliably detected using residuals from a least square fit (or any other model estimated by ML, or any other convex loss function). Simply put, multivariable outliers can only be reliably detected using their residuals from a model fitted using an estimation procedure not susceptible to be swayed by them.

The belief that outliers will necessary stand out in the residuals of a classical fit ranks somewhere up there with other hard to debunk statistical no-no's such as interpreting p-values as measure of evidence or drawing inference on a population from a biased sample. Except perhaps that this one may well be much older: Gauss himself recommended the use of robust estimator such as the median and the mad (instead of the classical mean and standard deviations) to estimate the parameters of a normal distribution from noisy observations (even going so far as deriving the consistency factor of the mad(1)).

To give a simple visual example based on real data, consider the infamous CYG star data. The red line here depicts the least square fit, the blue line the fit obtained using a robust linear regression fit. The robust fit here is namely the FastLTS (2) fit, an alternative to the LS fit which can be used to detect outliers (because it uses an estimation procedure that ensures that the influence of any observation on the estimated coefficient is bounded). The R code to reproduce it is:

library(robustbase)
data(starsCYG)
plot(starsCYG)
lm.stars <- lm(log.light ~ log.Te, data = starsCYG)
abline(lm.stars$coef,col="red",lwd=2) lts.stars <- ltsReg(log.light ~ log.Te, data = starsCYG) abline(lts.stars$coef,col="blue",lwd=2)


Interestingly, the 4 outlying observations on the left do not even have the largest residuals with respect to the LS fit and the QQ plot of the residuals of the LS fit (or any of the diagnostic tools derived from them such as the Cook's distance or the dfbeta) fail to show any of them as problematic. This is actually the norm: no more than two outliers are needed (regardless of the sample size) to pull the LS estimates in such a way that the outliers would not stand out in a residual plot. This is called the masking effect and it is well documented. Perhaps the only thing remarkable about the CYGstars dataset is that it is bivariate (hence we can use visual inspection to confirm the result of the robust fit) and that there actually is a good explanation for why these four observations on the left are so abnormal.

This is, btw, the exception more than the rule: except in small pilot studies involving small samples and few variables and where the person doing the statistical analysis was also involved in the data collection process, I have never experienced a case where prior beliefs about the identity of the outliers were actually true. This is by the way quiet easy to verify. Regardless of whether outliers have been identified using an outlier detection algorithm or the researcher's gut feeling, outliers are by definition observations that have an abnormal leverage (or 'pull') over the coefficients obtained from an LS fit. In other words, outliers are observations whose removal from the sample should severely impact the LS fit.

Although I have never personally experienced this either, there are some well documented cases in the literature where observations flagged as outliers by an outlier detection algorithm were latter found to have been gross errors or generated by a different process. In any case, it is neither scientifically warranted nor wise to only remove outliers if they can somehow be understood or explained. If a small cabal of observations is so far removed from the main body of the data that it can single handedly pull the results of a statistical procedure all by itself it is wise (and i might add natural) to treat it apart regardless of whether or not these data points happens to be also suspect on other grounds.

(1): see Stephen M. Stigler, The History of Statistics: The Measurement of Uncertainty before 1900.

(2): Computing LTS Regression for Large Data Sets (2006) P. J. Rousseeuw, K. van Driessen.

(3): High-Breakdown Robust Multivariate Methods (2008). Hubert M., Rousseeuw P. J. and Van Aelst S. Source: Statist. Sci. Volume 23, 92-119.

• This is good stuff (+1). However, I think you misuse conventional terminology and have co-opted "outlier" to refer to "influential observation." The concepts are both valuable, and you treat the latter well here, but they are not as interchangeable as you seem to indicate. For instance, an influential observation that is consistent with the majority of the data would fit your characterization of "observations that have an abnormal leverage (or 'pull') over the coefficients obtained from an LS fit" but would not be considered by most writers to be an "outlier" per se. – whuber Feb 25 '13 at 19:41
• @whuber: Good point. Indeed I consider, as do recent textbooks on robust statistics (for example, Robust Statistics: Theory and Methods. Wiley) such observations (so called 'good leverage points') as harmful. The justification is that they deflate the standard error of the estimated coefficients causing the user to place unwarranted confidence in the strength of the observed relation. Considering good leverage points as outliers also makes the formal approach more consistent: after all good leverage point do have an outsized influence on the s.e. which are a component of the LS/ML fit. – user603 Feb 25 '13 at 19:52
• +1 Very nice example. Real data that shows two fits that are nearly orthogonal, and in which the highly-influential four in the upper-left won't have the largest residuals after an OLS fit. – Wayne Feb 25 '13 at 20:02

In general, I am wary of removing "outliers." Regression analysis can be correctly applied in the presence of non-normally distributed errors, errors that exhibit heteroskedasticity, or values of the predictors/independent variables that are "far" from the rest. The true problem with outliers is that they don't follow the linear model that every other data point follows. How do you know whether this is the case? You don't.

If anything, you don't want to look for values of your variables that are outliers; instead, you want to look for values of your residuals that are outliers. Look at these data points. Are their variables recorded correctly? Is there any reason that they wouldn't follow the same model as the rest of your data?

Of course, the reason why these observations may appear as outliers (according to the residual diagnostic) could be because your model is wrong. I have a professor that liked to say that, if we threw away outliers, we'd still believe that the planets revolve around the sun in perfect circles. Kepler could have thrown away Mars and the circular orbit story would have looked pretty good. Mars provided the key insight that this model was incorrect and he would have missed this result if he ignored that planet.

You mentioned that removing the outliers doesn't change your results very much. Either this is because you only have a very small number of observations that you removed relative to your sample or they are reasonably consistent with your model. This might suggest that, while the variables themselves may look different from the rest, that their residuals are not that outstanding. I would leave them in and not try to justify my decision to remove some points to my critics.

• +1 Don't throw away data because its an outlier. Find out why some data are outlying. – Fomite Sep 12 '11 at 2:23
• this is terrible advice. It is very common for outliers to be so far away from the rest of the data as to pull the regression line towards them in such a way that they won't stand out on a residual plot (or worst: yield large residuals for the genuine data points). In fact, it can be shown that as soon as you have more than a single outlier, it cannot be reliably detected using a residual plot from a classical regression. This is called the masking effect and i well documented notably in many real data examples. – user603 Feb 24 '13 at 23:42
• By the way, this is also why i'd avoid using the Mars example: it illustrates a procedure that only works if you are dealing with a single outlier. In most application there are no such guarantee. It gives a wrong sense of confidence in a generally flawed methodology (which as statistician is really what we should thrive to prevent). – user603 Feb 25 '13 at 0:09

+1 to @Charlie and @PeterFlom; you're getting good information there. Perhaps I can make a small contribution here by challenging the premise of the question. A boxplot will typically (software can vary, and I don't know for sure what SPSS is doing) label points more than 1.5 times the Inter-Quartile Range above (below) the third (first) quartile as 'outliers'. However, we can ask how often should we expect to find at least one such point when we know for a fact that all points come from the same distribution? A simple simulation can help us answer this question:

set.seed(999)                                     # this makes the sim reproducable

outVector = vector(length=10000)                  # to store the results
N = 100                                           # amount of data per sample

for(i in 1:10000){                                # repeating 10k times
X = rnorm(N)                                    # draw normal sample
bp = boxplot(X, plot=FALSE)                     # make boxplot
outVector[i] = ifelse(length(bp\$out)!=0, 1, 0)  # if there are 'outliers', 1, else 0
}

mean(outVector)                                   # the % of cases w/ >0 'outliers'
[1] 0.5209


What this demonstrates is that such points can be expected to occur commonly (>50% of the time) with samples of size 100, even when nothing is amiss. As that last sentence hints, the probability of finding a faux 'outlier' via the boxplot strategy will depend on the sample size:

   N    probability
10    [1] 0.2030
50    [1] 0.3639
100    [1] 0.5209
500    [1] 0.9526
1000    [1] 0.9974


There are other strategies for automatically identifying outliers, but any such method will sometimes misidentify valid points as 'outliers', and sometimes misidentify true outliers as 'valid points'. (You can think of these as type I and type II errors.) My thinking on this issue (for what it's worth) is to focus on the effects of including / excluding the points in question. If your goal is prediction, you can use cross validation to determine whether / how much including the points in question increase the root mean squared error of prediction. If your goal is explanation, you can look at dfBeta (i.e., look at how much the beta estimates of your model change depending on whether the points in question are included or not). Another perspective (arguably the best) is to avoid having to choose whether aberrant points should be thrown out, and just use robust analyses instead.

• The procedures you recommend only work reliably if there are at most a single outlier (regardless of the size of your dataset) which is an unrealistic assumption. Tukey calibrated the whisker rule to exclude roughly 1% of the observations on each end if the data is drawn from a Gaussian distribution. Your simulations confirm that. Tukey's opinion was that the losses caused by disregarding such a small portion of the data in those cases where the observations are well behaved is for all practical concerns inconsequential. Specially in regards to the benefits in the cases when the data is not. – user603 Feb 24 '13 at 23:59
• Thanks for your comment, @user603; that's a thought-provoking position. Which procedures that I recommend are you objecting to: using, eg, dfbeta to detect possible outliers, or using robust analyses (prototypically Tukey's bisquare as an alternative loss function) as protection against their influence instead of choosing which data points to throw out? – gung - Reinstate Monica Feb 25 '13 at 0:20
• thanks for pointing out the lack of clarity in my comment (i was constrained by the length limit). Of course, i specifically mean the first ones: dfbeta and cross validation (the latter is problematic only if the observations used to perform the cross-validation are randomly drawn from the original sample. An example of case where cross-validation could be used would be in so-called quality control setting where the observations used for testing are drawn from a temporally disjoint sample). – user603 Feb 25 '13 at 0:24
• Thanks for clarifying, @user603. I'll have to play w/ these ideas to understand them more thoroughly. My intuition is that it would be pretty difficult not to notice outliers that are distorting your results; it seems like you would need to have outliers distorting your results on both sides equally, in which case your betas would end up approximately unbiased & your results would simply be less 'significant'. – gung - Reinstate Monica Feb 25 '13 at 0:33
• My intuition is that it would be pretty difficult not to notice outliers that are distorting your results but unfortunately, the fact is that it ain't so. Also look at the example i provide in my answer. – user603 Feb 25 '13 at 10:47

You should first look at plots of the residuals: Do they follow (roughly) a normal distribution? Do they show signs of heteroskedasticity? Look at other plots as well (I do not use SPSS, so cannot say exactly how to do this in that program, nor what boxplots you are looking at; however, it's hard to imagine that asterisks mean "not that bad" they probably mean that these are highly unusual points by some criterion).

Then, if you have outliers, look at them and try to figure out why.

Then you can try the regression with and without the outliers. If the results are similar, life is good. Report the full results with a footnote. If not similar, then you should explain both regressions.

• Thank you very much Peter. I have inspected the Q-Q plots and the data do not appear to be outstandinly non-normal. When I delete the outliers, they don't seem to make much of a difference to the results. So, therefore, should I just leave them in? I would still be interested to hear others' thoughts on the casewise diagnostics table in SPSS. Many thanks. – Anon Sep 11 '11 at 11:22
• Yes, I would then leave them in with a footnote something like "analysis with several outliers deleted showed very similar results" – Peter Flom Sep 12 '11 at 9:37
• Even assuming one could reliable find outliers using such a procedure (and most of the time, one can't) that still leaves strangely unaddressed the problem of what to do when you can't "figure out"/explain the outliers. I second the advice to stay clear of SPSS. – – user603 Feb 25 '13 at 18:04

(late add: this topic has some whiskers on it. But I'm just seeing it for the first time.)

I disagree strongly with some of User603's premises. Some of this reinforces what Charlie said above.

In the abstract, there is no OBJECTIVE definition of an outlier, UNTIL you mentally or formally propose a data-generating model. I view outliers as the gods possibly trying to tell me I'm not as smart as I think I am, unless I can attribute those outliers, with very high certainty, to measurement or data collection errors.

Tossing data that inconveniently does not fit your model is a very dubious practice, because it makes the model look stronger and more certain than it really is.

Even when explanation is the goal, if you suspect outliers may be present, then re-sampling 500 times is a good technique for demonstrating how certain/uncertain you really are with your model.

In a production mode for predictive models, I am opposed to tossing even KNOWN BAD DATA from the training dataset ... UNLESS the cause of the bad data has been fixed, so that similar bad data will not be showing up in the future, in the data to be scored. Real world model performance tracking has to include how well your model does on bad data, as long as similar bad data is still flowing in as part of the data stream.