14
$\begingroup$

I am wondering if there is a simple way of detecting outliers.

For one of my projects, which was basically a correlation between the number of times respondents participate in physical activity in a week and the number of times they eat outside the home (fast food) in a week, I drew a scatterplot and literally removed the data points that were extreme. (The scatterplot showed a negative correlation.)

This was based on value judgement (based on the scatterplot where these data points were clearly extreme). I did not do any statistical tests.

I am just wondering if this is a sound way of dealing with outliers.

I have data from 350 people so loss of (say) 20 data points is not a worry to me.

$\endgroup$
  • $\begingroup$ See also the answers to the similar question rigorous definition of an outlier $\endgroup$ – Jonas Sep 24 '12 at 3:51
  • 3
    $\begingroup$ Also very closely related is stats.stackexchange.com/questions/175. Many potential outlier detection methods are described in answers at stats.stackexchange.com/questions/213. But more to the point would be some context: what are you doing with this scatterplot? What conclusions are you trying to draw from it? Some conclusions will depend little on what you do with outliers, whereas others might depend on them critically. This indicates that the methods you use to identify and treat outliers must depend on the intended analysis. $\endgroup$ – whuber Sep 24 '12 at 14:38
  • $\begingroup$ In economics datasets the standard practice is just to say "We windsorize the dataset at 2.5% and 97.5%", or alternatively 1% and 99%. Then they just delete the observations that are out of that quantile range. $\endgroup$ – user14281 Sep 27 '12 at 8:43
  • $\begingroup$ @Harokitty Winsorising seems to mean clipping the values rather than discarding them. $\endgroup$ – Peter Wood Oct 18 '12 at 14:40
  • $\begingroup$ I would recommend that you also report the scatter-plot unedited apart from any errors in data recording. It is possible that there may be one or more additional distinct populations. For an exampe, see the Wikipedia entry for Hertzsprung–Russell diagram at en.wikipedia.org/wiki/Hertzsprung%E2%80%93Russell_diagram $\endgroup$ – Robert Jones Dec 30 '16 at 22:39
21
$\begingroup$

There is no simple sound way to remove outliers. Outliers can be of two kinds:

1) Data entry errors. These are often the easiest to spot and always the easiest to deal with. If you can find the right data, correct it; if not, delete it.

2) Legitimate data that is unusual. This is much trickier. For bivariate data like yours, the outlier could be univariate or bivariate.

a) Univariate. First, "unusual" depends on the distribution and the sample size. You give us the sample size of 350, but what is the distribution? It clearly isn't normal, since it's a relatively small integer. What is unusual under a Poisson would not be under a negative binomial. I'd kind of suspect a zero-inflated negative binomial relationship.

But even when you have the distribution, the (possible) outliers will affect the parameters. You can look at "leave one out" distributions, where you check if data point q would be an outlier if the data had all points but q. Even then, though, what if there are multiple outliers?

b) Bivariate. This is where neither variable's value is unusual in itself, but together they are odd. There is a possibly apocryphal report that the census once said there were 20,000 12 year old widows in the USA. 12 year olds aren't unusual, widows aren't either, but 12 year old widows are.

Given all this, it might be simpler to report a robust measure of relationship.

$\endgroup$
  • $\begingroup$ Thanks. I am thinking perhaps a confidence ellipse would be a good indicator of outliers as it would show the percent of data that would lie within a certain confidence level (given bivariate normal distribution). $\endgroup$ – Amarald Sep 24 '12 at 4:19
  • $\begingroup$ Your data cannot be bivariate normal, since it is composed of non-negative integers $\endgroup$ – Peter Flom - Reinstate Monica Sep 24 '12 at 10:01
18
$\begingroup$

I have done a lot of research on outliers, particularly when I worked on energy data validation at Oak Ridge from 1978 to 1980. There are formal tests for univariate outliers for normal data (e.g. Grubbs' test and Dixon's ratio test). There are tests for multivariate outliers and time series. The book by Barnett and Lewis on "Outliers in Statistical Data" is the bible on outliers and covers just about everything.

When I was at Oak Ridge working on data validation we had large multivariate data sets. For univariate outliers there is a direction for extremes (highly above the mean and highly below the mean). But for multivariate outliers there are many directions to look for outliers. Our philosophy was to consider what the intended use of the data is. If you are trying to estimate certain parameters such as a bivariate correlation or a regression coefficient then you want to look in the direction that provides the greatest effect on the parameter of interest. At that time I had read Mallows' unpublished paper on influence functions. The use of influence functions to detect outliers is covered in Gnanadesikan's multivariate analysis book. Of course you can find it in Barnett and Lewis also.

The influence function for a parameter is defined at points in the multivariate space of the observations and essentially measures the difference between the parameter estimate when the data point is included compared with when it is left out. You can do such estimates with each sample point but usually you can derive a nice functional form for the influence function that gives insight and faster computation.

For example in my paper in the American Journal of Mathematical and Management Science in 1982 "The Influence Function and Its Application to Data Validation" I show the analytic formula for the influence function for bivariate correlation and that the contours of constant influence are hyperbolae. So the contours show the direction in the plane where the influence function increases the fastest.

In my paper I show how we applied the influence function for bivariate correlation with the FPC Form 4 data on generation and consumption of energy. There is a clear high positive correlation between the two and we found a few outliers that were highly influential on the estimate of correlation. Further investigation showed that at least one of the points was in error and we were able to correct it.

But an important point that I always mention when discussing outliers is that automatic rejection is wrong. The outlier is not always an error and sometimes it provides important information about the data. Valid data should not be removed just because it doesn't conform with our theory of reality. Whether or not it is difficult to do, the reason why the outlier occurred should always be investigated.

I should mention that this is not the first time multivariate outliers have been discussed in this site. A search for outliers would probably lead to several questions where multivariate outliers have been discussed . I know that I have referenced my paper and these books before and given links to them.

Also when outlier rejection is discussed many of us on this site have recommended against it especially if it is done based solely on a statistical test. Peter Huber often mentions robust estimation as an alternative to outlier rejection. The idea is that robust procedures will downweight the outliers reducing their effect on estimation without the heavy-handed step of rejecting them and using a non-robust estimator.

The influence function was actually originally developed by Frank Hampel in his PhD dissertation in the early 1970s (1974 I think). His idea was actually to use influence functions to identify estimators that were not robust against outliers and to help develop robust estimators.

Here is a link to a previous discussion on this topic where I mentioned some work of mine on detecting outliers in time series using influence functions.

$\endgroup$
2
$\begingroup$

Another simple approach to dealing with outliers is to use non-parametric statistics. Probably with your sample size a Spearman's rho would work well as an index of the correlation. (Note, though, that non-parametric, rank-order statistics do not help you much with non-linear relationships.)

If you want to use a Pearson's r (a parametric statistic), and if you are not able to compute Cook's distance, you might use a standard rule of thumb that any data point that is more than 2.67 standard deviations (s.d.) from the mean, or 4.67 s.d. from the mean is an outlier or extreme, respectively. These are typical cutoff values for outliers and extreme data points that are used in one standard statistical analysis program (SPSS).

Just because a data point is an outlier does not mean it is bad data to be discarded. You might compute your correlation with and without extreme points and go from there.

$\endgroup$
1
$\begingroup$

You might want to try Cook's Distance. See the wikipedia article for suggested cutoffs. Also, if you are heading toward some regression model, then you may wish to try robust regression.

$\endgroup$

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

  • 1
    $\begingroup$ This seems more like a comment than an answer; answers are usually lengthier and more detailed. E.g. if you included reasoning as to why Cook's distance is a good test for outliers and so one, this would be an answer. $\endgroup$ – Peter Flom - Reinstate Monica Sep 29 '12 at 19:36
1
$\begingroup$

Firstly, don't remove atypical values unless you are sure that are out of the study! They may contain some important information (variability). You should drop them if it's obvious that the outlier is due to incorrectly entered or measured data. If you don't know the sampling method used to obtain your data, then you should identify atypical values and their effects as follows:

  1. Degree of abnormality: it is expected a 5% of the observations with standarized residuals ($e_i*$) > 2. If you have some more higher residuals you can suspect outliers.

  2. Degree of distance to the center gravity in the space of x's: $h_{ii}$ (leverage). When some $h_{ii}$ is very high you have an observation that may distort your model because is out of the range of your study.

  3. Degree of influence on the fitted model: Influential points are those which have enough weight to change your model. Then, cofficients of fitted model using all n observations are very different from coefficients of the fitted model using all points but not this observation $i$-th.
    Cook's distance or Cook's D is a commonly used estimate of the influence of a data point. : $DC_i=ei*^2·h_{ii}/[(1-h_{ii})·p]$

Possible solutions:

  • Transforming variables and / or adding new variables to the model.
  • For influential observations which are nothing but outliers, if not many, you can remove those individuals.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.