# Outlier detection using regression

Can regression be used for out lier detection. I understand that there are ways to improve a regression model by removing the outliers. But the primary aim here is not to fit a regression model but find out out liers using regression

-
When fit improves after fitting a model without an outlayer, there is evidence that this is an extreme value. This may be moot if you have a lot of data, because fit will be relatively less improved. – Roman Luštrik Jun 23 '14 at 6:37
@RomanLuštrik: this is a very hesoteric definition of outliers. For example, it is not consistent with the view of outliers used in Glen_b's answer (or for that matter with the definition of outliers used in textbook on the subject such as "Robust Statistics: Theory and Methods"). Care to cite a source to support your definition? – user603 Jun 23 '14 at 10:35
I can't cite any reference. You are of course right, what is an improvement in fit? Improvement can be a highly subjective matter and should be used as a guide, not a robotic cut-off value and judged on case-to-case basis. – Roman Luštrik Jun 23 '14 at 12:19
Iteratively Reweighted Least Squares is a robust regression method commonly used to find outliers in data. – whuber Jun 30 '14 at 1:51

Your best option to use regression to find outliers is to use robust regression.

Ordinary regression can be impacted by outliers in two ways:

First, an extreme outlier in the y-direction at x-values near $\bar x$ can affect the fit in that area in the same way an outlier can affect a mean.

Second, an 'outlying' observation in x-space is an influential observation - it can pull the fit of the line toward it. If it's sufficiently far away the line will go through the influential point:

In the left plot, there's a point that's quite influential, and it pulls the line quite a way from the large bulk of the data. In the right plot, it's been moved even further away -- and now the line goes through the point. When the x-value is that extreme, as you move that point up and down, the line moves with it, going through the mean of the other points and through the one influential point.

An influential point that's perfectly consistent with the rest of the data may not be such a big problem, but one that's far from a line through the rest of the data will make the line fit it, rather than the data.

If you look at the right-hand plot, the red line - the least squares regression line - doesn't show the extreme point as an outlier at all - its residual is 0. Instead, the large residuals from the least squares line are in the main part of the data!

This means you can completely miss an outlier.

Even worse, with multiple regression, an outlier in x-space may not look particularly unusual for any single x-variable. If there's a possibility of such a point, it's potentially a very risky thing to use least squares regression on.

Robust regression

If you fit a robust line - in particular one robust to influential outliers - like the green line in the second plot - then the outlier has a very large residual.

In that case, you have some hope of identifying outliers - they'll be points that aren't - in some sense - close to the line.

Removing outliers

You certainly can use a robust regression to identify and thereby remove outliers.

But once you have a robust regression fit, one that is already not badly affected by outliers, you don't necessarily need to remove the outliers -- you already have a model that's a good fit.

-
"you don't necessarily need to remove the outliers" sometimes, finding the outliers is the purpose of the study (e.g. fraud identification) – user603 Jun 23 '14 at 11:04
@user603 I agree, reasonably often it is - but finding $\neq$ removing – Glen_b Jun 23 '14 at 11:05
(+1) Nice answer, but it is a pity that you don't mention any methods of robust regression. E.g. how was the green line plotted on the right subplot (and why do you prefer that algorithm over the others)? Maybe this link could be useful here: Fast linear regression robust to outliers -- arguably the best thread on CV discussing robust regression. – amoeba Jun 23 '14 at 11:16

Can regression be used for outlier detection.

The primary aim here is not to fit a regression model but find out out liers using regression

Building on Roman Lustrik's comment, here is a heuristic to find outliers using (multiple linear) regression.

Lets say you have sample size $n$. Then, do the following:

1. Fit a regression model on the $n$ examples. Note down its residual sum of squares error $r_{total}$.

2. For each sample i, fit a regression model on the n-1 examples (excluding example i) and note down the corresponding residual sum of squares error $r_i$.

3. Now, compare $r_i$ with $r_tot$ for each $i$, if $r_i << r_{total}$, then $i$ is a candidate outlier.

Setting these candidate outlier points aside, we can repeat the whole exercise again with the reduced sample. In the algorithm, we are picking examples in the data which are influencing the regression fit in a bad way (which is one way to label an example as an outlier).

-
Have you tried this strategy on the dataset shown here? More fundamentally, your strategy amounts to claiming that outliers can be found reliably from the results of a chain of fits minimizing a convex loss function, which is a known fallacy when there is more than a single outlier (this links shows this for the related problem of finding multivariate outliers but the results apply to regression as well). – user603 Jun 23 '14 at 11:56
I am happy to remove my answer. But first, I don't understand both refs you give and moreover, I am not sure why they make my answer incorrect. Where is a 'strategy' is the first ref? Can you point to a specific answer there? Which page and line of the second ref is relevant here and discusses the 'fallacy'? – Theja Jun 23 '14 at 13:22
Sorry, I only could come back to this now. The comment section is a bit short to provide an example and I won't use the 'Answer' section since it's not the OP's question. Still, have you had time to try your methodology on the data I linked to? – user603 Jun 28 '14 at 21:30