# Outlier in linear regression

My dataset consists of transactions at the daily level; they can be negative or positive, for a total of one year of data.

I want to fit a model that predicts daily transactions, spendings or income, lets say for one extra month.

My dependent variable will be transaction amount; my features will be a set of variables, in this example, just day and type of transaction.

I want to fit a naive linear regression.

Now let's imagine my data looked like:

a. I can see a clear (positive) outlier in the data. My regression will be extremely sensitive to that data point. If the amount variable would have been just positive, I would have taken the log. What to do here instead?

b. Is there any approach other than regression that could relax for the presence of the outliers? Given the obvious seasonality of the data, a time series approach would be more appropriate? Any hint?

Thanks

• Its fairly normal to just exclude outliers altogether. Is this not appropriate in your context? Apr 17, 2017 at 3:22

Yes, there are several methods you can use in which you can keep the outlier in your model (because it can still tell you something useful), but limit its effect on the linear regression coefficients.

These methods are referred to as Robust Regression. Basically, these methods place weights on the observations based on the residuals to reduce the effect of outliers on your regression function.

1. Least Absolute Residuals (LAR)

Also called minimum $L_{1}$ norm regression. Here you want to minimize $L_{1}$ to find your coefficients:

$L_1 = \sum | Y_{i} - (\beta_{0} + \beta_{1}X_{i1} + ... + \beta_{p-1}X_{i,p-1})|$

In R: LAD() in Quantreg package.

2. Least Median of Squares (LMS)

Here you want to minimize the median squared deviation:

median{$[Y_{i} - (\beta_{0} + \beta_{1}X_{i1} + ... + \beta_{p-1}X_{i,p-1})]^{2}$}

In R: lmsreg(Y~X) in MASS package.

3. Iteratively Reweighted Least Squares (IRLS)

Here you want to calculate weights for each observation. Two popular methods (sometimes combined using Huber for the first iteration and Bisquare for all other iterations is common) are the Huber weight function ($w_{h}$) and the Bisquare weight function ($w_{b}$). You can continue the process of calculating your regression model using the weights with several iterations until your coefficients converge.

$w_{h} = 1$ if $|u| \leq 1.345$ or $w_{h} = 1.345 / |u|$ if $|u| > 1.345$

and

$w_{b} = [1 - (u \ 4.685)^{2}]^{2}$ if $|u| \leq 4.685$ or $w_{b} = 0$ if $|u| > 4.685$

where, $u_{i} = e_{i} / MAD$ and $MAD = 1/.6745 \space median\{|e_{i} - median\{e_{i}\}|\}$

In R: rml(Y~X, method = ) where method = mm is the Bisquare method and the default is Huber.

• When you get the results from LAR or so, how do you get the general regression statistics such as t-test, f-test, R square, etc. if the function 'summary' is not working for these? For instance, I use function 'lad' from 'L1pack' package. But I cannot find these general regression statistics using 'summary' command. Is there a way to do so?
– Eric
May 5, 2017 at 17:36
• Is it possible to simply use the general regression statistics from 'lm' command using the same data and only change the coefficient values from the ones derived from 'lad' command?
– Eric
May 5, 2017 at 17:42
• I have never used it by attempt 'LAD()' from the 'Quantreg' package. Let me know how this works for you May 6, 2017 at 17:51
• My issue is, there are only coefficient and intercept values but no t-test, f-test, R square, etc. which I also need as well.
– Eric
May 6, 2017 at 17:53
• Try 'summary.lad()' in the 'ldr' package. I believe this was designed for your problem. May 6, 2017 at 18:01