# Dealing with outliers with Linear Regression

I'm trying to predict the prices of Airbnb's given some of their attributes using a very simple linear regression model.

Despite obtaining a median absolute error of around 24 euros on the training and testing set which I'm happy with, when I plot the residual error there's a lot of large outliers.

My question is; how should one deal with them? Should they be identified and dealt with in the pre-processing stage. Or simply omitted in the prediction phase?

from sklearn import linear_model
from sklearn import metrics

lin = linear_model.LinearRegression()

lin.fit(X_train, y_train)
y_train_predict = lin.predict(X_train)
y_test_predict = lin.predict(X_test)

plt.figure(figsize=(20, 20))
plt.scatter(y_train_predict, y_train_predict - y_train, c='b', s=40, alpha=0.5)
plt.scatter(y_test_predict, y_test_predict - y_test, c = 'g', s=40)


I see your intention is towards predictive power, not inference. So consider that your pre-processing pipeline should be run separately in each fold during training, otherwise you incur optimistic bias on your performance estimates: what looks like a outlier in the whole dataset might not look like within a traning fold. So if you intend on removing those points, do so through automatized criteria.

If the data acquisition was actually faulty (and you have strong reasons to believe so), you are justified removing what seems to be outliers. See Should I report the descriptive statistics in publication before or after outliers removal?

You could try your hand on robust statistics though, more specifically robust estimator for linear regression. These are resistant to leverage points, and are completely automatic from the training point of view.

• I'm not sure I understand what you mean by this, could you elaborate? pre-processing pipeline should be run separately in each fold during training – Charles.fried Oct 30 '17 at 12:25
• @Charles.fried How would you pick what looks like an outlier? If you compare it to the whole train dataset, then your pick is biased: you're infusing information your model wouldn't have access to, i.e. information leaking. Now, if you do that to each fold separately, then there's no bias. – Firebug Oct 30 '17 at 13:44
• @Charles.fried Treat it like any other pre-processing step, e.g. whitening with PCA or simply standardizing variables. – Firebug Oct 30 '17 at 13:44
• Appreciate your help, I've manually gone through the top 20 mistakes and I can classify them in three categories: 1. The price have been set excessively high by the host, and does not reflect the specification of the apartment. (outlier) 2. The high number of people the listing accommodates is not fairly reflected by our model. 3. The standard of finish is high and cannot be identified using standard features. How would I go about increasing the importance of my accomodates feature which is clearly being underestimated by my model? – Charles.fried Oct 30 '17 at 14:41

Always deal with outliers in the preprocessing stage. Check whether it it's an error or a genuine outlier. General approach is to emphasize on why an example is an outlier, then change the value with the mean or median and model over it.

• Considering I have over 20 features how should you go about identifying which one is causing the issue? – Charles.fried Oct 30 '17 at 10:55
• It isn't always possible to deal with outliers only in the preprocessing stage. What is an outlier might depend on which model you are fitting, a multivariate outlier might not be outlier in the marginal distributions. – kjetil b halvorsen Nov 8 '17 at 17:05