1
$\begingroup$

I'm trying to predict the prices of Airbnb's given some of their attributes using a very simple linear regression model.

Despite obtaining a median absolute error of around 24 euros on the training and testing set which I'm happy with, when I plot the residual error there's a lot of large outliers.

My question is; how should one deal with them? Should they be identified and dealt with in the pre-processing stage. Or simply omitted in the prediction phase?

from sklearn import linear_model
from sklearn import metrics

lin = linear_model.LinearRegression()

lin.fit(X_train, y_train)
y_train_predict = lin.predict(X_train)
y_test_predict = lin.predict(X_test)

plt.figure(figsize=(20, 20))
plt.scatter(y_train_predict, y_train_predict - y_train, c='b', s=40, alpha=0.5)
plt.scatter(y_test_predict, y_test_predict - y_test, c = 'g', s=40)

Residual Plot

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

I see your intention is towards predictive power, not inference. So consider that your pre-processing pipeline should be run separately in each fold during training, otherwise you incur optimistic bias on your performance estimates: what looks like a outlier in the whole dataset might not look like within a traning fold. So if you intend on removing those points, do so through automatized criteria.

If the data acquisition was actually faulty (and you have strong reasons to believe so), you are justified removing what seems to be outliers. See Should I report the descriptive statistics in publication before or after outliers removal?

You could try your hand on robust statistics though, more specifically robust estimator for linear regression. These are resistant to leverage points, and are completely automatic from the training point of view.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I'm not sure I understand what you mean by this, could you elaborate? pre-processing pipeline should be run separately in each fold during training $\endgroup$
    – Charles
    Commented Oct 30, 2017 at 12:25
  • $\begingroup$ @Charles.fried How would you pick what looks like an outlier? If you compare it to the whole train dataset, then your pick is biased: you're infusing information your model wouldn't have access to, i.e. information leaking. Now, if you do that to each fold separately, then there's no bias. $\endgroup$
    – Firebug
    Commented Oct 30, 2017 at 13:44
  • $\begingroup$ @Charles.fried Treat it like any other pre-processing step, e.g. whitening with PCA or simply standardizing variables. $\endgroup$
    – Firebug
    Commented Oct 30, 2017 at 13:44
  • $\begingroup$ Appreciate your help, I've manually gone through the top 20 mistakes and I can classify them in three categories: 1. The price have been set excessively high by the host, and does not reflect the specification of the apartment. (outlier) 2. The high number of people the listing accommodates is not fairly reflected by our model. 3. The standard of finish is high and cannot be identified using standard features. How would I go about increasing the importance of my accomodates feature which is clearly being underestimated by my model? $\endgroup$
    – Charles
    Commented Oct 30, 2017 at 14:41
0
$\begingroup$

Always deal with outliers in the preprocessing stage. Check whether it it's an error or a genuine outlier. General approach is to emphasize on why an example is an outlier, then change the value with the mean or median and model over it.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Considering I have over 20 features how should you go about identifying which one is causing the issue? $\endgroup$
    – Charles.fried
    Commented Oct 30, 2017 at 10:55
  • $\begingroup$ It isn't always possible to deal with outliers only in the preprocessing stage. What is an outlier might depend on which model you are fitting, a multivariate outlier might not be outlier in the marginal distributions. $\endgroup$
    – kjetil b halvorsen ♦
    Commented Nov 8, 2017 at 17:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.