15
$\begingroup$

I have a prediction model tested with four methods as you can see in the boxplot figure below. The attribute that the model predicts is in range of 0-8.

You may notice that there is one upper-bound outlier and three lower-bound outliers indicated by all methods. I wonder if it is appropriate to remove these instances from the data? Or is this a sort of cheating to improve the prediction model?

enter image description here

$\endgroup$
  • 1
    $\begingroup$ (1) I see results for four methods, not three. (2) How could removing evidence of the prediction capabilities possibly improve the methods? $\endgroup$ – whuber Feb 21 '17 at 18:56
  • $\begingroup$ @whuber (1) is fixed. For the (2), so you mean removing an instance that is very inaccurately predicted, would not lead to better prediction performance in overall (this was what I meant with "improve model"? $\endgroup$ – renakre Feb 21 '17 at 19:01
  • 7
    $\begingroup$ removing an observation for whatever reason (say the 4 least well fitting points) is itself a model choice. You should evaluate the forecasting performance of this second model choice too. The salient point is to preserve the integrity of the final test set used to evaluate the performance of the overall prediction method. It is not clear from your question whether you plan to refit the models (Lasso etc) after deletion of the badly predicted data. $\endgroup$ – user603 Feb 21 '17 at 19:15
  • 2
    $\begingroup$ As a side remark, I'd add that sometime great value is hidden in the outliers and it is worthy to take careful look at them. $\endgroup$ – Dror Atariah Feb 24 '17 at 9:36
  • $\begingroup$ @DrorAtariah Thanks Dror, I agree. Extreme cases are valuable. $\endgroup$ – renakre Feb 24 '17 at 9:38
22
$\begingroup$

It is almost always a cheating to remove observations to improve a regression model. You should drop observations only when you truly think that these are in fact outliers.

For instance, you have time series from the heart rate monitor connected to your smart watch. If you take a look at the series, it's easy to see that there would be erroneous observations with readings like 300bps. These should be removed, but not because you want to improve the model (whatever it means). They're errors in reading which have nothing to do with your heart rate.

One thing to be careful though is the correlation of errors with the data. In my example it could be argued that you have errors when the heart rate monitor is displaced during exercises such as running o jumping. Which will make these errors correlated with the hart rate. In this case, care must be taken in removal of these outliers and errors, because they are not at random

I'll give you a made up example of when to not remove outliers. Let's say you're measuring the movement of a weight on a spring. If the weight is small relative to the strength of the weight, then you'll notice that Hooke's law works very well: $$F=-k\Delta x,$$ where $F$ is force, $k$ - tension coefficient and $\Delta x$ is the position of the weight.

Now if you put a very heavy weight or displace the weight too much, you'll start seeing deviations: at large enough displacements $\Delta x$ the motion will seem to deviate from the linear model. So, you might be tempted to remove the outliers to improve the linear model. This would not be a good idea, because the model is not working very well since Hooke's law is only approximately right.

UPDATE In your case I would suggest pulling those data points and looking at them closer. Could it be lab instrument failure? External interference? Sample defect? etc.

Next try to identify whether the presnece of these outliers could be correlated with what you measure like in the example I gave. If there's correlation then there's no simple way to go about it. If there's no correlation then you can remove the outliers

$\endgroup$
  • 2
    $\begingroup$ It is always a cheating to remove outliers to improve a regression model. Do you consider spline regression as cheating? FWIW, it does down-weight observations in order to improve the [local] regression model~ $\endgroup$ – user603 Feb 21 '17 at 19:08
  • 1
    $\begingroup$ I would disagree "It is always a cheating to remove outliers to improve a regression model." there are many tools to do regression diagnostics, and the goal of that is detect and "remove" outliers and refit the model. $\endgroup$ – Haitao Du Feb 21 '17 at 19:10
  • 6
    $\begingroup$ @hxd1011 the tools such as Grubbs are not to automatically remove outliers. They only indicate that there might be an outlier, then you decide if it's an outlier indeed. It's a very dangerous approach to improve fit diagnostics by removing outliers automatically. You have to analyze them case by case. $\endgroup$ – Aksakal Feb 21 '17 at 19:17
  • 2
    $\begingroup$ Ok,I get it. My original language was too rigid. I edited the opening sentence. Thanks for feedback to commenters $\endgroup$ – Aksakal Feb 21 '17 at 23:15
  • 1
    $\begingroup$ @renakre, if you don't think these are outliers, then don't remove the observations. However, what you might need to consider is the measure of goodness of forecast other than square error. For instance, if these instances are not so important to you then maybe you don't need to weight them at square, and instead use absolute deviation etc. The measure should reflect the importance of forecast error, such as dollar losses on each prediction error. Also, the fact that these are counts doesn't automatically mean that there's no instrument errors, the web page plugins that count clicks may fail $\endgroup$ – Aksakal Feb 22 '17 at 13:52
4
$\begingroup$

I originally wanted to post this as a comment to another answer, but it got too long to fit.

When I look at your model, it doesn't necessarily contain one large group and some outliers. In my opinion, it contains 1 medium-sized group (1 to -1) and then 6 smaller groups, each found between 2 whole numbers. You can pretty clearly see that when reaching a whole number, there are fewer observations at those frequencies. The only special point is 0, where there isn't really a discernable drop in observations.

In my opinion, it's worth addressing why this distribution is spread like this:

  • Why does the distribution have these observation count drops at whole numbers?
  • why does this observation count drop not happen at 0?
  • What is so special about these outliers that they're outliers?

When measuring discrete human actions, you're always going to have outliers. It can be interesting to see why those outliers don't fit your model, and how they can be used to improve future iterations of your model.

$\endgroup$
  • $\begingroup$ +1. The whole-number gap seems to not always be right at the whole numbers, so it may be more of us seeing a pattern that doesn't exist, but it could be an artifact of data collection, coding, or discretization that could shed light on the data as a whole. There may even be a gap at 0 that's obscured by the large number of overlapping and perhaps jittered dots. Definitely worth pursuing back to the origin to see if the data's what we think it is. $\endgroup$ – Wayne Feb 28 '17 at 20:18
2
$\begingroup$

There are pros and cons to removing outliers and build model for "normal pattern" only.

  • Pros: the model performance is better. The intuition is that, it is very hard to use ONE model to capture both "normal pattern" and "outlier pattern". So we remove outliers and say, we only build a model for "normal pattern".

  • Cons: we will not be able to predict for outliers. In other words, suppose we put our model in production, there would be some missing predictions from the model

I would suggest to remove outliers and build the model, and if possible try to build a separate model for outlier only.

For the word "cheating", if you are writing paper and explicitly list how do you define and remove outliers, and the mention improved performance is on the clean data only. It is not cheating.

$\endgroup$
  • 3
    $\begingroup$ I do not mind being downvoted, but could someone tell me the reason? $\endgroup$ – Haitao Du Feb 21 '17 at 18:43
  • $\begingroup$ I upvoted :) Do you also think it is a good idea to remove the outliers and then resample the data for further testing the prediction model? $\endgroup$ – renakre Feb 21 '17 at 18:45
  • 1
    $\begingroup$ @renakre i would suggest you to think about what to do in production. Let's say, if you found outlier is only 1%, and it is fine to produce no output in production. Then just remove them. If you found outlier is 30%, and it is not OK to skip predictions in production. Then try to have a separate model for it. $\endgroup$ – Haitao Du Feb 21 '17 at 18:48
  • $\begingroup$ We are mostly testing things to see if we can predict some outcome variable. Does if it is fine to produce no output in production means the same thing? So, if we begin to use our model in a real application to test the outcome variable and use the predicted score in the application, then it would not be okay to remove outliers (especially if they are many as you mentioned)? Is this what you meant? $\endgroup$ – renakre Feb 21 '17 at 18:59
  • 1
    $\begingroup$ @renakre You are dead on ! That is what we have done recently with AITOBOX where the forecast limits are not only based on the psi weights but the re-sampled errors populated with outliers. This is done not only for ARIMA models but causal models where the uncertainty in the predictors is also incorporated in a similar fashion. $\endgroup$ – IrishStat Feb 21 '17 at 19:07
2
$\begingroup$

I believe it is only reasonable to remove outliers when one has a solid qualitative reason for doing so. By this I mean that one has information that another variable, that is not in the model, is impacting the outlier observations. Then one has the choice of removing the outlier or adding additional variables.

I find that when I have outlier observations within my dataset, by studying to determine why the outlier exists, I learn more about my data and possible other models to consider.

$\endgroup$
  • 1
    $\begingroup$ Welcome to stats.SE! Please take a moment to view our tour. It would be helpful if you expanded your answer to more fully answer the question (such as outlier determination based upon boxplot, the impacts this method may have on the prediction model, &c.). $\endgroup$ – Tavrock Feb 28 '17 at 22:20
2
$\begingroup$

I'm not even convinced that they are "outliers". You might want to look make a normal probability plot. Are they data or residuals from fitting a model?

$\endgroup$
  • $\begingroup$ they are the difference between the predicted and real values. $\endgroup$ – renakre Mar 5 '17 at 16:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.