Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I have a prediction model tested with four methods as you can see in the boxplot figure below. The attribute that the model predicts is in range of 0-8.
You may notice that there is one upper-bound outlier and three lower-bound outliers indicated by all methods. I wonder if it is appropriate to remove these instances from the data? Or is this a sort of cheating to improve the prediction model?
It is almost always a cheating to remove observations to improve a regression model. You should drop observations only when you truly think that these are in fact outliers.
For instance, you have time series from the heart rate monitor connected to your smart watch. If you take a look at the series, it's easy to see that there would be erroneous observations with readings like 300bps. These should be removed, but not because you want to improve the model (whatever it means). They're errors in reading which have nothing to do with your heart rate.
One thing to be careful though is the correlation of errors with the data. In my example it could be argued that you have errors when the heart rate monitor is displaced during exercises such as running o jumping. Which will make these errors correlated with the hart rate. In this case, care must be taken in removal of these outliers and errors, because they are not at random
I'll give you a made up example of when to not remove outliers. Let's say you're measuring the movement of a weight on a spring. If the weight is small relative to the strength of the weight, then you'll notice that Hooke's law works very well: $$F=-k\Delta x,$$ where $F$ is force, $k$ - tension coefficient and $\Delta x$ is the position of the weight.
Now if you put a very heavy weight or displace the weight too much, you'll start seeing deviations: at large enough displacements $\Delta x$ the motion will seem to deviate from the linear model. So, you might be tempted to remove the outliers to improve the linear model. This would not be a good idea, because the model is not working very well since Hooke's law is only approximately right.
UPDATE In your case I would suggest pulling those data points and looking at them closer. Could it be lab instrument failure? External interference? Sample defect? etc.
Next try to identify whether the presnece of these outliers could be correlated with what you measure like in the example I gave. If there's correlation then there's no simple way to go about it. If there's no correlation then you can remove the outliers
It is always a cheating to remove outliers to improve a regression model. Do you consider spline regression as cheating? FWIW, it does down-weight observations in order to improve the [local] regression model~
$\endgroup$
I originally wanted to post this as a comment to another answer, but it got too long to fit.
When I look at your model, it doesn't necessarily contain one large group and some outliers. In my opinion, it contains 1 medium-sized group (1 to -1) and then 6 smaller groups, each found between 2 whole numbers. You can pretty clearly see that when reaching a whole number, there are fewer observations at those frequencies. The only special point is 0, where there isn't really a discernable drop in observations.
In my opinion, it's worth addressing why this distribution is spread like this:
When measuring discrete human actions, you're always going to have outliers. It can be interesting to see why those outliers don't fit your model, and how they can be used to improve future iterations of your model.
There are pros and cons to removing outliers and build model for "normal pattern" only.
Pros: the model performance is better. The intuition is that, it is very hard to use ONE model to capture both "normal pattern" and "outlier pattern". So we remove outliers and say, we only build a model for "normal pattern".
Cons: we will not be able to predict for outliers. In other words, suppose we put our model in production, there would be some missing predictions from the model
I would suggest to remove outliers and build the model, and if possible try to build a separate model for outlier only.
For the word "cheating", if you are writing paper and explicitly list how do you define and remove outliers, and the mention improved performance is on the clean data only. It is not cheating.
if it is fine to produce no output in production means the same thing? So, if we begin to use our model in a real application to test the outcome variable and use the predicted score in the application, then it would not be okay to remove outliers (especially if they are many as you mentioned)? Is this what you meant?
$\endgroup$
I believe it is only reasonable to remove outliers when one has a solid qualitative reason for doing so. By this I mean that one has information that another variable, that is not in the model, is impacting the outlier observations. Then one has the choice of removing the outlier or adding additional variables.
I find that when I have outlier observations within my dataset, by studying to determine why the outlier exists, I learn more about my data and possible other models to consider.
I'm not even convinced that they are "outliers". You might want to look make a normal probability plot. Are they data or residuals from fitting a model?
