I'm doing a cost prediction model for mechanic parts (12000 rows and 43 variables). All the numeric variable(10 numeric variables total) shows no normal distribution. After the log transformation, some numeric variables has the normal distribution, and the data has a slight linear relationship.
I decide to remove outliers by using Studentized Residuals of the linear model built with all variables. If the Studentized Residuals is bigger than 3 or less than -3, the corresponding record will be removed as an outlier.
However, I forget to do the log transformation for the numeric variables when I remove the outliers, and there are about 700 records are removed. I use the rest of the data to run the random forest, the result is much better than result using log scale to remove.
I think using this method to remove outliers is not appropriate because the data in original scale has no normal distribution. Then I tried the following different approaches to compare the result:
- Run the model with data in log scale with outliers. (Worse)
- Run the model with data in original scale with outliers.(Worse)
- Run the model with data in log scale without outliers, use cooks distance/studentized residuals to remove. (220**) (Worse)
- Run the model with data in log scale without outliers, use mahalanobis distance and chi-square to remove. (460) (Worse)
- Run the model with data in original scale without outliers, use studentized residuals to remove. (720) (Good)
RMSE and MAE are the indicator I used for the evaluation. **the number means how many records are removed.
The more interesting finding is that: for the prediction with residuals bigger than 30, 360 same part IDs from test set show in 3 different models(LASSO, GBM and RF) with outliers in original scale. The outliers generated by the inappropriate way(remove outliers using studentized residual without the log transformation) includes 180 of them. The appropriate way only include about 60 or 70 of them.
I'm curious why this accidentally inappropriate method can bring a better result. Even the result of the the linear regression in log scale deleting outliers generated by the inappropriate way is better than the result after the right way removing outliers.