I have recently built a model, designed for prediction. Initially, I chose model A over B - better RMSE and better MAPE. However, after carefully evaluating each prediction on my test dataset for those two models, I have concluded that prediction accuracy is consistently higher for model B in terms of those two statistics on most of test dataset observations, except for last few outliers, which blurred the single-numbered statistics. Excluding 10 worst observations from calculating RMSE/MAPE led me to chosing B over A at the end.
I have applied solution that required looking at each observation and compare fit errors in tail of error distribution. The simpler solution can be to calculate statistic on first 90-95% of best fits. Are there any other, better, more grounded in statistics theory solutions?
In case you ask, because I asked myself, why would I want to be blind for observations I am making greatest errors at. Answer is: dependent variable for those observations was probably flawed (wrongly calculated) and my prediction is closer to truth than original value was. But I could only make such a conclusion after I fit the model.