# How to deal with overestimation of small values and underestimation of high values in XGBoost? [closed]

I'm running XGBoost to predict prices on a cars dataset, I was wondering what alternatives are there for this kind of problem where smaller values are overestimated and higher prices underestimated.

I tried applying log to prices since it has a skewed to the right distribution, but still having this undesirable effect.

Also, as a bonus question, log(price) improved the prediction score, the mean relative error or MRE calculated as mean(ABS(RD)) by 2 percent, if anyone has the intuition onto why this could have happened that would be great.

In the image below RD is the relative difference between predictions and the actual values, and the price bucket is a bucketized variable where the number indicates the price low interval bound over 1000.

• (1) In the title you say that small values are overestimated, in the first paragraph that small values are underestimated. Can you please clarify? (2) This sounds much like straightforward regression toward the mean, see also here. (3) What "prediction score" are you using specifically? I have a suspicion why your log might improve it but need to understand more precisely what the KPI is. Commented Feb 14, 2019 at 9:20
• @StephanKolassa The RD metric on the y axis is (prediction-price)/price, was that the question? Commented Feb 14, 2019 at 9:23
• Did you take absolute values, and this absolute value went down by 2%? Or how else did the RD "improve by 2%"? Commented Feb 14, 2019 at 9:41
• @StephanKolassa Oh yes regarding the Bonus question what improved by 2% is the relative error, so yes the absolut. Commented Feb 14, 2019 at 9:55
• @StephanKolassa just clarified the 1st paragraph, thanks for the catch. Commented Feb 14, 2019 at 9:56

• You can back transform a log-scale fit $\hat{y}$ to an expectation fit by $\exp(\hat{y}+\frac{\hat{\sigma}^2}{2})$, see here. Of course, you can aim for the conditional median or any other functional of the future density, but be aware that the point forecasts may differ dramatically, so you should really know what you are doing and how your point prediction will be used by whoever consumes it. Commented Feb 14, 2019 at 15:53