2
$\begingroup$

I have been using a machine learning algorithm to predict a continuous variable, although am having an issue whereby whichever method I use, there is a systematic bias at low and high values of the variable.

In the below, the orange represents the true=predicted line (the ideal outcome), and the blue dots are predictions from the trained model on an independent test set. I have added the black line to represent more-or-less, the average prediction of the model for a given true value.

enter image description here

I have tried a number of techniques to avoid this over-prediction at low values of the variable, and under-prediction at high values, however they all suffer from the same issue.

My question is: can I correct for this systematic bias in prediction by using this knowledge to post-hoc adjust my predictions from the ML algorithm?

In other words, I get a given prediction, say y=8, from the algorithm. I then use the horizontal difference between the orange line and black line at this prediction value to upwardly-correct this value, moving it more towards 10 (perhaps the true value).

Something in me doesn't feel quite right about doing this. I think that if there were this systematic bias in the machine learning predictions, then the algorithm (minimising RMSE) would have done its best to minimise this.

$\endgroup$
  • 1
    $\begingroup$ It's possible that stacking could help. See this paper by David Wolpert: cs.utsa.edu/~bylander/cs6243/wolpert92stacked.pdf $\endgroup$ – jld Dec 15 '15 at 20:50
  • $\begingroup$ I have a similar problem here stats.stackexchange.com/questions/205858/… and I try to solve it by sustracting the bias .. .this has worked very well for one day in real time now and I will record the performance in the futures ... and yes it feels unsatisfying. $\endgroup$ – Ric Apr 7 '16 at 9:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.