4
$\begingroup$

I'm working on a project where my business counterparts want my model's distribution of continuous predictions to more closely represent the distribution of actual values. Below is an example of what my model looks like now.

enter image description here

This is how the business would like it to behave instead.

enter image description here

Which loss function would encourage the model to more closely resemble the second graph? Currently I'm using RMSE as my loss function, but I'm curious if other loss functions like median absolute error would lead to a model with a better spread.

$\endgroup$

1 Answer 1

5
$\begingroup$

I strongly suspect the following:

The Actuals you train your model on (the yellowish bars in your histogram) consist of both signal and noise. Any prediction method will predict the predictable part - which is, by definition, the signal. Noise is not predictable, again by definition.

Therefore your predictions will be less dispersed than the actual observations.

Note that this does not mean that your model is "bad". It is simply a straightforward consequence of observations consisting both of modelable signal and non-modelable noise.

To be quite honest, I fail to see why your business would prefer to see noisy predictions. Because that is what they are asking for. I don't know of a single business decision that would be improved by adding noise to the predictions (which are the inputs to your decision method). Note that this is of course different from planning for or accounting for noise, e.g., by producing not only for the expected demand, but by adding a safety amount to account for noisy demand!


That said, here is an extremely simple way of satisfying your business if what they want is noise: do not go with expectation point predictions. Instead, produce full density predictions. Then sample from those. If your density forecast is correct, the histogram from your samples should be identical to the actually observed histogram.

(I still think you should try to understand what your colleagues actually want, and what business decisions they will take based on your predictions.)

$\endgroup$
2
  • $\begingroup$ I understand and completely agree with your points about fitting to noise and trying to understand the business problem better. Unfortunately I can't convince the stakeholders to evaluate the model differently. ... How would you define the density predictions? Would you separate the data into n-tiles and then create a multi-class classifier that outputs probability of membership in each n-tile or would you predict a given point's mean and standard deviation? $\endgroup$
    – Ryan Zotti
    Commented Jul 22, 2019 at 21:26
  • 1
    $\begingroup$ I would definitely not discretize the data into bins, unless I had a really special reason for it. Predicting the mean + SD would be better if you can assume a normal distribution. Depending on your problem, another distribution might be more reasonable. Or a Generalized Additive Model, e.g., Poisson regression. Take a look at proper scoring rules to evaluate your density prediction. $\endgroup$ Commented Jul 23, 2019 at 6:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.