# Loss function for probabilities with uncertainties

I have a problem for which I want to build a model that predicts probabilities with uncertainties.

As an example, let's say I want to predict the probability that it's going to rain today. My model can predict 90% +- 2%, and I will know that it is very certain about its estimate. It could also predict 40% +- 40% if it's very uncertain, and I will know that I know nothing.

Is there an obvious loss function I can use to compare the performance of several models like this?

Ideally, it would give less weight to uncertain predictions: if you're uncertain, it's not as bad to be wrong, but not as good to be right either.

You'd probably want to formulate the output as a distribution over probabilities. So maybe 90%+-2% is a uniform distribution over [88%, 92%]. Then you could apply any ordinary loss function by taking its expectation over your output distribution. For example, if you're doing log loss, replace $$-\log(0.9)$$ (assuming it rained that day) with $$-\int_{0.88}^{0.92} \log(x)= x(1-\log(x)) |_{0.88}^{0.92}=0.92 (1- \log(0.92)) - 0.88(1 - \log(0.88))$$