I have a problem for which I want to build a model that predicts probabilities with uncertainties.
As an example, let's say I want to predict the probability that it's going to rain today. My model can predict 90% +- 2%, and I will know that it is very certain about its estimate. It could also predict 40% +- 40% if it's very uncertain, and I will know that I know nothing.
Is there an obvious loss function I can use to compare the performance of several models like this?
Ideally, it would give less weight to uncertain predictions: if you're uncertain, it's not as bad to be wrong, but not as good to be right either.