4
$\begingroup$

I have a problem for which I want to build a model that predicts probabilities with uncertainties.

As an example, let's say I want to predict the probability that it's going to rain today. My model can predict 90% +- 2%, and I will know that it is very certain about its estimate. It could also predict 40% +- 40% if it's very uncertain, and I will know that I know nothing.

Is there an obvious loss function I can use to compare the performance of several models like this?

Ideally, it would give less weight to uncertain predictions: if you're uncertain, it's not as bad to be wrong, but not as good to be right either.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

You'd probably want to formulate the output as a distribution over probabilities. So maybe 90%+-2% is a uniform distribution over [88%, 92%]. Then you could apply any ordinary loss function by taking its expectation over your output distribution. For example, if you're doing log loss, replace $-\log(0.9)$ (assuming it rained that day) with $-\int_{0.88}^{0.92} \log(x)= x(1-\log(x)) |_{0.88}^{0.92}=0.92 (1- \log(0.92)) - 0.88(1 - \log(0.88))$

Improve this answer
$\endgroup$
2
  • $\begingroup$ Unfortunately I don't think this is going to do it. Given a binary sample this score isn't unique for different values of [p1, p2], so it won't help to distinguish between different models. $\endgroup$
    – Dion
    Dec 9, 2021 at 11:09
  • $\begingroup$ It's exactly as unique for [p1, p2] as a typical loss function is for p in the sense that changing p1, and p2 will almost never give you the same loss as before for arbitrary changes (as is the case for changing p for a typical loss function). Maybe you could illustrate this requirement with an example? $\endgroup$
    – Alex Eftimiades
    Dec 13, 2021 at 16:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.