1
$\begingroup$

i was comparing the results of 3 different techniques for regression task( Deep ensembles, variational inference and concrete dropout) and i got these results

enter image description here

from the table looks like everything is fine and variational inference seems the best option since it leads to lower values, but if i apply $\sigma$-scaling to improve the calibration of the predictive distribution i get negative values of the NLL

enter image description here

but which one among these values is the best one ? should I consider only the modulus or should I take the smallest result overall? In general, should i use the values before calibration to compare the networks or those achieved after applying calibration?

Improve this question
$\endgroup$
7
  • 2
    $\begingroup$ How are you computing NLL? Anyways, lower is better, and sign does not matter. But if you let us know how you're computing NLL we can double check that the numbers are comparable at all. $\endgroup$
    – John Madden
    Oct 31, 2022 at 19:59
  • $\begingroup$ @JohnMadden i am adopting a gaussian model with heteroscedastic error. My neural network learn the variance and the mean in the second to last layer and then plug this values in a final probabilistic layer that model a gaussian. In this way i get both the aleatoric and the epistemic error. $\endgroup$
    – Alucard
    Oct 31, 2022 at 20:28
  • 1
    $\begingroup$ OK cool; we like to use the terms "score" or "predictive NLL" around here for that concept (apps.dtic.mil/sti/pdfs/ADA459827.pdf). Since the NLL's are negative in the second case, they are smaller than the NLL's in the first case, so they are indeed better. $\endgroup$
    – John Madden
    Oct 31, 2022 at 20:30
  • $\begingroup$ sorry, i am a bit confused . in the first message you said sign does not matter but in the second you said that since the NLL are negative they are smaller so they are better. $\endgroup$
    – Alucard
    Oct 31, 2022 at 21:02
  • 1
    $\begingroup$ @JohnMadden ahhhhhhh okok I had interpreted "sign does not matter" as a way to say to consider only the absolute value. My bad, I misunderstood. $\endgroup$
    – Alucard
    Nov 1, 2022 at 18:08

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.