Suppose your model produces (discrete) probability distributions and you have some truth distributions you want to compare to.

For each sample $i$, you can compute the loss as the KL divergence or the CrossEntropy, something like this:

$$ L_i = \sum_j p_{ij} \log \frac{p_{ij}}{q_{ij}} $$

But what are some motivations for picking the loss over the samples? i.e. what kind of error distribution do we expect in typical situations and why?

For example, you could minimize $\sum_i |L_i|$ or $\sum_i L_i^2$. They would have different weightings of errors.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.