# What should the form of error be on CrossEntropy or KL-divergence loss function across samples of distributions?

Suppose your model produces (discrete) probability distributions and you have some truth distributions you want to compare to.

For each sample $$i$$, you can compute the loss as the KL divergence or the CrossEntropy, something like this:

$$L_i = \sum_j p_{ij} \log \frac{p_{ij}}{q_{ij}}$$

But what are some motivations for picking the loss over the samples? i.e. what kind of error distribution do we expect in typical situations and why?

For example, you could minimize $$\sum_i |L_i|$$ or $$\sum_i L_i^2$$. They would have different weightings of errors.