# Multidimensional KL loss

I read this question on Kullback-Liebler Divergence

Now i'm have a multidimensional distributions, like these:

for example I try to predict if a person in image is a male:

sample(img)  |     P_true     |     P_pred
|                |
| n/a  |no | yes |  n/a |  no | yes
A           |  0   | 1 |  0  |  0.2 | 0.6 | 0.2
B           |  0   | 0 |  1  |  0.1 | 0.0 | 0.9
C           |  1   | 0 |  0  |  0.4 | 0.2 | 0.4


where A,B,C are my examples, P_true is the groundtruth (the correct labels) and P_pred, n/a is the label that i can't say if is a male or not.

The p_pred are estimated probability vector (obtained by a softmax).

what is the formula for compute KL divergence with multidimensional probability vector?

• Its the same formula as in univar case, just replace simple integral with multivariate integral Commented Apr 11, 2017 at 8:54
• thus i have a kl value for each element of the array, correct? Commented Apr 11, 2017 at 9:00
• You must tell us much more, included how to read your arrays! But Kullback-Leibler is a distance between distributions, not between vales. Commented Apr 11, 2017 at 9:04
• i edit my question explaining how read the table Commented Apr 11, 2017 at 9:09
• KLD is a divergence, not a distance. Commented Jul 9, 2021 at 15:53

This is one-dimensional case when you have $$P_\textrm{true}$$ and $$P_\textrm{pred}$$ distributions [each value in your table n/a, no, yes are random variables with assigned probabilities to them].
Let's consider $$P_\textrm{true}$$ as $$P$$ and $$P_\textrm{pred}$$ as $$Q$$ then $$D_{KL}(P\Vert Q)=\sum_ip_i \log \frac{p_i}{q_i}.$$
There is a problem with computing $$\log \frac{0}{0.2},$$ so I'd recommend to replace all zeros with small values $$\epsilon.$$