# multidimensional KL loss

I read this question on Kullback Liebler Divergence

Now i'm have a multidimensional distributions, like these:

for example i try to predict if a person in image is a male:

sample(img)  |     P_true     |     P_pred
|                |
| n/a  |no | yes |  n/a |  no | yes
A           |  0   | 1 |  0  |  0.2 | 0.6 | 0.2
B           |  0   | 0 |  1  |  0.1 | 0.0 | 0.9
C           |  1   | 0 |  0  |  0.4 | 0.2 | 0.4


where A,B,C are my examples, P_true is the groundtruth (the correct labels) and P_pred, n/a is the label that i can't say if is a male or not.

The p_pred are estimated probability vector (obtained by a softmax).

what is the formula for compute KL divergence with multidimensional probability vector?

• Its the same formula as in univar case, just replace simple integral with multivariate integral – kjetil b halvorsen Apr 11 '17 at 8:54
• thus i have a kl value for each element of the array, correct? – sdrabb Apr 11 '17 at 9:00
• You must tell us much more, included how to read your arrays! But Kullback-Leibler is a distance between distributions, not between vales. – kjetil b halvorsen Apr 11 '17 at 9:04
• i edit my question explaining how read the table – sdrabb Apr 11 '17 at 9:09

This is one-dimensional case when you have $P_{true}$ and $P_{pred}$ distributions [each value in your table n/a, no, yes are random variables with assigned probabilities to them].
Let's consider $P_{true}$ as $P$ and $P_{pred}$ as $Q$ then $D_{KL}(P||Q)=\sum_ip_i \log \frac{p_i}{q_i}$.
There is a problem with computing $\log \frac{0}{0.2}$ so I'd recommend to replace all zeros with small values $\epsilon$.