Multidimensional KL loss

Question

I read this question on Kullback-Liebler Divergence

Now i'm have a multidimensional distributions, like these:

for example I try to predict if a person in image is a male:

sample(img)  |     P_true     |     P_pred
             |                |            
             | n/a  |no | yes |  n/a |  no | yes
 A           |  0   | 1 |  0  |  0.2 | 0.6 | 0.2 
 B           |  0   | 0 |  1  |  0.1 | 0.0 | 0.9
 C           |  1   | 0 |  0  |  0.4 | 0.2 | 0.4

where A,B,C are my examples, P_true is the groundtruth (the correct labels) and P_pred, n/a is the label that i can't say if is a male or not.

The p_pred are estimated probability vector (obtained by a softmax).

what is the formula for compute KL divergence with multidimensional probability vector?

Answer 1

This is one-dimensional case when you have $P_\textrm{true}$ and $P_\textrm{pred}$ distributions [each value in your table n/a, no, yes are random variables with assigned probabilities to them].

Let's consider $P_\textrm{true}$ as $P$ and $P_\textrm{pred}$ as $Q$ then $D_{KL}(P\Vert Q)=\sum_ip_i \log \frac{p_i}{q_i}.$

There is a problem with computing $\log \frac{0}{0.2},$ so I'd recommend to replace all zeros with small values $\epsilon.$

Now you are able to calculate KL-divergence for each example A,B,C and even more :)

That's it.