I read this question on Kullback Liebler Divergence

Now i'm have a multidimensional distributions, like these:

for example i try to predict if a person in image is a male:

sample(img)  |     P_true     |     P_pred
             |                |            
             | n/a  |no | yes |  n/a |  no | yes
 A           |  0   | 1 |  0  |  0.2 | 0.6 | 0.2 
 B           |  0   | 0 |  1  |  0.1 | 0.0 | 0.9
 C           |  1   | 0 |  0  |  0.4 | 0.2 | 0.4

where A,B,C are my examples, P_true is the groundtruth (the correct labels) and P_pred, n/a is the label that i can't say if is a male or not.

The p_pred are estimated probability vector (obtained by a softmax).

what is the formula for compute KL divergence with multidimensional probability vector?

  • $\begingroup$ Its the same formula as in univar case, just replace simple integral with multivariate integral $\endgroup$ – kjetil b halvorsen Apr 11 '17 at 8:54
  • $\begingroup$ thus i have a kl value for each element of the array, correct? $\endgroup$ – sdrabb Apr 11 '17 at 9:00
  • $\begingroup$ You must tell us much more, included how to read your arrays! But Kullback-Leibler is a distance between distributions, not between vales. $\endgroup$ – kjetil b halvorsen Apr 11 '17 at 9:04
  • $\begingroup$ i edit my question explaining how read the table $\endgroup$ – sdrabb Apr 11 '17 at 9:09

This is one-dimensional case when you have $P_{true}$ and $P_{pred}$ distributions [each value in your table n/a, no, yes are random variables with assigned probabilities to them].

Let's consider $P_{true}$ as $P$ and $P_{pred}$ as $Q$ then $D_{KL}(P||Q)=\sum_ip_i \log \frac{p_i}{q_i}$.

There is a problem with computing $\log \frac{0}{0.2}$ so I'd recommend to replace all zeros with small values $\epsilon$.

Now you are able to calculate KL-divergence for each example A,B,C and even more :)

That's it.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.