I was playing with the KL divergence. My simple example is to calculate the divergence between two 2-dimensional normal distribution using PyTorch. The code for doing it is just below.

I get an unexpected huge number as result (~3e15), so I suspect there is something I am missing about KL. Can you point me to understand what?

import numpy as np
import torch 
from torch.distributions.kl import kl_divergence
from torch.distributions.multivariate_normal import MultivariateNormal

gaus1_mu=np.array([0.1, 0.1])
gaus1_sigma=np.array([[0.5, 0.5], [0.5, 0.5]])

gaus2_mu=np.array([0.1, 1.])
gaus2_sigma=np.array([[0.5, 1.5], [0.5, 0.5]])

gaus1_t=MultivariateNormal(loc=torch.from_numpy(gaus1_mu), covariance_matrix=torch.from_numpy(gaus1_sigma)) 
gaus2_t=MultivariateNormal(loc=torch.from_numpy(gaus2_mu), covariance_matrix=torch.from_numpy(gaus2_sigma)) 

result=kl_divergence(gaus1_t, gaus2_t)           #~3e15

closed as off-topic by whuber Nov 16 '18 at 15:11

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – whuber
If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ You haven't even defined gaus3_t in your code. $\endgroup$ – whuber Nov 16 '18 at 15:12
  • $\begingroup$ I see it has been put on hold, but I don't understand why. Should I suppose this issue is not about the KL divergence, and it's instead a bug in PyTorch? $\endgroup$ – Vincenzo Lavorini Nov 17 '18 at 17:20
  • $\begingroup$ I don't think it's a bug in PyTorch. However, it would be very helpful if you were to explain in a common language--mathematics or English--what it is you expect this code to do. In particular, please tell us what you intend gaus1_sigma1 and gaus2_sigma to represent. If I interpret them correctly, the former is singular and the latter isn't even a covariance matrix. $\endgroup$ – whuber Nov 17 '18 at 19:36
  • $\begingroup$ They simply are two random 2-d gaussians. And with random I mean that I just put numbers in their mean values and covariance matrices. But as I read your comment, I understand that the problem is that I cannot simply put numbers in a square matrix to create a covariance matrix, and by conseguence my two funciton are not proper distributions. But in this case, I wonder why PyTorch do not raise an error when I create the MultivariateNormal with them $\endgroup$ – Vincenzo Lavorini Nov 18 '18 at 9:46