3
$\begingroup$

I want to compute the Kullback-Leibler divergence (KL) of two Gaussians, the first with mean of 1 and the second -1, where both have the same variance say, 1.

In MATLAB, the distributions are: 

y1 = normpdf([-10:0.1:10], -1, 1)
y2 = normpdf([-10:0.1:10],  1, 1)

The code I used to compute the KL is: 

KL = sum(Apdf .* (log2(Apdf)-log2(Bpdf)))

Are these the inputs I should use for the KL? The result I got is 28, shouldn't it be 2?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ I suppose some people will say that your question if off-topic for this board since you are asking about how to do a computation in a specific language/package/environment. Nevertheless, you haven't told us how you attempted to to a KL divergence calculation in MATLAB. As far as I know, there's no built-in function or Mattworks supplied toolbox to calculate KL divergence. Therefore, perhaps you are using a 3td party function/toolbox, but you haven't told us which one or in what manner you used it. $\endgroup$
    – Mark L. Stone
    Aug 11, 2016 at 15:02
  • 1
    $\begingroup$ In any event, it's hard to see how y1 and y2 could be adequate inputs to any function to compute the KL divergence of two Gaussians. Specifically, note that your argument [-10 10] calculates and returns the pdf only at the 2 argument values, -10 and 10, not at an array of closely spaced values. Perhaps you meant something like -10:0.01:10, which is an array of 2001 values. $\endgroup$
    – Mark L. Stone
    Aug 11, 2016 at 15:13
  • $\begingroup$ yes, indeed, it is -10:0.1:10. I just changed it. The KL is : KL = sum(Apdf .* (log2(Apdf)-log2(Bpdf))) $\endgroup$
    – Adrien W.
    Aug 11, 2016 at 15:27

2 Answers 2

Reset to default
4
$\begingroup$

There are two reasons why you did not get the answer 2.

1) The KL divergence being 2 is based on use of the natural log, which in MATLAB is log.

2) If you used log instead of log2 in your code, you would get the result 20. The reason is that in performing the integration, you neglected to multiply by the discretization increment between points, which in your calculation was 0.1.

Here is a correct solution:

>> Apdf = normpdf([-10:0.1:10], -1, 1);
>> Bpdf = normpdf([-10:0.1:10],  1, 1);
>> KL = 0.1 * sum(Apdf .* (log(Apdf)-log(Bpdf)))
KL =
   2.000000000000000
Improve this answer
$\endgroup$
2
$\begingroup$

I believe you should get about 2.89. For continuous distributions (such as Gaussian) KL is has an integral definition. In order to make the sum into a numerical integration, just divide by the sampling rate of 10 (from the steps of 0.1).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.