KL divergence from PDF vs. mean and variance

Question

I am trying to implement the KL divergence between two Gaussian distributions in Python. Since I have the mean and variance from both distributions, I was working with the following formula:

$$ KL(p, q) = \log \frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2 \sigma_2^2} - \frac{1}{2} $$

as referenced here. But when testing and comparing with the KL definition for PDFs:

$$ KL(p,q) = \sum_{x}p(x)\log\frac{p(x)}{q(x)} $$

I get completely different results. Here is a snippet of my code:

    # Distributions' mean and std
    mean1 = 5
    std1 = 3
    mean2 = 10
    std2 = 4

    # PDFs
    x = np.linspace(-10, 30,10000).reshape(-1,1)
    pdf1 = norm.pdf(x, loc = mean1, scale = std1)
    pdf2 = norm.pdf(x, loc = mean2, scale = std2)

    # KLD from mean and variance
    kld_mv = np.log(std2/std1) + (std1**2 + (mean1 - mean2)**2)/(2*std2**2) - 0.5
    print('KLD_MV: ' + str(kld_mv))

    # KLD from PDF
    kld_pdf = np.sum(pdf1*np.log(pdf1/pdf2))
    print('KLD_PDF: ' + str(kld_pdf))

And the results are:

KLD_MV: 0.8501820724517808
KLD_PDF: 212.5242607810292

Are these two formulations supposed to give different values? I can't really find where the error is, however, most values for KLD that I've seen online are close to zero, which leads me to believe that the value from PDF calculation is incorrect. Maybe my understanding of how to implement KLD for PDFs is wrong? Thanks in advance for any help!

Answer 1

The KL divergence formula you wrote is for discrete distributions, although you can just replace the sum with an integral for continous distributions.

Then the error is that when you numerically integrate by discretising as you've done here, you need to multiply by the width of each "vertical slice", which is in your case, 40/10000. You can see that if you multiply your numerical result by this factor, you end up with the closed form.