4
$\begingroup$

Say, we want to approximate a distribution $p(x)$ with $q(x|\theta)$. We do not know the distribution $p(x)$ but we can draw samples from $p(x)$. The KL divergence between the two distributions is $$ \begin{aligned} KL(p\Vert q) &= -\int \ln\frac{q(x)}{p(x)}p(x)dx\\ \end{aligned} $$ We can approximate the above quantity by drawing samples from $p(x)$. Replacing the above quantity with expectation, we get $$KL(p\Vert q)=\frac{1}{N}\sum_{i=1}^N\{-\ln \,q(x_i|\theta) + \ln\,p(x_i)\}$$ Minimising this quantity with respect to $\theta$ is the same as maximising the likelihood of $q(x_i|\theta)$.

However, Christopher Bishop writes it as just $\sum_{i=1}^N\{-\ln \,q(x_i|\theta) + \ln\,p(x_i)\}$. Shouldn't there be a $1/N$ in front ?

$\endgroup$

1 Answer 1

4
$\begingroup$

From an optimization perspective the two are equivalent.

$\frac{1}{N}$ is just a constant that multiplies your objective function. As such, it does not affect the location of the interest points (minimums, maximums) with respect to your parameter $\theta$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.