2
$\begingroup$

We all know that $D(p||q) = \sum_x p(x)log\frac{p(x)}{q(x)}$ and it is used to quantify the difference between the true distribution p and the observed distribution q. However, I do not get the intuition on why p(x) is used as the weight in the formula to calculate D(p||q). In the probabilistic point of view D(p||q) can be considered as $D(p||q) = E_{x \sim{p}}log\frac{p(x)}{q(x)}$, hence, q(x) is viewed as a constant? It would be nice if someone can help to explain the intuition of using p(x) as weight.

$\endgroup$
2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.