We all know that $D(p||q) = \sum_x p(x)log\frac{p(x)}{q(x)}$ and it is used to quantify the difference between the true distribution p and the observed distribution q. However, I do not get the intuition on why p(x) is used as the weight in the formula to calculate D(p||q). In the probabilistic point of view D(p||q) can be considered as $D(p||q) = E_{x \sim{p}}log\frac{p(x)}{q(x)}$, hence, q(x) is viewed as a constant? It would be nice if someone can help to explain the intuition of using p(x) as weight.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.