I am trying to understand the explanation of the KL divergence per below. It refers, as i understand it, to an expectation in the second term. "Approximating the expectation over q in this term". However, we are multiplying q(x) with the log of p(x) (rather than with p(x). Is it still correct to refer to this construct as an expected value? please let me know.

enter image description here

  • 1
    $\begingroup$ yes, it is the expected value of $log(p(x))$ under the distribution $q$ $\endgroup$
    – seanv507
    Dec 17 '15 at 22:49
  • $\begingroup$ @seanv507 to clarify for future viewers: KL-divergence is the expected value of the difference between of information in mass functions p(x) and q(x) under the distribution q, i.e. E_q [(log(q(x)) - log(p(x))] = E_q [ I_q(x) - I_p(x) ] $\endgroup$ Mar 4 '19 at 11:41

Expected value is a quantity that can be computed for any function of the outcomes.

Let $\Omega$ be the space of all possible outcomes and let $q:\Omega \rightarrow \mathbb{R}$ be a probability distribution defined on $\Omega$. For any function $f:\Omega \rightarrow S$ where $S$ is an arbitrary set that is closed under addition and scalar multiplication (e.g. $S = \mathbb{R}$) we can compute the expected value of $f$ under distribution $q$ as follows: $$ \mathbb{E}[f] = \mathbb{E}_{x \sim q}[f(x)] = \sum_{x \in \Omega} q(x) f(x) $$

In the KL-divergence, we have that $f(x) = \ln{\frac{q(x)}{p(x)}}$ for some fixed $p(x)$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.