2
$\begingroup$

When minimizing the KL divergence in machine learning, why the expression of KL is from data to model, instead of from model to data ?

$\endgroup$
  • $\begingroup$ not sure if you're referring to variational inference, but if you are, minimizing the kl divergence in the other direction is called expectation propagation $\endgroup$ – aleshing Sep 12 '18 at 20:48
1
$\begingroup$

One explanation is that this is what maximizing (log) likelihood gives you.

Suppose you have observed coin tosses $y_i \in \{0,1\}, i=1,\dots,n$ and wish to estimate the coin flip probability $p$. Maximizing likelihood is \begin{align*} \arg \max_p \prod_{i=1}^n p^{y_i}(1-p)^{1-y_i} &= \arg\max_p \sum_{i=1}^n y_i \log p + (1-y_i)\log(1-p) \\ &= \arg\max_p -\sum_{i=1}^n y_i \log \frac{1}{p} + (1-y_i)\log \frac{1}{1-p} \\ &= \arg\min_p \sum_{i=1}^n y_i \log \frac{y_i}{p} + (1-y_i)\log \frac{1-y_i}{1-p} \\ &= \arg\min_p \sum_{i=1}^n \text{KL}(y_i,p) \end{align*} since $0 \log 0 = 0$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.