When minimizing the KL divergence in machine learning, why the expression of KL is from data to model, instead of from model to data ?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ not sure if you're referring to variational inference, but if you are, minimizing the kl divergence in the other direction is called expectation propagation $\endgroup$– aleshingCommented Sep 12, 2018 at 20:48
-
$\begingroup$ See also stats.stackexchange.com/questions/482362/kl-divergence-pq-vs-qp/… $\endgroup$– kjetil b halvorsen ♦Commented Sep 21, 2021 at 22:27
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
One explanation is that this is what maximizing (log) likelihood gives you.
Suppose you have observed coin tosses $y_i \in \{0,1\}, i=1,\dots,n$ and wish to estimate the coin flip probability $p$. Maximizing likelihood is \begin{align*} \arg \max_p \prod_{i=1}^n p^{y_i}(1-p)^{1-y_i} &= \arg\max_p \sum_{i=1}^n y_i \log p + (1-y_i)\log(1-p) \\ &= \arg\max_p -\sum_{i=1}^n y_i \log \frac{1}{p} + (1-y_i)\log \frac{1}{1-p} \\ &= \arg\min_p \sum_{i=1}^n y_i \log \frac{y_i}{p} + (1-y_i)\log \frac{1-y_i}{1-p} \\ &= \arg\min_p \sum_{i=1}^n \text{KL}(y_i,p) \end{align*} since $0 \log 0 = 0$.
