6
$\begingroup$

It's not symmetric, so it can't really be used as a distance metric.

I suppose given two known distributions p(x) and q(x), if one found another distribution z(x) but knew it came from either p or q, you could compare the divergence from each to settle the question.

However I'm from an ML background and I'd like to understand how it might be used in that context (practically not theoretically speaking) - can it be used / is it commonly used to create classifiers or extract features?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ I have never seen it used that way. Outside of its use in proving theoretical results I have seen it being applied as a sort of Bayesian type of Normality test. I.e. given a set of possible models which assume normality in residuals you could use this metric to collect odds ratios about the hypothesis that the models residuals/forecast errors are actually normal. This involves comparing the normality density estimated on the residuals to an emperical estimate like kde. In theory this could aid in selecting well specified models....but haven't seen it used in ML. $\endgroup$
    – Zachary Blumenfeld
    Commented Dec 9, 2015 at 8:23

2 Answers 2

Reset to default
8
$\begingroup$

The Kullback-Leibler divergence is widely used in variational inference, where an optimization problem is constructed that aims at minimizing the KL-divergence between the intractable target distribution P and a sought element Q from a class of tractable distributions.

The "direction" of the KL divergence then must be chosen such that the expectation is taken with respect to Q to make the task feasible.

Many approximating algorithms (which can also be used to fit probabilistic models to data) can be interpreted in this way. Among those are Mean Field, (Loopy) Belief Propagation (generalizing forward-backward and Viterbi for HMMs), Expectation Propagation, Junction graph/tree, tree-reweighted Belief Propagation and many more.

References

  • Wainwright, M. J. and Jordan, M. I. Graphical models, exponential families, and variational inference, Foundations and Trendstextregistered in Machine Learning, Now Publishers Inc., 2008, Vol. 1(1-2), pp. 1-305
  • Yedidia, J. S.; Freeman, W. T. & Weiss, Y. Constructing Free-Energy Approximations and Generalized Belief Propagation Algorithms, Information Theory, IEEE Transactions on, IEEE, 2005, 51, 2282-2312
Improve this answer
$\endgroup$
4
  • $\begingroup$ Apologies, you might have to dumb this down a little bit or be more specific as opposed to abstract. I know minimization of cross entropy is used as a optimization metric for NN model training if that's what you mean. Can you cite some concrete examples? $\endgroup$
    – lollercoaster
    Commented Dec 9, 2015 at 23:03
  • $\begingroup$ @lollercoaster What would qualify as a "concrete example"? $\endgroup$
    – ziggystar
    Commented Dec 10, 2015 at 8:34
  • $\begingroup$ an optimization problem is constructed that aims at minimizing the KL-divergence between the intractable target distribution P and a sought element Q from a class of tractable distributions -> can you cite an example perhaps you've worked with with a real dataset and how this helped? $\endgroup$
    – lollercoaster
    Commented Dec 11, 2015 at 20:44
  • $\begingroup$ @lollercoaster For the things mentioned in the question, you do not use the KL divergence explicitly. It is used when deriving and justifying the algorithm. There are many papers that use loopy Belief Propagation for machine learning, just use a search engine, e.g. $\endgroup$
    – ziggystar
    Commented Dec 14, 2015 at 7:56
6
$\begingroup$

KL is widely used in machine learning. The two main ways I know

  • compression: compressing a document is actually all about finding a good generative model for it. Given that the true model has probability distribution $p(x)$ while you use the approximate $q(x)$, you will have to use excess bits to encode a sequence of $X$ values. The extra cost you pay is KL(p,q)

  • bayesian approximate inference: bayesian methods are great for ML, but it's also extremely computationally expensive to obtain the posterior. Two solutions: either you use sampling methods (MCMC, gibbs, etc) OR you use approximate inference methods which aim at finding a simple (for example Gaussian) approximation to the posterior. Most approximate inference methods refer to KL in some way: so called "variational" (this name sucks) methods minimize KL(q,p), etc. Approximate inference is present in a lot of machine learning research, so KL is too

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.