# Use of KL Divergence in practice

It's not symmetric, so it can't really be used as a distance metric.

I suppose given two known distributions p(x) and q(x), if one found another distribution z(x) but knew it came from either p or q, you could compare the divergence from each to settle the question.

However I'm from an ML background and I'd like to understand how it might be used in that context (practically not theoretically speaking) - can it be used / is it commonly used to create classifiers or extract features?

• I have never seen it used that way. Outside of its use in proving theoretical results I have seen it being applied as a sort of Bayesian type of Normality test. I.e. given a set of possible models which assume normality in residuals you could use this metric to collect odds ratios about the hypothesis that the models residuals/forecast errors are actually normal. This involves comparing the normality density estimated on the residuals to an emperical estimate like kde. In theory this could aid in selecting well specified models....but haven't seen it used in ML. Commented Dec 9, 2015 at 8:23

The Kullback-Leibler divergence is widely used in variational inference, where an optimization problem is constructed that aims at minimizing the KL-divergence between the intractable target distribution P and a sought element Q from a class of tractable distributions.

The "direction" of the KL divergence then must be chosen such that the expectation is taken with respect to Q to make the task feasible.

Many approximating algorithms (which can also be used to fit probabilistic models to data) can be interpreted in this way. Among those are Mean Field, (Loopy) Belief Propagation (generalizing forward-backward and Viterbi for HMMs), Expectation Propagation, Junction graph/tree, tree-reweighted Belief Propagation and many more.

### References

• Wainwright, M. J. and Jordan, M. I. Graphical models, exponential families, and variational inference, Foundations and Trendstextregistered in Machine Learning, Now Publishers Inc., 2008, Vol. 1(1-2), pp. 1-305
• Yedidia, J. S.; Freeman, W. T. & Weiss, Y. Constructing Free-Energy Approximations and Generalized Belief Propagation Algorithms, Information Theory, IEEE Transactions on, IEEE, 2005, 51, 2282-2312
• Apologies, you might have to dumb this down a little bit or be more specific as opposed to abstract. I know minimization of cross entropy is used as a optimization metric for NN model training if that's what you mean. Can you cite some concrete examples? Commented Dec 9, 2015 at 23:03
• @lollercoaster What would qualify as a "concrete example"? Commented Dec 10, 2015 at 8:34
• an optimization problem is constructed that aims at minimizing the KL-divergence between the intractable target distribution P and a sought element Q from a class of tractable distributions -> can you cite an example perhaps you've worked with with a real dataset and how this helped? Commented Dec 11, 2015 at 20:44
• @lollercoaster For the things mentioned in the question, you do not use the KL divergence explicitly. It is used when deriving and justifying the algorithm. There are many papers that use loopy Belief Propagation for machine learning, just use a search engine, e.g. Commented Dec 14, 2015 at 7:56

KL is widely used in machine learning. The two main ways I know

• compression: compressing a document is actually all about finding a good generative model for it. Given that the true model has probability distribution $p(x)$ while you use the approximate $q(x)$, you will have to use excess bits to encode a sequence of $X$ values. The extra cost you pay is KL(p,q)

• bayesian approximate inference: bayesian methods are great for ML, but it's also extremely computationally expensive to obtain the posterior. Two solutions: either you use sampling methods (MCMC, gibbs, etc) OR you use approximate inference methods which aim at finding a simple (for example Gaussian) approximation to the posterior. Most approximate inference methods refer to KL in some way: so called "variational" (this name sucks) methods minimize KL(q,p), etc. Approximate inference is present in a lot of machine learning research, so KL is too