Use of KL Divergence in practice It's not symmetric, so it can't really be used as a distance metric. 
I suppose given two known distributions p(x) and q(x), if one found another distribution z(x) but knew it came from either p or q, you could compare the divergence from each to settle the question. 
However I'm from an ML background and I'd like to understand how it might be used in that context (practically not theoretically speaking) - can it be used / is it commonly used to create classifiers or extract features?
 A: The Kullback-Leibler divergence is widely used in variational inference, where an optimization problem is constructed that aims at minimizing the KL-divergence between the intractable target distribution P and a sought element Q from a class of tractable distributions.
The "direction" of the KL divergence then must be chosen such that the expectation is taken with respect to Q to make the task feasible.
Many approximating algorithms (which can also be used to fit probabilistic models to data) can be interpreted in this way. Among those are Mean Field, (Loopy) Belief Propagation (generalizing forward-backward and Viterbi for HMMs), Expectation Propagation, Junction graph/tree, tree-reweighted Belief Propagation and many more.
References


*

*Wainwright, M. J. and Jordan, M. I.
Graphical models, exponential families, and variational inference,
Foundations and Trendstextregistered in Machine Learning, Now Publishers Inc., 2008, Vol. 1(1-2), pp. 1-305

*Yedidia, J. S.; Freeman, W. T. & Weiss, Y. Constructing Free-Energy Approximations and Generalized Belief Propagation Algorithms,
Information Theory, IEEE Transactions on, IEEE, 2005, 51, 2282-2312

A: KL is widely used in machine learning. The two main ways I know


*

*compression: compressing a document is actually all about finding a good generative model for it. Given that the true model has probability distribution $p(x)$ while you use the approximate $q(x)$, you will have to use excess bits to encode a sequence of $X$ values. The extra cost you pay is KL(p,q)

*bayesian approximate inference: bayesian methods are great for ML, but it's also extremely computationally expensive to obtain the posterior. Two solutions: either you use sampling methods (MCMC, gibbs, etc) OR you use approximate inference methods which aim at finding a simple (for example Gaussian) approximation to the posterior. Most approximate inference methods refer to KL in some way: so called "variational" (this name sucks) methods minimize KL(q,p), etc. Approximate inference is present in a lot of machine learning research, so KL is too
