Comparing similarity between multivariate distributions I'm a non-mathematician trying to get a better intiution into how we can go about saying if one distribution is closer, or further away from another.
For example, if we have a set of patients who each have 20 blood test results and a multiclass label of diagnoses (A, B, C and D). How would one meaningfully say one class is closer or further away from another?
I imagine you can quantify this using KNN or 2-D PCA in seeing how their features (when reduced in dimension) relate to each other and have specific metrics which can provide a quantitation of similarity/difference?
In a similar fashion- would a model which performs well at differentiating between different classes given the features point towards there being is greater distance between classes?
Finally I've come across information theory and Kullback–Leibler divergence - presumably this is also useful in saying whether and how much one (multivariate) distribution compares with another?
Pointers on how to think about this much appreciated.
 A: To elaborate on whuber's comment, consider the case of only categorial features, which means that the distribution is a frequency vector $\vec{h}=(h_1,\ldots,h_d)$. You can compute the Euclidean distance as suggested (your proposal of a 2D PCA projection implicitly uses Euclidean distance, because PCA approximately preserves Euclidean distances), i.e.
$$d(\vec{h},\vec{g})=\sqrt{\sum_{i=1}^d (h_i-g_i)^2}=\sqrt{\langle\vec{h}-\vec{g},\vec{h}-\vec{g}\rangle}=\sqrt{(\vec{h}-\vec{g})^T(\vec{h}-\vec{g})}$$
This does, however, not take into account the distances between the levels that the components of the vector represent. Consider, e.g., the case of a color image where the components represent cells in a color space: In this case, the distance between the cell midpoints should be taken into account (how to possibly define it, is a can of worms that I do not wnat to open here). One way to take it into account is by introducing an "interaction matrix" $A$
$$d(h,g)=\sqrt{(\vec{h}-\vec{g})^T A(\vec{h}-\vec{g})}$$
While it might be possible to estimate $A$ from training data (e.g. $A=\Sigma^{-1}$, which leads to the Mahalanobis distance), it is generally better to construct $A$ on basis of domain knowledge. The same holds for other distance measures between histograms like the "earth mover's distance" which requires a (domain specific) distance between the cells.
A: 
Finally I've come across information theory and Kullback–Leibler divergence - presumably this is also useful in saying whether and how much one (multivariate) distribution compares with another?

The KL divergence certainly can be a very useful metric to compare distributions because it (informally) defines a "distance", in terms of information, between one distribution, $p(x)$, and another distribution, $q(x)$, where the greater the distance, the more unique the two distributions are from each other. The use of "distance" also provides the notion that a KL divergence will always be non-negative, such that if two distributions are equal to each other, $p(x) = q(x)$, then the KL divergence will be 0.
In information theory, the KL divergence can be represented as the difference between the cross entropy of $p$ and $q$ and entropy of $p$:
$$D_{\mathbb{KL}}(p\,||\,q) = \mathbb{H}(p,q) - \mathbb{H}(p)$$
Entropy, and more generally information theory, deals in quantifying uncertainty.
Particularly, entropy, $\mathbb{H}(p)$, defines the innate uncertainty present in a random variable when sampled from, in this case, distribution $p$. If we are totally certain of the outcome for our random variable when sampled from distribution $p$, then its entropy is 0. The more uncertainty present however, the higher the entropy.
The cross entropy, $\mathbb{H}(p, q)$, defines how much more uncertainty there will be if we sample a random variable, with distribution $p$, when encoded to a new distribution $q$. The above representation of the KL divergence can be interpreted as finding that "excess" amount of uncertainty present in the cross entropy between distributions $p$ and $q$.
By this same logic (and to circle back to the top), if we had a hypothetical/magical dial that we could turn to reduce that "excess" information (kl divergence), we would be mutating distribution $q$ to be more alike distribution $p$.
