# Distance invariant approaches to find the “main” difference between two distributions in R$^n$

If I have a population of vectors in R$^n$ and some special subset has a different distribution, I could try and use PCA to describe the main axes of these distributions and, if they are aligned differently, that would give me a quick way to describe the difference in orientation between the two. However PCA is dependent on the scale of the components of the vectors. So what is an analagous approach that can cleanly describe the "main" difference in the the two populations given that they have different covariance matrices, but doesn't depend on the scaling given to the componenents? It is possible the answer is more easily stated in terms of the two Mahalanobis distances one gets from the two distributions.

Anyway, I'm sure there is a standard way to describe the "main" difference between two such distributions, but I don't know what it is.

P.S. Yes, assume they have the same mean.

-
The Mahalanobis distance assumes a specific covariance matrix. If the two distribution are to have different covariance matrices how are you defining the distances that you take the ratio of? Does each use its onw estimated covariance? Then how do you interpret this ratio of distances? –  Michael Chernick Aug 31 '12 at 18:03
The whole population has one covariance matrix and hence one mahalanobis distance. I can use that distance to measure the length of any vector. The special sub population has a different covariance matrix and thus its own special mahalanobis distance, which can also be used to measure the length of any vector. So every vector has two lengths. I was considering whether the vectors where ratio of those two lengths is as big as possible and as small as possible would be useful. At the same time, I expect this is a standard problem, and I am just not acquainted with the standard approaches. –  John Robertson Aug 31 '12 at 18:09
Doesn't sound like a familiar or standard problem to me. –  Michael Chernick Aug 31 '12 at 18:13
It seemed like a natural question to me, maybe I have thought something out wrong. I will have to think it out. –  John Robertson Aug 31 '12 at 18:54
I didn't say it was unnatural just not standard. –  Michael Chernick Aug 31 '12 at 19:14

Here is what I have come up with.

The short version is that if $A$ and $B$ are the covariance matrices of two distributions then the eigenvectors and eigenvalues of $AB^{-1}$ give such a characterization. Basically an eigenvector $u$ with eigenvalue $\lambda$ is a direction in which distribution $A$ is $\sqrt{\lambda}$ times as thick as distribution $B$. So if $\sqrt{\lambda}$ is large then $u$ is a direction where the $A$ distribution is thick and the $B$ distribution is thin and if $\sqrt{\lambda}$ is small then the opposite is true, $B$ is thick in direction $u$ and $A$ is thin in that direction.

Any two such eigenvectors are orthogonal in either mahalanobis distance (the one for $A$ or the one for $B$) and if $u$ is one of these eigenvectors with eigenvalue $\lambda$ then the mahalanobis length of $u$ using the $B$ covariance matrix is $\sqrt{\lambda}$ times the mahalanobis length of $u$ using the $A$ covariance matrix.

The largest eigenvalue corresponds to the direction in which A has the most variance relative to B (i.e. where A is the thickest compared to B) and the smallest corresponds to the direction in which B has the most variance relative to A.

The Math If $AB^{-1} u = \lambda u$ then $B^{-1}u = \lambda A^{-1}u$. The $A$ distribution mahalanobis inner product is $<x,y>=x^tA^{-1}y$ and the $B$ distribution mahalanobis inner product is $<x,y>=x^tB^{-1}y$. From these it is easy to prove that any two eigenvectors with different eigenvalues are orthogonal using either inner product and that $\|u\|_B = \sqrt{\lambda} \|u\|_A$ where $\|u\|_A$ is the $A$ distribution mahalanobis length of $u$ and $\|u\|_B$ is the $B$ distribution mahalanobis length of $u$ (note $\lambda$ is always positive).

The eigenvectors form an orthogonal basis using either mahalanobis inner product.

Explanation The point is given any two zero mean multivariable normal distributions in R^n, there is a linear change of coordinates so that BOTH of those normals are independent on each component seperately, i.e. after this change of coordinates both of our normal distributions on R^n are really just n independent one dimensional normals. The eigenvalues described above are exactly the n ratios of the variances one gets from those n pairs of distributions on R.

-