Mahalanobis distance via PCA when $n<p$

Question

I have an $n\times p$ matrix, where $p$ is the number of genes and $n$ is the number of patients. Anyone whose worked with such data knows that $p$ is always larger than $n$. Using feature selection I have gotten $p$ down to a more reasonable number, however $p$ is still greater than $n$.

I would like to compute the similarity of the patients based on their genetic profiles; I could use the euclidean distance, however Mahalanobis seems more appropriate as it accounts for the correlation among the variables. The problem (as noted in this post) is that Mahalanobis distance, specifically the covariance matrix, doesn't work when $n < p$. When I run Mahalanobis distance in R, the error I get is:

 Error in solve.default(cov, ...) :    system is computationally
 singular: reciprocal condition number = 2.81408e-21

So far to try solve this, I've used PCA and instead of using genes, I use components and this seems to allow me to compute the Mahalanobis distance; 5 components represent about 80% of the variance, so now $n > p$.

My questions are: Can I use PCA to meaningfully get the Mahalanobis distance between patients, or is it inappropriate? Are there alternative distance metrics that work when $n < p$ and there is also much correlation among the $n$ variables?

Answer 1

If you keep all the components from a PCA - then the Euclidean distances between patients in the new PCA-space will equal their Mahalanobis distances in the observed-variable space. If you'll skip some components, that will change a little, but anyway. Here I refer to to unit-variance PCA-components, not the kind whose variance is equal to eigenvalue (I am not sure about your PCA implementation).

I just mean, that if you want to evaluate Mahalanobis distance between the patients, you can apply PCA and evaluate Euclidean distance. Evaluating Mahalanobis distance after applying PCA seems something meaningless to me.

Answer 2

Take a look into the following paper:

Zuber, V., Silva, A. P. D., & Strimmer, K. (2012). A novel algorithm for simultaneous SNP selection in high-dimensional genome-wide association studies. BMC bioinformatics, 13(1), 284.

It exactly deals with your problem. The authors suppose the use of a new variable-importance measurements, besides that they earlier introduced a penalized estimation method for the correlation-matrix of explanatory variables which fits your problem. They also use the Mahalanobis distance for decorrelation!

The methods are included in the R-package 'care', available on CRAN

Answer 3

PCA scores (or PCA results) are used in the literature to calculate Mahalanobis distance between sample and a distribution of samples. For an example, see this article. Under the "Analysis methods" section, the authors state:

Data sets of fluorescence spectra (681) are reduced into a lower dimension (11) by evaluating the principal components (PCs) of the correlation matrix (681 × 681). PC scores are estimated by projecting the original data along the PCs. Classification among the data sets has been done using Mahalanobis distance model by computing Mahalanobis distances for the PC scores.

I have seen other examples of PCA/Mahalanobis distance based discriminant analysis in the literature and in the help menu of the GRAMS IQ chemometrics software. This combination makes sense since Mahalanobis distance does not work well when the number of variables is greater than the number of available samples, and PCA reduces the number of variables.

One-class classification machine learning algorithms (i.e. Isolation Forest, One-ClassSVM, etc.) are possible alternatives to the PCA/Mahalanobis distance based discriminant analysis. In our lab, Isolation Forest combined with data pre-processing has produced good results in the classification of Near Infrared spectra.

On a slightly related note, outlier or novelty detection with PCA/Mahalanobis distance, for high dimentional data, often requires calculation of the Mahalanobis distance cutoff. This article suggests that the cutoff can be calculated as the square root of the chi-squared distribution's critical value, assuming that the data is normally distributed. This critical value requires the number of degrees of freedom and the probability value associated with the data. The article appears to suggest that the number of principal components retained equals the number of degrees of freedom needed to calculate the critical value because the authors used the number of features in the data set for their calculation.