# Mahalanobis distance via PCA when $n<p$

I have an $n\times p$ matrix, where $p$ is the number of genes and $n$ is the number of patients. Anyone whose worked with such data knows that $p$ is always larger than $n$. Using feature selection I have gotten $p$ down to a more reasonable number, however $p$ is still greater than $n$.

I would like to compute the similarity of the patients based on their genetic profiles; I could use the euclidean distance, however Mahalanobis seems more appropriate as it accounts for the correlation among the variables. The problem (as noted in this post) is that Mahalanobis distance, specifically the covariance matrix, doesn't work when $n < p$. When I run Mahalanobis distance in R, the error I get is:

 Error in solve.default(cov, ...) :    system is computationally
singular: reciprocal condition number = 2.81408e-21


So far to try solve this, I've used PCA and instead of using genes, I use components and this seems to allow me to compute the Mahalanobis distance; 5 components represent about 80% of the variance, so now $n > p$.

My questions are: Can I use PCA to meaningfully get the Mahalanobis distance between patients, or is it inappropriate? Are there alternative distance metrics that work when $n < p$ and there is also much correlation among the $n$ variables?

• The PCA could break the variable correlations, unless you use something like an oblique rotation. I'm also not sure how the variance apportioning in PCA will affect the Mahalanobis distance between similar patients. – Michelle Mar 7 '12 at 4:48
• If PCA breaks the variable correlations, could I use some other distance metric (such as Pearson's distance) instead of the Mahalanobis distance? – user4673 Mar 8 '12 at 19:22
• I don't know enough to recommend further. I was hoping someone else would jump into the conversation. :) Given how variable reduction techniques like PCA work, I am curious as to whether any distance metric can be used on the outputs. – Michelle Mar 8 '12 at 19:40
• I'm not an expert in the area. I just want to share, that I've a smilar problem of yours, and I've used 'cov.shrink' from corpcor package. – Bruno Sousa Jun 8 '12 at 13:56

If you keep all the components from a PCA - then the Euclidean distances between patients in the new PCA-space will equal their Mahalanobis distances in the observed-variable space. If you'll skip some components, that will change a little, but anyway. Here I refer to to unit-variance PCA-components, not the kind whose variance is equal to eigenvalue (I am not sure about your PCA implementation).

I just mean, that if you want to evaluate Mahalanobis distance between the patients, you can apply PCA and evaluate Euclidean distance. Evaluating Mahalanobis distance after applying PCA seems something meaningless to me.

• Here's a short article discussing the relation: Brereton, R. G. The Mahalanobis distance and its relationship to principal component scores Journal of Chemometrics, Wiley-Blackwell, 2015, 29, 143–145. dx.doi.org/10.1002/cem.2692. Note that PCA in chemometrics is by default a pure rotation of the data, thus not the unit-variance version Dmitry refers to. – cbeleites unhappy with SX Aug 19 '16 at 8:14

Take a look into the following paper:

Zuber, V., Silva, A. P. D., & Strimmer, K. (2012). A novel algorithm for simultaneous SNP selection in high-dimensional genome-wide association studies. BMC bioinformatics, 13(1), 284.

It exactly deals with your problem. The authors suppose the use of a new variable-importance measurements, besides that they earlier introduced a penalized estimation method for the correlation-matrix of explanatory variables which fits your problem. They also use the Mahalanobis distance for decorrelation!

The methods are included in the R-package 'care', available on CRAN

PCA scores (or PCA results) are used in the literature to calculate Mahalanobis distance between sample and a distribution of samples. For an example, see this article. Under the "Analysis methods" section, the authors state:

Data sets of fluorescence spectra (681) are reduced into a lower dimension (11) by evaluating the principal components (PCs) of the correlation matrix (681 × 681). PC scores are estimated by projecting the original data along the PCs. Classification among the data sets has been done using Mahalanobis distance model by computing Mahalanobis distances for the PC scores.

I have seen other examples of PCA/Mahalanobis distance based discriminant analysis in the literature and in the help menu of the GRAMS IQ chemometrics software. This combination makes sense since Mahalanobis distance does not work well when the number of variables is greater than the number of available samples, and PCA reduces the number of variables.

One-class classification machine learning algorithms (i.e. Isolation Forest, One-ClassSVM, etc.) are possible alternatives to the PCA/Mahalanobis distance based discriminant analysis. In our lab, Isolation Forest combined with data pre-processing has produced good results in the classification of Near Infrared spectra.

On a slightly related note, outlier or novelty detection with PCA/Mahalanobis distance, for high dimentional data, often requires calculation of the Mahalanobis distance cutoff. This article suggests that the cutoff can be calculated as the square root of the chi-squared distribution's critical value, assuming that the data is normally distributed. This critical value requires the number of degrees of freedom and the probability value associated with the data. The article appears to suggest that the number of principal components retained equals the number of degrees of freedom needed to calculate the critical value because the authors used the number of features in the data set for their calculation.

• Welcome to our site. Because it isn't explicit in your answer, could you explain how it addresses the $n\lt p$ circumstance posited in the question? – whuber May 23 '19 at 17:13
• When n < p, PCA is used to reduce p. The number of components retained from PCA analysis (nc) is usually much smaller than p, and it may also be smaller than n depending on the size of n. Thus, PCA changes the "n by p" problem to an "n by nc" problem. In our lab where we analyse NIR spectra, p represents the number of wavelengths, which is typically greater than two thousands. n represents the number of training samples (~ 20 to 150). nc is typically 3 to 11. When n > nc, PCA/Mahalanobis analysis produces good results. – Cherif Diallo May 23 '19 at 17:59
• The issue isn't what PCA does. That's well covered in other threads on this site. The problem is that the mahalanobis distance typically cannot be computed in this case because of the singularity of the covariance matrix: this is the circumstance described in the question. – whuber May 23 '19 at 18:03
• Thank you for your kind comments. It appears that we have different interpretations of what the user is asking. The accepted answer states, "if you want to evaluate Mahalanobis distance between the patients, you can apply PCA and evaluate Euclidean distance. Evaluating Mahalanobis distance after applying PCA seems something meaningless ...". I think the last sentence is at odds with the literature I have seen and cited. What do you think? Best regards. – Cherif Diallo May 23 '19 at 19:15
• Got it. Thanks :)! – Cherif Diallo May 23 '19 at 19:30