Mahalanobis distance on singular data

I have an issue which I could not solve, although I tried and I got some help on R help forums too. I am trying to calculate Mahalanobis distances on a data frame, where I have several hundreds of groups and several hundreds of variables. Whatever I do, however I subset it I get the "system is computationally singular: reciprocal condition number" error.

It is clear that it is singular, but is there any way to get rid of it and run Maha? Should I forget Maha? Than what to use?

I have uploaded the data file to my ftp: http://mkk.szie.hu/dep/talt/lv/CentInpDuplNoHeader.txt It is a tab delimited txt file with no headers.

I was working with the (R) StatMatch Mahalanobis (also tried stats Mahalanobis) function. I have a deadline for this project (not a homework:)), and I could always use this funct, so I thought I will be able to quit the calculations short, but now I am just lost. link to previous question, sorry for crossposting, I have no idea how to migrate the previous one. http://stackoverflow.com/questions/13078909/system-is-computationally-singular-reciprocal-condition-number I would really appreciate any help.

Thanks for any help

-
it's essentially a duplicate of this question where the OP wants to compute the Mahalanobis distance of a dataset with singular covariance. –  user603 Jul 28 '13 at 7:32

Why do you think there is no way that matrix could be singular?

A QR decomposition shows that the rank of this 380 x 372 matrix is just 300. In other words, it is highly singular:

url <- "http://mkk.szie.hu/dep/talt/lv/CentInpDuplNoHeader.txt"
m <- as.matrix(df)

dim(m)
# [1] 380 372
qr(m)$rank # [1] 300  Examining the matrix's singular values is another way to see the same thing: head(table(svd(df)$d))

# 5.76661502353373e-13 2.57650568058543e-12  0.00929562094651422
#                   71                    1                    1
#   0.0277990885015625   0.0398152894712022   0.0469713341003743
#                    1                    1                    1

-
Thanks @Josh for the fast reply. This is a dataset, for different physical and chemical properties of different geological layers and I would not expect it to be highly singular. Whatever, I also calculated svd and You are absolutely right. Anyhow I should still calculate de Maha distance. Is there any idea or solution how to do that if i have such a highly singular datasset? In my previous studies it never happened, although those were smaller and more specified ones. Thanks –  user1775772 Oct 26 '12 at 0:14
FYI As someone suggested, I added a small value (between 0.0001-0.0009) to every datavalue. It helped to get rid of the singularity. Thanks for your help –  user1775772 Oct 26 '12 at 13:08

A singular matrix means that some of the vectors are linear combinations of others. Thus, some vectors do not add any useful information to the Mahalanobis distance calculation. A generalized inverse or pseudoinverse effectively calculates an "inverse-like" matrix that ignores some of this noninformative information. This is superior to other methods that effectively add in a small amount of incorrect information (i.e. add a small constant to all data). Pseudoinverse covariance matrices have been used successfully with the Mahalanobis distance, see http://www.sciencedirect.com/science/article/pii/0146664X79900522.

-

What I would suggest as a solution is Penalized Mahalanobis distance. You can see this blog post for details http://stefansavev.com/blog/better-euclidean-distance-with-the-svd-penalized-mahalanobis-distance/. You can also check "The Elements of Statistical Learning", by Hastie et al. in particular the sections on ridge regression (it is related) and look up in the index Mahalanobis Distance

-
We're after self-contained answers, not answers that depend crucially on links that may rot (and it's your own blog; unfortunately blogs are often ephemeral). –  Nick Cox Jun 16 at 8:41