7
$\begingroup$

I have an issue which I could not solve, although I tried and I got some help on R forum.

I am trying to calculate Mahalanobis distances on a data.frame, where I have several hundreds of groups and several hundreds of variables. Whatever I do, I get the system is computationally singular: reciprocal condition number error.

It is clear that it is singular, but is there any way to get rid of it and run Mahalanobis? Should I forget solve this using another approach? If yes, then what else to use?

I have uploaded the data file to my FTP: It is a tab delimited txt file with no headers.

I was working with the R StatMatch Mahalanobis (also tried stats Mahalanobis) function. I have a deadline for this project (not a homework!), and I could always use this function, so I thought I will be able to keep the calculations short, but now I am lost.

Migrated from Stack Overflow.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ it's essentially a duplicate of this question where the OP wants to compute the Mahalanobis distance of a dataset with singular covariance. $\endgroup$
    – user603
    Commented Jul 28, 2013 at 7:32

3 Answers 3

Reset to default
6
$\begingroup$

Why do you think there is no way that matrix could be singular?

A QR decomposition shows that the rank of this 380 x 372 matrix is just 300. In other words, it is highly singular:

url <- "http://mkk.szie.hu/dep/talt/lv/CentInpDuplNoHeader.txt"
df <- read.table(file = url, header = FALSE)
m <- as.matrix(df)

dim(m)
# [1] 380 372
qr(m)$rank
# [1] 300

Examining the matrix's singular values is another way to see the same thing:

head(table(svd(df)$d))

# 5.76661502353373e-13 2.57650568058543e-12  0.00929562094651422 
#                   71                    1                    1 
#   0.0277990885015625   0.0398152894712022   0.0469713341003743 
#                    1                    1                    1
Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks @Josh for the fast reply. This is a dataset, for different physical and chemical properties of different geological layers and I would not expect it to be highly singular. Whatever, I also calculated svd and You are absolutely right. Anyhow I should still calculate de Maha distance. Is there any idea or solution how to do that if i have such a highly singular datasset? In my previous studies it never happened, although those were smaller and more specified ones. Thanks $\endgroup$
    – user1775772
    Commented Oct 26, 2012 at 0:14
  • $\begingroup$ FYI As someone suggested, I added a small value (between 0.0001-0.0009) to every datavalue. It helped to get rid of the singularity. Thanks for your help $\endgroup$
    – user1775772
    Commented Oct 26, 2012 at 13:08
4
$\begingroup$

A singular matrix means that some of the vectors are linear combinations of others. Thus, some vectors do not add any useful information to the Mahalanobis distance calculation. A generalized inverse or pseudoinverse effectively calculates an "inverse-like" matrix that ignores some of this noninformative information. This is superior to other methods that effectively add in a small amount of incorrect information (i.e. add a small constant to all data). Pseudoinverse covariance matrices have been used successfully with the Mahalanobis distance, see http://www.sciencedirect.com/science/article/pii/0146664X79900522.

Improve this answer
$\endgroup$
-1
$\begingroup$

What I would suggest as a solution is Penalized Mahalanobis distance. You can see this blog post for details http://stefansavev.com/blog/better-euclidean-distance-with-the-svd-penalized-mahalanobis-distance/. You can also check "The Elements of Statistical Learning", by Hastie et al. in particular the sections on ridge regression (it is related) and look up in the index Mahalanobis Distance

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ We're after self-contained answers, not answers that depend crucially on links that may rot (and it's your own blog; unfortunately blogs are often ephemeral). $\endgroup$
    – Nick Cox
    Commented Jun 16, 2015 at 8:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.