Robust PCA (as developed by Candes et al 2009 or better yet Netrepalli et al 2014) is a popular method for multivariate outlier detection, but Mahalanobis distance can also be used for outlier detection given a robust, regularized estimate of the covariance matrix. I'm curious about the (dis)advantages of using one method over the other.

My intuition tells me that the greatest distinction between the two is the following: When the data set is "small" (in a statistical sense), robust PCA will give a lower-rank covariance while robust covariance matrix estimation will instead give a full-rank covariance due to the Ledoit-Wolf regularization. How does this in turn affect outlier detection?

  • $\begingroup$ Interesting question but I cannot see how an answer can be motivated without a specific use-case. Do you have "grossly corrupted observations"? Do you have generally noisy data? A number of RPCA implementations essentially are robust covariance estimation techniques (see Jolliffe's Princ. Component Analysis, Ed. 2nd Ch. 10) where the PCs are estimated from the regularised estimate of the covariance. Thus, the distinctions from the two approach you mention is far from clear-cut. In general, automatic outlier detection is successful within the context of a particular application. $\endgroup$ – usεr11852 Oct 21 '17 at 15:19
  • 1
    $\begingroup$ The “noisy data” problem isn’t outlier detection. I think the outlier detection problem is restrictive enough on its own to allow for a general comparison between these two methods without a use case. This is a question about methodology. $\endgroup$ – mme Oct 21 '17 at 15:43
  • $\begingroup$ Maybe I tried to say too much in too little space, sorry for that. What I want to draw attention at is that the two approaches you mention are not distinct. You should consider focusing more on the comparison between a projection pursuit approach (what you call RPCA) and a robust covariance estimation approach (what you call Mahalanobis distances). Robust covariance estimation in itself is a perfectly valid methodology for RPCA implementations (eg. google "PCA M-Estimation"). Not too mention the presence of weighted PCA approaches that you somehow do not mention in the context of RPCA. $\endgroup$ – usεr11852 Oct 21 '17 at 16:25
  • $\begingroup$ No need for apology :) The two methods are very much distinct, particularly on small datasets. One of the ways they are different is mentioned at the end of my question. While (robust) PCA can be seen as a projection problem, it can also be interpreted as a covariance estimation problem, so there is perhaps less of a distinction in the parameter estimation method than in the application and performance. $\endgroup$ – mme Oct 21 '17 at 16:33
  • $\begingroup$ @MustafaSEisa/ Nice question! I think it can be answered on methodogical grounds: in fact it is one of my pete peeves. I will attempt a tentative answer asap. In the mean time; I think a fruitfull way to approach it in more general terms, is to look at the consequences of using models with nested but unequal group of invariance. As I try to do here in a slightly different context. $\endgroup$ – user603 Oct 24 '17 at 11:55

This paper compares some methods in this area. They refer to the Robust PCA approach you linked to as "PCP" (principal components pursuit) and the family of methods you linked to for robust covariance estimation as M-estimators.

They argue that

PCP is designed for uniformly corrupted coordinates of data, instead of corrupted data points (i.e., outliers), therefore, the comparison with PCP is somewhat unfair for this kind of data

and show that PCP (aka robust PCA) can fail for outlier detection in some cases.

They also talk about three kinds of "enemies of subspace recovery," i.e. different kinds of outliers, and which kinds of methods might do well for dealing with each one. Comparing your own outliers with the three kinds of "enemies" discussed here might help you pick an approach.

  • $\begingroup$ Thanks for this David, I will take a look at the paper. However, there is a version of robust PCA which imposes a rotationally-invariant penalty on the datum (rows of the data matrix) instead of a penalty on coordinates (such as in the Candes case). Thoughts? $\endgroup$ – mme Oct 26 '17 at 17:27
  • $\begingroup$ I'm not sure I understand your question. Are you asking me to compare the two approaches you discussed in your question with a different robust PCA approach? $\endgroup$ – David J. Harris Oct 26 '17 at 18:09
  • $\begingroup$ In your answer, you distinguish between the two methods by pointing out that the $\ell_1$ penalty in robust PCA is not rotationally-invariant and so is better suited to corruptions in the canonical basis. I’m just asking if you’ve considered or thought about the case in which a sum of (Euclidean) row norms is used in place of the $\ell_1$ coordinate penalties. $\endgroup$ – mme Oct 26 '17 at 18:21
  • $\begingroup$ If your answer is, “No” that’s totally fine I’m just wondering. $\endgroup$ – mme Oct 26 '17 at 18:22
  • $\begingroup$ Oh, I see. Would that be a special case of Mahalanobis distance? $\endgroup$ – David J. Harris Oct 26 '17 at 19:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.