19
$\begingroup$

Robust PCA (as developed by Candes et al 2009 or better yet Netrepalli et al 2014) is a popular method for multivariate outlier detection, but Mahalanobis distance can also be used for outlier detection given a robust, regularized estimate of the covariance matrix. I'm curious about the (dis)advantages of using one method over the other.

My intuition tells me that the greatest distinction between the two is the following: When the data set is "small" (in a statistical sense), robust PCA will give a lower-rank covariance while robust covariance matrix estimation will instead give a full-rank covariance due to the Ledoit-Wolf regularization. How does this in turn affect outlier detection?

Improve this question
$\endgroup$
6
  • $\begingroup$ Interesting question but I cannot see how an answer can be motivated without a specific use-case. Do you have "grossly corrupted observations"? Do you have generally noisy data? A number of RPCA implementations essentially are robust covariance estimation techniques (see Jolliffe's Princ. Component Analysis, Ed. 2nd Ch. 10) where the PCs are estimated from the regularised estimate of the covariance. Thus, the distinctions from the two approach you mention is far from clear-cut. In general, automatic outlier detection is successful within the context of a particular application. $\endgroup$
    – usεr11852
    Commented Oct 21, 2017 at 15:19
  • 1
    $\begingroup$ The “noisy data” problem isn’t outlier detection. I think the outlier detection problem is restrictive enough on its own to allow for a general comparison between these two methods without a use case. This is a question about methodology. $\endgroup$
    – Mustafa Eisa
    Commented Oct 21, 2017 at 15:43
  • $\begingroup$ Maybe I tried to say too much in too little space, sorry for that. What I want to draw attention at is that the two approaches you mention are not distinct. You should consider focusing more on the comparison between a projection pursuit approach (what you call RPCA) and a robust covariance estimation approach (what you call Mahalanobis distances). Robust covariance estimation in itself is a perfectly valid methodology for RPCA implementations (eg. google "PCA M-Estimation"). Not too mention the presence of weighted PCA approaches that you somehow do not mention in the context of RPCA. $\endgroup$
    – usεr11852
    Commented Oct 21, 2017 at 16:25
  • $\begingroup$ No need for apology :) The two methods are very much distinct, particularly on small datasets. One of the ways they are different is mentioned at the end of my question. While (robust) PCA can be seen as a projection problem, it can also be interpreted as a covariance estimation problem, so there is perhaps less of a distinction in the parameter estimation method than in the application and performance. $\endgroup$
    – Mustafa Eisa
    Commented Oct 21, 2017 at 16:33
  • $\begingroup$ @MustafaSEisa/ Nice question! I think it can be answered on methodogical grounds: in fact it is one of my pete peeves. I will attempt a tentative answer asap. In the mean time; I think a fruitfull way to approach it in more general terms, is to look at the consequences of using models with nested but unequal group of invariance. As I try to do here in a slightly different context. $\endgroup$
    – user603
    Commented Oct 24, 2017 at 11:55

1 Answer 1

Reset to default
8
+100
$\begingroup$

This paper compares some methods in this area. They refer to the Robust PCA approach you linked to as "PCP" (principal components pursuit) and the family of methods you linked to for robust covariance estimation as M-estimators.

They argue that

PCP is designed for uniformly corrupted coordinates of data, instead of corrupted data points (i.e., outliers), therefore, the comparison with PCP is somewhat unfair for this kind of data

and show that PCP (aka robust PCA) can fail for outlier detection in some cases.

They also talk about three kinds of "enemies of subspace recovery," i.e. different kinds of outliers, and which kinds of methods might do well for dealing with each one. Comparing your own outliers with the three kinds of "enemies" discussed here might help you pick an approach.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Thanks for this David, I will take a look at the paper. However, there is a version of robust PCA which imposes a rotationally-invariant penalty on the datum (rows of the data matrix) instead of a penalty on coordinates (such as in the Candes case). Thoughts? $\endgroup$
    – Mustafa Eisa
    Commented Oct 26, 2017 at 17:27
  • $\begingroup$ I'm not sure I understand your question. Are you asking me to compare the two approaches you discussed in your question with a different robust PCA approach? $\endgroup$
    – David J. Harris
    Commented Oct 26, 2017 at 18:09
  • $\begingroup$ In your answer, you distinguish between the two methods by pointing out that the $\ell_1$ penalty in robust PCA is not rotationally-invariant and so is better suited to corruptions in the canonical basis. I’m just asking if you’ve considered or thought about the case in which a sum of (Euclidean) row norms is used in place of the $\ell_1$ coordinate penalties. $\endgroup$
    – Mustafa Eisa
    Commented Oct 26, 2017 at 18:21
  • $\begingroup$ If your answer is, “No” that’s totally fine I’m just wondering. $\endgroup$
    – Mustafa Eisa
    Commented Oct 26, 2017 at 18:22
  • $\begingroup$ Oh, I see. Would that be a special case of Mahalanobis distance? $\endgroup$
    – David J. Harris
    Commented Oct 26, 2017 at 19:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.