4
$\begingroup$

If there are multiple outliers in the data set, the Mahalanobis distance suffers from masking and swamping effects. In order to rectify this problem, robust estimation of location and scale, such as Minimum Covariance Determinant or Minimum Volume Ellipsoid has been implemented. I thought that these estimators do not affected by masking and swamping.

Actually I have generated 60 observation from bivariate normal distribution plus 5 outliers, according to Tukey–Huber contamination model. I was expecting MCD to identify only 5, but surprisingly MCD identified 7, two observation more than the real number of outliers. What is responsible for that? Please correct me if I understand wrong. Thanks in advance.

$\endgroup$
  • $\begingroup$ covMcd in robustbase and uni.plot in mvoutlier package of R. $\endgroup$ – user2802663 Jan 16 '15 at 20:07
6
$\begingroup$

Yes, it is certainly possible for the (Fast)MCD to suffer from swamping and masking effects. That said, I think the question is best approached quantitatively, rather than in absolutes.

Given contaminated data, the objective of every robust estimator is to find a fit of the parameter of your model as close as possible to the one you would have had without the outliers. In order to achieve that, we have to trade fit quality in the presence of outliers against fit quality in their absence.

For FastMCD, the target model is typically elliptical, so the objective is to find a good fit, close to the MLE solution, of $(\mu,\varSigma)$ on clean data in a way that still insures an acceptable fit on contaminated data. The usual measure of distance between $(\hat{\mu},\hat{\varSigma})$ and $(\mu,\varSigma)$ for this problem is the bias measure $b(\hat{\varSigma},\varSigma)$, defined in [0].

Now to get to your original question, swamping and masking will increase $b(\hat{\varSigma},\varSigma)$. For example, the classical estimator of covariance does not suffer from swamping so in the absence of outliers, it gets low (good) values of $b(\hat{\varSigma},\varSigma)$. However, it takes only a handful of nasty outliers to cause it to have very large values of $b(\hat{\varSigma},\varSigma)$.

Here, (Fast-)MCD offers a less steep trade-off (slightly higher values of $b(\hat{\varSigma},\varSigma)$ when there are no outliers versus much lower values when there are some).


Edit:

We can illustrate this trade off by measuring the $b(\hat{\varSigma},\varSigma)$ of several shift and affine equivariant estimators of scatter in R on your settings.

Basically, I m going to generate bivariate datasets of size $n=60$, add 5 outliers to them and see how good various estimators are doing.

Since the estimators I will compare are shift and affine equivariant I can, wlog, set $\varSigma=I_2$ (the rank two identity matrix) and $\mu=0_2$ (a vector of 0 of length 2). Because of this, we also have that:

$$b(\hat{\varSigma},\varSigma)=\log\left(\frac{\lambda_1(\hat{\varSigma})}{\lambda_2(\hat{\varSigma})}\right)$$

($\lambda_p(A)$ denotes the p-eigenvalue of $A$).

I generate outliers as realization of $\mathcal{N}_2(d_2,I_2/100),d\in\mathbb{R}$. I use a small variance ($I_2/100$) here because it is known that concentrated outliers are the most difficult ones for affine equivariant estimators [1]. As for the value of $d$, I will take several values in a range $(0,20)$. The resulting R code:

library("ICSNP")
library("robustbase")
library("rrcov")

n<-60
p<-2
e<-5
m<-10

set.seed(123)
x<-matrix(rnorm(n*p),nc=2)
z<-matrix(rnorm(e*p,0,1/100),nc=2)
d<-seq(0,20,l=m)
results<-matrix(NA,m,6)
e<-vector("list",6)
#bias measure when varsigma=I_p
fx01<-function(ll,e){
    A<-eigen(e[[ll]])$values
    log(max(A)/min(A))
}

for(i in 1:m){#i<-1
    w<-sweep(z,2,rep(d[i],p),FUN="+")
    X<-rbind(x,w)
    e[[1]]<-cov(X)
    e[[2]]<-covMcd(X,nsamp="best")$cov
	e[[3]]<-symm.huber(X)
	e[[4]]<-HR.Mest(X)$scatter
    e[[5]]<-CovMest(X)@cov
    e[[6]]<-CovMMest(X)@cov
    results[i,]<-c(lapply(1:length(e),fx01,e=e),recursive=TRUE)
}
matplot(d,results,type="l",lwd=2,lty=1,col=1:6,ylab="bias",xlab="Distance of outliers")
legend("topleft",lty=1,lwd=2,col=1:6,legend=c("Classical","FMCD","Huber","HRM","CovMest","CovMMest"))

The resulting plot:

enter image description here

As you can see, the classical estimator of covariance, though it never suffers from swamping effect (by definition) and does best when the data contains no outliers, is in fact the worst choice when the data contains outliers. In that situation, the Huber M estimator of covariance is somewhere in the middle and all the other estimators do better. The best overall in this experiment seems to be FMCD, despite the fact that it suffers from swamping effect. The results would have been different had the rate of contamination or the number of dimensions been higher so it is always worth doing these numerical experiments.

  • [0]: Yohai, V.J. and Maronna, R.A. (1990). The Maximum Bias of Robust Covariances. Communications in Statistics--Theory and Methods, 19, 3925--2933.
  • [1] Rocke D. M. and Woodruff D. L. (1996). Identification of Outliers in Multivariate Data. Journal of the American Statistical Association, 91, 1047--1061.
$\endgroup$
  • $\begingroup$ Thanks for your answer. Is there any better substitute for FAST-MCD, which has the ability to identify exactly the number of outliers in the data set? $\endgroup$ – user2802663 Jan 16 '15 at 20:50
  • 2
    $\begingroup$ I don't think it is possible to guarantee this in general, unless the outliers are really really far from the good data. After all, the usual definition of outliers is 'observations that are too far from the pattern of the majority of the data'. Clearly, this leaves some room for interpretations (is 2.5 too large a value for a standard gaussian? What about 3.5?). In the example above, I don't see any swamping for FMCD for d>2.5 for example. $\endgroup$ – user603 Jan 16 '15 at 22:09
  • $\begingroup$ Thanks again, your explanation is very helpful and informative, like sitting in the class :) $\endgroup$ – user2802663 Jan 18 '15 at 4:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.