1
$\begingroup$

I am searching some documents and examples related multivariate outlier detection with robust (minimum covariance estimation) mahalanobis distance. I have 6 variables and want to plot them to show outliers also. Do you have any sources?

Here are the codes, but I think something going wrong. Because I have over 2 million cases it has taken only n=500.

> CovMcd(new)

Call:
CovMcd(x = new)
-> Method:  Fast MCD(alpha=0.5 ==> h=1342195); nsamp = 500; (n,k)mini = (300,5) 

Robust Estimate of Location: 
logdgr  logtr       lph       lpm       lpr  
   4.391     2.956    -2.722    -4.802    -4.802  

Robust Estimate of Covariance: 
          logdgr  logtr  lph     lpm     lpr   
logdeger  1.0183    0.8981    0.6427  0.7112  0.7113
loghacim  0.8981    1.0173    0.9613  0.9539  0.9541
lph       0.6427    0.9613    1.6921  1.3770  1.3772
lpd       0.7112    0.9539    1.3770  1.3085  1.3087
lpr       0.7113    0.9541    1.3772  1.3087  1.3089
> summary(mcd)

Call:
CovMcd(x = data)

Robust Estimate of Location: 
[1]  3.5  2.0

Robust Estimate of Covariance: 
      [,1]  [,2]
[1,]  3.5   0.8 
[2,]  0.8   0.8 

Eigenvalues of covariance matrix: 
[1]  3.7192  0.5808

Robust Distances: 
[1]  2.0833  0.8333  2.0833  2.0833  0.8333  2.0833
> mest <- CovMest(new)
> show(mcd)

Call:
CovMcd(x = data)
-> Method:  Fast MCD(alpha=0.5 ==> h=4); nsamp = 500; (n,k)mini = (300,5) 

Robust Estimate of Location: 
[1]  3.5  2.0

Robust Estimate of Covariance: 
      [,1]  [,2]
[1,]  3.5   0.8 
[2,]  0.8   0.8 
$\endgroup$
  • $\begingroup$ Thanks, one more thing, how do I get number of total outliers in this?What is the R code? $\endgroup$ – user22463 Feb 12 '15 at 2:10
  • $\begingroup$ the function covMcd in robustbase both produce a vector of robust Mahalanobis distances (usually called statistical distances) wrt to the FMCD estimates of covariance and location. Try ?covMcd and look for mah as well as ?covPlot. The outliers are the observations for which mcd.wt is 0. $\endgroup$ – user603 Feb 12 '15 at 10:29
  • $\begingroup$ I performed the analsis but could not get the ourlier cases and could not compare classical mahalanobis score and mcd.. It seems mcd.wt did not work, or am I missing something? Yeah there is a manual regarding covMcd but never given even an example step by step. We make statistics difficult. $\endgroup$ – user22463 Feb 12 '15 at 18:35
  • $\begingroup$ maybe you should post a reproducible example? $\endgroup$ – user603 Feb 12 '15 at 18:42
  • $\begingroup$ This is not a minimal, reproducible example (if you try to run it on a clean environment you will get Error: could not find function "CovMcd") $\endgroup$ – user603 Feb 12 '15 at 19:00
1
$\begingroup$

What you are trying to do stems from the following basic idea:

  1. Compute robust distances (using Minimum Covariance Determinant, MCD estimate) of each observation $$rd(x_{i}),\ i = 1 \ldots n,\ x_i \in \mathbb{R}^p, $$ where $n$ and $p$ are the number of observations (rows) and variables (columns) respectively.
  2. Compare each $rd(x_{i})$ to $\sqrt{\chi^2_{p,.975}}$. Declare $x_i$ an outlier if $$rd(x_{i}) > \sqrt{\chi^2_{p,.975}}$$

  3. The robust distances are given by: $$rd(x_i) = \sqrt{(x_i - \mu_{mcd} )'S_{mcd}^{-1} (x_i - \mu_{mcd} )}$$ where $ \mu_{mcd}$ and $S_{mcd}$ are the robust MCD estimates of location (mean vector) and scatter (covariance matrix) respectively.

So therefore, what you need to do is:

A. Get the robust MCD estimate of location ($ \mu_{mcd}$) and scatter ($S_{mcd}$) for your data. You're on the way with the CovMCD(x) function in R. Note that this function implements the FastMCD algorithm by default which repeatedly takes subsamples of your data (each of size denoted by $h$) to make estimates. The number of subsamples to take by default in this function is 500 which explains the reason you're seeing 500 there. The function is not taking 500 observations out of your data. It is instead taking subsamples of size $n_{subsample} = h$ repeatedly to make estimates. 500 of these subsamples will be taken to make estimates. Check Peter J. Rousseeuw & Katrien Van Driessen (1999) A Fast Algorithm for the Minimum Covariance Determinant Estimator, Technometrics, 41:2, 212-223 for more details.

B. Get the robust distances for each row or obsersavtion using the formula in item 3 above.

C. Compare each distance $rd(x_i)$ to $\sqrt{\chi^2_{p,.975}}$ and declare $x_i$ an outlier if $rd(x_{i}) > \sqrt{\chi^2_{p,.975}}$.

And example R code is provided below:

require(rrcov)
data(hbk)
mcd <- rrcov::CovMcd(hbk[,1:3]) # use only first three columns  
# get mcd estimate of location
mean_mcd <- mcd@raw.center
# get mcd estimate scatter
cov_mcd <- mcd@raw.cov

# get inverse of scatter
cov_mcd_inv <- solve(cov_mcd)

# compute distances

# compute the robust distance
robust_dist <- apply(hbk[,1:3], 1, function(x){
  x <- (x - mean_mcd)
  dist <- sqrt((t(x)  %*% cov_mcd_inv %*% x))
  return(dist)
})

# set cutoff using chi square distribution
threshold <- sqrt(qchisq(p = 0.975, df = ncol(hbk[,1:3]))) # df = no of columns

# find outliers
outliers <-  which(robust_dist >= threshold) # gives the row numbers of outliers
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.