3
$\begingroup$

I am working with a PLS model (library pls) in R, where I am developing calibration models for NIRS data. I have been using other commercial software before that allowed me to detect outliers based on Mahalanobis distances.

However, although R has proven to be superior in all senses to any software I have used before, I cannot find a good way of detecting potential outliers in a new dataset to be predicted. my model follows the structure:

model<-plsr(as.matrix(dataset)~constituent, CV=TRUE, ncomp=15)

And then I predict:

predictions<-predict(model, newdata=NEWDATA,ncomp=15)

But how can I assess the fitness of the predictions?

When I try to apply the "mahalanobis" command from {stats}, I do the following:

mahalanobis.data<-mahalanobis(new-dataset,colMeans(dataset),var(dataset))

And either R crashes, or gives error messages. I have tried also with some of the rrcovHD and mvoutlier commands, but I suddenly get half of my dataset detected as outlier.

Since I have n< p, conventional multivariate outlier detection does not apply here, as far as I am concerned.

Is there any package I have overlooked, or something I am not applying properly?

Thanks in advance

$\endgroup$
  • $\begingroup$ Yes, I mixed the > and < signs, sorry. Now I fixed it in the question also. The message I get is: Error in covMcd(X) : n <= p -- you can't be serious! (which, of course, is a logical message to get) $\endgroup$ – User1234 Nov 4 '14 at 16:42
  • $\begingroup$ and what is $n$ and $p$? Also, are all your variable continuous? Are all your cases complete or do you have missing cells in some rows? $\endgroup$ – user603 Nov 4 '14 at 18:30
  • 1
    $\begingroup$ My variables are 2100 columns with absorbance values from a NIRS spectrometer, with no missing cells and no zero values. As for n= number of observations and p= parameters (i.e. n=rows, p=columns) $\endgroup$ – User1234 Nov 4 '14 at 19:22
  • $\begingroup$ Typically, people use robust PCA method for this type of data. There are a couple of approaches in the rrcov library. Have you tried those? $\endgroup$ – user603 Nov 4 '14 at 19:58
  • 1
    $\begingroup$ Yes, I have used both rrcov and rrcovHD libraries: using PCAHubert() from the first, or OutlierPCOut() from the second. Although I get outlier estimation from the training set, I cannot use that model to estimate outliers from the test set. And I did not find the solutions in the documentation either. That is why I suspect I am not using the right approach... $\endgroup$ – User1234 Nov 5 '14 at 8:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.