2
$\begingroup$

Our model processes millions of multivariate observations; manual outlier detection is impractical. I am looking for a method of automatic outlier detection.

I have been trying to use R package mvoutliers, especially function pcout, and get the error

More than 50% equal values in one or more variables!

The problem is that our data is quite sparse and many variables include More than 50% equal values.

  1. Is there any way around (some data preprocessing) that would still allow me to use pcout?
  2. Is there another recommended R package/function/method for automatic outlier detection?
Improve this question
$\endgroup$
8
  • 5
    $\begingroup$ What do you plan to do with the data after outlier detection? If your plan is to remove outliers, maybe there is a robust method you could use intead. $\endgroup$
    – Roland
    Commented Mar 14, 2013 at 10:35
  • $\begingroup$ Have you tried lofactor() function from DMwR package? $\endgroup$
    – sitems
    Commented Mar 14, 2013 at 10:44
  • 1
    $\begingroup$ I am planning, among others, to perform regression analysis; we use OLS regression as of business constraints, robust regression is out of question here; but we can use robust outlier detection $\endgroup$
    – Herbert
    Commented Mar 14, 2013 at 10:54
  • $\begingroup$ Can you then do outlier detection post-analysis? That is, look for high leverage points, etc. in the model you fit? Of course, you can also do one-dimensional outlier detection on each of your variables before the regression. $\endgroup$
    – Peter Flom
    Commented Mar 14, 2013 at 11:01
  • $\begingroup$ @PeterFlom - this is impractical because the regression coefficients would be skewed. Univariate detection is not so effective in our dataset. $\endgroup$
    – Herbert
    Commented Mar 14, 2013 at 12:15

2 Answers 2

Reset to default
8
$\begingroup$

In my experience, the term outliers doesn't make sense without the context of the application. That is, if you want to exclude data points from your data set, you should be able to give reasons why this or that data point is removed. These reasons may suggest appropriate filtering rules.

Therefore I think that something like the "recommended R package/function/method for automatic outlier detection" cannot exist in general, it can exist only for particular types of data/applications.

Improve this answer
$\endgroup$
5
$\begingroup$

An outlier is only an outlier with respect to some model. What's hugely discrepant under one set of assumptions is just an ordinary point under another, something to be expected in the ordinary course of events.

If your data are often of a particular form, it might help to think about what you mean by 'outlier' for that kind of distributional situation, and tailor your automated procedure to deal with the situation you're in, rather than try to squeeze it into something it's not suited to.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.