3
$\begingroup$

The data has about 40 features and 500,000 instances. And the data is sparse. I wish to fit a svm model with the data. To fit svm, I need to first scale the data. However, if the data contains many outliers, scaling is likely to not work very well. So the problem is how can I find outliers in the data?

$\endgroup$
13
  • 1
    $\begingroup$ Can you tell us what you have tried already? Can you provide a scree-plot of your data? Standard PCA algorithms are fine for your case. Your data is barely above 152 MBytes (even if you treat them as dense) to begin with. To that aspect try a sparse PCA algorithm too. $\endgroup$
    – usεr11852
    Commented Nov 24, 2015 at 6:41
  • $\begingroup$ If you can formalize "not work well", then you can use that definition as a metric for "outlier". $\endgroup$
    – Neil G
    Commented Nov 24, 2015 at 6:47
  • 2
    $\begingroup$ .... Just use OGK (Orthogonalized Gnanadesikan-Kettenring) and/or MCD (minimum covariance determinant ) and pick the most coherent. And having 40 dimensions is just "multivariate". $\endgroup$
    – usεr11852
    Commented Nov 24, 2015 at 8:07
  • 1
    $\begingroup$ Many threads here on outliers: did you search before you posted? It's hard to summarize, but the negative principle I would mention first is that there is no simple universal definition of outliers. Outliers are defined only relative to some model, even if it's tacit or informal. $\endgroup$
    – Nick Cox
    Commented Nov 24, 2015 at 10:48
  • 1
    $\begingroup$ If you just want a resistant method to scale the data, then why not use one? The IQR will work just fine. $\endgroup$
    – whuber
    Commented Nov 24, 2015 at 14:45

2 Answers 2

1
$\begingroup$

You can compute the Mahalanobis distance for each observation. If $S$ is the estimated covariance matrix of the data and $\overline{x}$ is its mean (vector), for item ${\bf x_i}$ the distance is $D_i = ({\bf x_i}-\mu)^\prime S^{-1}({\bf x_i}-\mu)$. If your data is multivariate normal, these distances will have values that are chi-square distributed with 40 (= number of features) degrees of freedom. So compare the distances with the 95th percentile of the chi-square random variable.

$\endgroup$
1
  • 4
    $\begingroup$ In fairness Mahalanobis distance is susceptible to outliers so I am not sure if this metric should be your choice. $\endgroup$
    – usεr11852
    Commented Nov 24, 2015 at 8:04
1
$\begingroup$

Following up @AlaskaRon's answer (and comments): this paper suggests a robust analogue of the Mahalanobis distance called the Minimum Covariance Determinant:

The MCD estimators of location and scatter, denoted $\hat\mu_{MCD}$ and $\hat \Sigma_{MCD}$, correspond to the sample mean and covariance matrix of this most central sub-sample.

e.g. via the covMcd function in R's robustbase package ... so, compute the MCD and pick some sensible threshold ...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.