3
$\begingroup$

I'm trying to learn R for ML purposes, and right now I'm building classifier for my data (10 dimensions, ~400 elements, 2 classes), which have some outliers in it, and a lot of missing values.

I'm using multi-imputation of missing values from Amelia package and e1071 SVM. My results are quite good: 80% quality on cross-validation.

Is there any best practices or advices for building classifier on such a bad data? Maybe I should somehow filter outliers first? Maybe I should consider some other method for imputing missing values?

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
4
$\begingroup$

If you feel like you are getting good results, what is the problem? Are you not identifying enough true positives? Are you identifying too many false positives? If it's one or the other, choose a more liberal or conservative threshold for predicting the classes. If it's both, there's probably not all that much you can do without getting more data or creating derived features. If you are using a SVM, then it would be natural to try using various kernels, since using a kernel is implicitly creating derived features.

As far as the modifications you suggest: removing outliers should not help you since one of the properties of SVM is that it effectively ignores all the data that is far from the decision boundary (assuming that the two classes are linearly separable in your feature space. If not, then you do need to worry about outliers).

Improve this answer
$\endgroup$
4
  • 3
    $\begingroup$ "since one of the properties of SVM is that it effectively ignores all the data that is far from the decision boundary". This statement is incorrect. SVM is actually very sensitive to outliers. There is still ongoing research on how to supress the influence of outliers like this one for example. $\endgroup$
    – TenaliRaman
    Commented May 13, 2012 at 16:18
  • $\begingroup$ I should have specified that SVM's resistance to outliers only hold if the classes are separable. I'll make the change. $\endgroup$
    – fgregg
    Commented May 13, 2012 at 17:51
  • $\begingroup$ fgregg: you mean linearly separable. $\endgroup$
    – user603
    Commented May 13, 2012 at 19:30
  • $\begingroup$ I mean linearly separable in some feature space. $\endgroup$
    – fgregg
    Commented May 13, 2012 at 19:33
3
$\begingroup$

You could start by removing the outliers from your design matrix (for this step you can only use whichever of those 10 variables that are continuous). You can do this rather easily using the $\verb+CovMcd()+$ function in the $\verb+rrcov()+$ library. $\verb+CovMcd()+$ will basically give you the index of the 50 percent most concentrated observations. Use only those to build your SVM classification model (but this time including all the variables in your model, even the discrete ones). Compare the performances of this model with that of the one you have now. Let us know if it improves the fit on the test set. It'll give good information on what type of outliers you have.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.