4
$\begingroup$

I am currently working on a classifier for the qualitative spectral analysis of alloys.

One of the problems that I faced is preparation of samples for the classifier training. Samples have to me machined into chips that can physically fit into the spectrometer's sample chamber.

During cross-validation I have noticed a few very peculiar miss-classifications that simply should not happen (the elemental composition is too different), esp. taking into consideration the overall performance of the classifier. Upon careful examination I found something that looks like contaminated (or possible mislabeled, although this is very unlikely) sample. Anomaly detector confirms that and makes it very simple to find such samples and remove them.

My question is, what would considered the a better practice in the machine learning:

  • Filter entire data set so neither training nor cross-validation data has any outliers.
  • Filter only the training set and leave outliers in the cross-validation set.

Real world samples for which this classifier is prepared are unlikely to have this sort of contamination.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

You pretty much answered your question with that last sentence. Assuming you are building a model to be used in a real world application, you want its training and evaluation setup to be as close as possible to said application.

If you have no outliers in reality, then get rid of them in your own data set too. Having no outliers in a real world application sounds too good to be true, though, so be sure about that or you will be way too optimistic about the model's performance.

Improve this answer
$\endgroup$
4
  • $\begingroup$ So a prudent measure would be to remove outliers from the training set and keep them in the cross validation set to have a better measure of the classifier performance? $\endgroup$
    – udushu
    Dec 28, 2014 at 3:22
  • $\begingroup$ Also, I would never say that the real world would not have outliers, it just going to be a different kind of outliers since the metal chips would be a wear debris and not machined chips. $\endgroup$
    – udushu
    Dec 28, 2014 at 3:23
  • 1
    $\begingroup$ @udushu if those outliers are not present in the real world data, then you shouldn't have them in the test set. You always want to maximize the similarity of how you evaluate the model and how it is actually going to be used. $\endgroup$
    – Marc Claesen
    Dec 28, 2014 at 9:07
  • 1
    $\begingroup$ ... however, it may be worth while to put the outliers into a separate set (they haven't been used for training so you can have them as separate outlier test set) so you can nevertheless check the performance for them. I agree with Marc that you do not want to mix them into the "normal" test results. I sometimes do this to check the robustness of Raman spectroscopic classifiers against spikes. $\endgroup$
    – cbeleites unhappy with SX
    Jan 5, 2015 at 15:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.