TL;DR: I'm reviewing a computer vision + machine learning module that someone else wrote, and I've discovered that she is manually cleaning up training data. Is that ever a good idea?

The Details

The module's pipeline is as follows:

  1. Get input image
  2. Get binary mask on input image
  3. Extract features on binary image
  4. Classify image based on the features using artificial neural nets

So far this is probably ok (I am going to suggest using the unthresholded images for feature extraction, but that is another matter).

This is the testing pipeline:

  1. Get input images
  2. Get binary mask on each input image
  3. Manually remove noise and other mask errors from the training images
  4. Train a classifier using the cleaned up masks

As far as I can tell, this is a terrible idea. It means that the training data will be different from the test data, which can only lead to a decrease in accuracy. When I brought this up with the module's creator, however, I met a lot of resistance - she was convinced this let the nets "identify the important features and ignore the noise".

The question:

When, if ever, would manually modifying training data be a good idea? Why or why not? Ideally, please include links to papers, book chapters, or other relevant materials in your answer. Thanks!

Other notes:

  • testing and training sets are sufficiently large (~10000 samples for each class)

My option is that manual cleaning is a perfectly acceptable part of a machine learning pipeline, so long as it is treated the same as any other pre-processing step (e.g. centering, scaling, PCA, imputation) and tested in your cross-validation.

In other words, it's ok for the human to clean test data for which they do not know the labels and then provide that data to the machine for prediction. In this case, your test error is of the human + machine classification system.

Of course, in reality, humans tend to peek at the answers when they work with the data.

  • $\begingroup$ My major concern is that humans can clean training data, but they cannot do the same at runtime (where you need the system to quickly produce a result for a given input). Because you cannot apply this cleaning process to data at runtime, I think it actually is unlike PCA, scaling, etc. $\endgroup$ – StaringFrog Jan 8 '15 at 16:30
  • 1
    $\begingroup$ @StaringFrog that's a valid point. One test of this would be to try the models on some un-cleaned test data and assess their accuracy. You could then compare the models trained on "cleaned" data vs the models trained on "raw" data. If the "raw" models performed better, you'd have a strong quantitative data point with which to make your argument. $\endgroup$ – Zach Jan 8 '15 at 17:51
  • $\begingroup$ Yeah that's what I was thinking. It would be great to find a paper that did a large-scale version of that test, but I haven't discovered any thus far. $\endgroup$ – StaringFrog Jan 8 '15 at 18:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.