TL;DR: I'm reviewing a computer vision + machine learning module that someone else wrote, and I've discovered that she is manually cleaning up training data. Is that ever a good idea?
The Details
The module's pipeline is as follows:
- Get input image
- Get binary mask on input image
- Extract features on binary image
- Classify image based on the features using artificial neural nets
So far this is probably ok (I am going to suggest using the unthresholded images for feature extraction, but that is another matter).
This is the testing pipeline:
- Get input images
- Get binary mask on each input image
- Manually remove noise and other mask errors from the training images
- Train a classifier using the cleaned up masks
As far as I can tell, this is a terrible idea. It means that the training data will be different from the test data, which can only lead to a decrease in accuracy. When I brought this up with the module's creator, however, I met a lot of resistance - she was convinced this let the nets "identify the important features and ignore the noise".
The question:
When, if ever, would manually modifying training data be a good idea? Why or why not? Ideally, please include links to papers, book chapters, or other relevant materials in your answer. Thanks!
Other notes:
- testing and training sets are sufficiently large (~10000 samples for each class)