Should I ever manually modify training data?

Question

TL;DR: I'm reviewing a computer vision + machine learning module that someone else wrote, and I've discovered that she is manually cleaning up training data. Is that ever a good idea?

The Details

The module's pipeline is as follows:

1. Get input image
2. Get binary mask on input image
3. Extract features on binary image
4. Classify image based on the features using artificial neural nets

So far this is probably ok (I am going to suggest using the unthresholded images for feature extraction, but that is another matter).

This is the testing pipeline:

1. Get input images
2. Get binary mask on each input image
3. Manually remove noise and other mask errors from the training images
4. Train a classifier using the cleaned up masks

As far as I can tell, this is a terrible idea. It means that the training data will be different from the test data, which can only lead to a decrease in accuracy. When I brought this up with the module's creator, however, I met a lot of resistance - she was convinced this let the nets "identify the important features and ignore the noise".

The question:

When, if ever, would manually modifying training data be a good idea? Why or why not? Ideally, please include links to papers, book chapters, or other relevant materials in your answer. Thanks!

Other notes:

• testing and training sets are sufficiently large (~10000 samples for each class)

Answer 1

My option is that manual cleaning is a perfectly acceptable part of a machine learning pipeline, so long as it is treated the same as any other pre-processing step (e.g. centering, scaling, PCA, imputation) and tested in your cross-validation.

In other words, it's ok for the human to clean test data for which they do not know the labels and then provide that data to the machine for prediction. In this case, your test error is of the human + machine classification system.

Of course, in reality, humans tend to peek at the answers when they work with the data.