Is this an example of data snooping, or otherwise some other kind of no-no?

Question

We just designed a text classification system, and now it's time for me to write up some of the results we got. We observed that our classifier tends to perform better if we exclude documents below a certain length ($L$ characters, say) so it has enough data to learn from and make judgments on, and I'm not sure how we should present this. Doing such a thing feels strange to me, even though the $L$ we chose is pretty reasonable, IMO, and eliminates a small number of documents (less than a few percent of the corpus), which were as short as they were because of data quality errors beyond our control. The documents we excluded were not at all reflective of the sorts of examples we'd intended to work with. To me, what we did feels a bit much like data snooping, or something near it.

The process we went through was roughly this: 0) a bunch of feature engineering; 1) select classifier hyperparameters via cross-validated grid search; 2) then determine the best value of $L$ by cross-validation (with different folds), while keeping the optimal hyperparameters from 1) fixed.

Even if it's not strictly data snooping, I'd appreciate any insights on whether there are any fallacies in what we did here and if so, what we could do to remedy those. Should we have selected the best $L$ before selecting the best set of hyperparameters, for example? Thanks!

Answer 1

You should have your training data reflect the actual conditions on which your classification system will be used. If the short ones do not work because of data quality errors then this suggests the problem is with those texts and not your model. There is always a danger that removing some cases that won't work will over optimize your model and thus lead to worse performance on new data but if the training data it doesn't work on does not reflect the data it will be used on then this is not as large a concern.

Do you have a way to remove the trouble data points before calibrating the model? You say there are data quality issues with some of them, can you remove the data points with data quality errors?

If it isn't possible to remove the data points for data quality error reasons, then the proper procedure would be to acquire a new set of data to evaluate the performance of your classifier.

I hope that helps.