We just designed a text classification system, and now it's time for me to write up some of the results we got. We observed that our classifier tends to perform better if we exclude documents below a certain length ($L$ characters, say) so it has enough data to learn from and make judgments on, and I'm not sure how we should present this. Doing such a thing feels strange to me, even though the $L$ we chose is pretty reasonable, IMO, and eliminates a small number of documents (less than a few percent of the corpus), which were as short as they were because of data quality errors beyond our control. The documents we excluded were not at all reflective of the sorts of examples we'd intended to work with. To me, what we did feels a bit much like data snooping, or something near it.
The process we went through was roughly this: 0) a bunch of feature engineering; 1) select classifier hyperparameters via cross-validated grid search; 2) then determine the best value of $L$ by cross-validation (with different folds), while keeping the optimal hyperparameters from 1) fixed.
Even if it's not strictly data snooping, I'd appreciate any insights on whether there are any fallacies in what we did here and if so, what we could do to remedy those. Should we have selected the best $L$ before selecting the best set of hyperparameters, for example? Thanks!