If I have some data, say 1000 labelled examples (A/B, 500 of each), from which I define an independent test set of 100 labelled examples (A/B, 50 of each), then
can I legitimately have a 'cursory look at the different distributions of features between type A and type B examples' using all 1000 examples,
without biasing the estimate of generalisation error I get when I train a predictive model on 900 examples and test exactly once on the test set?
My gut feeling is that it is not a problem because the independent test set is used as a final check of overfitting, and looking at the differences between type A's and type B's for some features will not put me in a position, either consciously or sub-consciously, to overfit to the data. (In contrast, tweaking model hyperparameters based on all of the data would allow overfitting).
The only situation I believe such data exploration could lead to overfitting is if I subconsciously changed the feature selection process to reward features that I had previously observed to have a slight difference in distribution between type A and type B which arose simply due to sampling and does not reflect a true difference between type A and type B...
What do you think??