Can 'independent' test set be included in data exploration without biasing generalisation error estimate?

Question

If I have some data, say 1000 labelled examples (A/B, 500 of each), from which I define an independent test set of 100 labelled examples (A/B, 50 of each), then

can I legitimately have a 'cursory look at the different distributions of features between type A and type B examples' using all 1000 examples,

without biasing the estimate of generalisation error I get when I train a predictive model on 900 examples and test exactly once on the test set?

My gut feeling is that it is not a problem because the independent test set is used as a final check of overfitting, and looking at the differences between type A's and type B's for some features will not put me in a position, either consciously or sub-consciously, to overfit to the data. (In contrast, tweaking model hyperparameters based on all of the data would allow overfitting).

The only situation I believe such data exploration could lead to overfitting is if I subconsciously changed the feature selection process to reward features that I had previously observed to have a slight difference in distribution between type A and type B which arose simply due to sampling and does not reflect a true difference between type A and type B...

What do you think??

Thank you!

Answer 1

No you shouldn't.

Consider the following two scenarios:

1. The test set is much smaller than your train set, to the point that it would probably not change the distribution of your features even if that was the case. In this case, why bother including it in your initial analysis?

2. The test set is large enough that including it could possible change the distribution of your features, in which case, including it would deceive you in your analysis. You may decide on a transformation based on the distribution which you wouldn't have thought of had you not seen the test data.

There are other reasons against looking at your test data. For instance, you may see a correlation between features on your train data, which doesn't exist if you include the test data, and that may convince you upon choosing a family of models against another family of models.

Also, I'm not sure what you mean by looking at the distribution. If that entails estimating the mean and variance of the features for the purpose of preprocessing your data, that's definitely wrong.

Another point, is that you may even go back and change your experiment collecting the data in the middle of the experiment if you look at the data while collecting more data, which again invalidates your experiment.

I'm not sure which reason could potentially apply in your scenario, but as long as the numbers are 900 vs 100, just leave those 100 alone.