Suppose we are working on some sort of classification problem, and we have subdivided our data into a training set and a test set (or validation set, or etc.).
We wish to prepare the data in the following sense; say we have a name field, and we wish to add a new feature indicating how many of the people in the dataset have the same last name.
This can be easily done for the training set or the test set independently or together, but there are a few issues that I can see.
- If we compute this number for the training set and test set independently, we run the risk of undercounting - in this example it is quite likely but for other edge counting problems it may be less likely.
- If we compute this number for the combined dataset, it seems like "cheating" -- mixing the train and test data in order to classify.
So my question I suppose is this. Is it "legal" to do data preparation across the combined data set, or does this pollute the result?
If it is legal for this unambiguous data preparation, how far does this go? Can I do imputation across the combined data set (say both sets have some missing data), or does this create problems?
I am mostly interested in the specific case of edge-counting problems, but the general question is also useful. I have not seen this addressed in courses/books I've seen.