0
$\begingroup$

Suppose we are working on some sort of classification problem, and we have subdivided our data into a training set and a test set (or validation set, or etc.).

We wish to prepare the data in the following sense; say we have a name field, and we wish to add a new feature indicating how many of the people in the dataset have the same last name.

This can be easily done for the training set or the test set independently or together, but there are a few issues that I can see.

  1. If we compute this number for the training set and test set independently, we run the risk of undercounting - in this example it is quite likely but for other edge counting problems it may be less likely.
  2. If we compute this number for the combined dataset, it seems like "cheating" -- mixing the train and test data in order to classify.

So my question I suppose is this. Is it "legal" to do data preparation across the combined data set, or does this pollute the result?

If it is legal for this unambiguous data preparation, how far does this go? Can I do imputation across the combined data set (say both sets have some missing data), or does this create problems?

I am mostly interested in the specific case of edge-counting problems, but the general question is also useful. I have not seen this addressed in courses/books I've seen.

$\endgroup$

1 Answer 1

1
$\begingroup$

If you count the names before splitting the data, the training and test sets won't be independent anymore. This can make the test set error underestimate the true generalization error. No bueno. Another way to think of it is that counting the names is a form of learning from the data, and you shouldn't be learning on the test set.

The thing to do instead is to count the names in the training set only. Instead of using the raw number, it would make sense to normalize the counts by the number of points in the training set, to get a score for how common each name is. When you need to know how common each name in the test set is, look up the value from the training set.

One reason to use normalized scores instead of counts is for cross-validation, where the size of your training set may differ across folds; the feature values shouldn't depend on this. It's also a bit more meaningful. E.g. say you want to predict a value for some new person in the future. It makes more sense to talk about the commonness of their name then to say that 40 people had the same name in the test set, back in the day.

Your situation with name counts is similar to the case with continuous values, where one often wants to normalize the features. In that case, the thing to do is estimate the mean and standard deviation on the training set, then use them to normalize both sets.

Edit in response to comment:

It seems like 'number of co-travelers' is essentially a feature of the ticket, even though you have to examine the ticket log to determine it. Say you're running the classifier on a new set of data. You'd look up number of co-travelers for each passenger from the ticket logs for that flight, not try to estimate it from the training set. So, it's not the same as trying to estimate how common a name is.

Actually, now that the nature of your data slightly more clear, it raises a certain distinction. My answer above assumed that the goal of counting names was to estimate how common a name is in general. However, that has a different meaning than counting the number of people with the same name that are passengers on a particular flight (e.g. a family traveling together). If your intended meaning is this second case, then treating it like the number co-travelers makes more sense (use the ticket logs).

$\endgroup$
4
  • $\begingroup$ This makes sense, but isn't always appropriate. For a specific example: another bit of data preparation I wanted to do was to check how many passengers share a ticket with the specified passenger; this isn't given as a feature, but derived from each passenger's ticket number. Typically these are 1-4 people, so it's not interesting to say what percentage of the data you're associated with (essentially zero), but the absolute number is interesting (along, with a group, etc.). How should I approach this? $\endgroup$ May 27, 2016 at 18:47
  • $\begingroup$ @RichardRast Edited the answer to address your question $\endgroup$
    – user20160
    May 29, 2016 at 5:46
  • $\begingroup$ So, using the training and test set (which together comprise the entire "population") would be a reasonable action in this case, since it could have been a feature we started with anyway? $\endgroup$ May 29, 2016 at 13:26
  • $\begingroup$ @RichardRast Yes, that sounds reasonable $\endgroup$
    – user20160
    May 30, 2016 at 3:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.