6
$\begingroup$

In many beginner ML lectures / tutorials, it's advised to remove those features that uniquely identify the example. For example, if predicting user behavior, a numeric user_id column should be removed. The stated reason is that a powerful classifier would use that column to fit perfectly on the training set, ignoring all the other columns, resulting in a useless model (of course the CV will show that).

I don't understand why a classifier might focus so much on user_id at the expense of the other columns. Any continuous column often has the power to uniquely identify the example. For example, if we use user_distance_from_nearest_store with high enough precision, its likely no two users have exactly the same distance, so the distance may uniquely identify users.

So I can't see why including user_id would cause any more problems that including any other completely unimportant continuous feature. Am I missing something?

(Of course, I understand user_id shouldn't be included, since it can't help the model and can hurt it due to noise. I'm just trying to understand why it might be so influential if it's included by accident.)

Edit: to clarify, by "numeric" column, I meant not categorical. Also, I was talking about a classifier, but the intuition will carry over to a regression.

Improve this question
$\endgroup$
0

3 Answers 3

Reset to default
5
$\begingroup$

The predictive value of an ID field will vary considerable from dataset to dataset, so in some cases it's probably ok to leave in, and in others not. One case where it could have high predictive value (and should be taken out) is where you're trying to predict the age of something, and the ID is assigned according to age (e.g., new accounts or users' IDs are incremented). Other than cases like this, a continuous feature wouldn't provide much value.

An overfit tree, which makes so many splits that each terminal node has only one observation, is just as likely to overfit on an ID column as any other column. Playing devil's advocate, one might even argue for leaving it in so as to have some sort of benchmark in your dataset for identifying variables that have no value.

Improve this answer
$\endgroup$
2
  • $\begingroup$ +1. But actually if the ID is assigned in some meaningful order, why not use it? It would have a real predictive power, even on new data. As to leaving it in as a benchmark, I think it might be worth it in the initial evaluation stage, and then you can just remove it before producing the final model. $\endgroup$
    – max
    Commented Jul 19, 2016 at 21:34
  • $\begingroup$ Yes, I agree with both points. An ID value is ok to use as long as you have it at the time that you need to make predictions in production. And yes, if it has minimal to no value I would remove it and any other predictors will less value $\endgroup$
    – Ryan Zotti
    Commented Jul 20, 2016 at 1:37
7
$\begingroup$

"Certainly, any continuous column (with enough resolution) has the power to uniquely identify the example." - that's not true for a predictor that is treated as continuous. For instance, if you generate Y and X at random using uniform distribution, then all of the values of X will most likely be distinct. However, when you regress Y on X using simple linear regression, the fit (in terms of $R^2$) will not be perfect. That's because there is a restriction on relationship between Y and X in the form $Y = x'\beta + \epsilon$. There are only two parameters in the model.

If you treat X as a categorical predictor with N levels where N is the sample size, then there is no such restriction: for each level of X, you are allowed estimate a distinct parameter that describes the average response at that level of X. That is, you end up with N observations and N parameters. For each such parameter, its estimate will be equal to the observed value of Y, so you will end up with a perfect fit.

One implication is that if your subject ID is formatted as a number and the ML package will see it as a quantitative predictor, then you won't have a problem with it, apart from including a meaningless predictor in your model.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ +1. I would suggest expanding on the second paragraph, though. In many classification algorithms, categorical variables are mapped to dummy variables: one variable per category that's either 1 or 0. In the case of a unique user_id, each individual will have one dummy variable that is 1 only for it. $\endgroup$
    – Wayne
    Commented Jul 19, 2016 at 17:42
  • $\begingroup$ @Wayne if the identifier is categorical, then you get as many features as the size of the training set. That alone will destroy any chance of successful learning. $\endgroup$
    – max
    Commented Jul 19, 2016 at 20:43
  • $\begingroup$ Your first paragraph is a bit unclear. I was saying that the user_id would uniquely identify each example. Only if the model is sufficiently (stupidly) complex, would it be able to perfectly fit on the training set. My question is about whether user_id is that different from any other useless numeric feature, and I think you answered it in your last paragraph (i.e., it's not really different). $\endgroup$
    – max
    Commented Jul 19, 2016 at 21:38
0
$\begingroup$

Another way to look at it is from the POV of a neural network. It is much easier to memorize which classes belong to which target, than to learn meaningful features.

Including the identifier also encourages co-adaptation, which overfits on the training set.

Improve this answer
$\endgroup$
1
  • $\begingroup$ But, how is it different from any other continuous feature (one that isn't an identifier)? $\endgroup$
    – max
    Commented Jul 19, 2016 at 20:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.