# Test features simply being a superset of train features

I am working on a two-class prediction task. A rather informative field in the dataset is user_id that has ~4m unique values which all occur in test set but it is a proper superset of user_ids available inside the training set (~2.5m). Now for my poly-2 regression based model it doesn’t make sense to have categorical feature values that it hasn’t seen before, so I plan to replace all user_ids in test set, which are missing in train set, with a single value (-999 if you may). But then the problem is that my model wouldn’t have seen any sample with -999 for its user_id. So what’s the best course of action here? I’m thinking about iterating over each train sample twice during the training, once with the original user_id and once with the special value -999; as if telling the model to rely on other features if user_id was missing. I know about imputation techniques but here I don't have any samples with missing values.

Edit: I should mention that user_id is one-hot encoded.

• Can you please explain a bit more about user id, is it akin to say roll number of a student in a class? – saha rudra Jan 9 '18 at 18:44
• @saharudra Yes, you can say that. Although my model is only considering them as nominal features identifying distinct users. – Nima Mohammadi Jan 9 '18 at 19:15
• This is a local competition and the problem is very similar to CTR prediction. The test set has two features (i.e. user_id and item_id) and while there is no overlap between items in two sets (item feature are given in another table) and there is no point in actually giving item_ids to the model, the train user_ids are a subset of test user_ids. Although the features are highly anonymized, the time users have signed up in the system can be probably inferred from this feature, but at the moment I'm only treating them as categorical features. – Nima Mohammadi Jan 9 '18 at 19:18
• This is a conceptual failure. user_id has no generalizability and hence no external validity and should not be considered as a predictor. – AdamO Jan 9 '18 at 21:07
• @AdamO This is actually common practice in CTR prediction challenges to use user_id. They usually use hashing trick to combine several features (device_id, platform_id, etc.) into a user_id and one-hot encode them into a space of several million distinct values to distinguish users. The evidence to the importance of user_id is that the choice of a proper hashing function leading to less collisions can vividly impact the performance. The troubling part is that the model doesn't see all these user_ids whose user's estimated affinity toward the item we seek, during training phase. – Nima Mohammadi Jan 9 '18 at 21:32

Unless 'user_id' has some information more than just being a unique identifier for a specific person...then you are creating a couple of issues in your model. If there is other information to be gathered (other than just a unique number) then it is best to try to capture that information in some other way. This could happen because your user_id is actually correlated to another variable and is acting as a stand-in for that variable.

For example: if user_id is actually representing how long the user joined the system -- then you translate the number to an ordinal or approximate time variable. However, you have to determine whether there are issues with with using user_id as a substitute for time. Did only 10 people join the first year and 100 the second and 1000 the third? Generally it is better to see if there is a sign-up date elsewhere in the data. In the time example you may be able to simply replace everyone who doesn't have an account with the shortest amount of time possible because they don't have an account.

Otherwise, if the variable can't be translated into something more meaningful:

1) You are over-fitting by using a variable that only applies to individuals in order to find a result. Example: It's great to know that Bob is more likely to make a claim than Judy is...but it has nothing to do with a model predicting whether John will. Your model will be tweaked to intentionally be better at the training set (even more than it normally would be).

2) You may not be capturing the actual information that is impacting your results. Does any individual user_id in the data occur more than once? If it doesn't occur more than once, and is having a meaningful effect on your model, then it is likely that the variable correlates to something else that is really having the impact. Try identifying what the correlation might be. If user_id does occur more than once then some models will simply weight the user_id based on how many times it occurs which is a frequency and not related the the actual user_id itself (and you can transform the data to represent this). Determine what the impact is of having multiple instances of a user id and how to deal with that.

It sounds like there is some data exploration still needed unless user_id means something different in your data set than it normally does.

• Thanks. I need to read your answer a couple more times. The train and test are ~90m and ~10m samples respectively, so we have more than one sample for each user (about 38 samples on average). My model only relies on user_id distinctly distinguishing users and I haven't made any assumption about it being ordinal. Excluding this field makes the accuracy worse though. Even tried to hash the values of user_id into smaller spaces (1m and 200k) but it worsen the performance both for the test set and my validation set (which I have my reservations about it being representative of the test data) – Nima Mohammadi Jan 9 '18 at 19:46
• My model doesn't take them as ordinal, but if it did, I guess hashing to a space of identical cardinality would set me straight. – Nima Mohammadi Jan 9 '18 at 19:49
• You run into the conceptual issue that you don't need to predict a known user's performance -- you already know that user's performance. If you try to break down into "known user" and "unknown user" you should just go ahead and split this into two models: One that tells you what you already know and one that predicts about things you don't know. There are a lot of reasons that including this could improve some metric you are reviewing. – Adam Sampson Jan 10 '18 at 0:57
• Also, it's not clear how you are determining "performance". Are you checking how well your model finds the current information? Or how well it predicts future values? Or some other measure? These are not all the same. – Adam Sampson Jan 10 '18 at 0:58
• Thanks. That's actually a very good idea. I didn't want to succumb to ensemble classifiers yet and wanted to work on my single model as much as I could. But this is actually seems inevitable that my rudimentary model needs to be split into two models for known and unknown users. – Nima Mohammadi Jan 10 '18 at 1:55