Say you have a variable (in this case industry) that you dummify (one hot encode) hence creating many new features in both the training and test sets for which you are getting ready to run a machine learning model. However, this dummy variable generates by several hundred more variables (features) in the test set then the training set.

What should you do?


What about just throwing out all the columns that aren't present in the training set? We have learned nothing about those industries, so I don't think they can teach you anything about your response variable.

There might be algorithm specific answers, but thinking about it as a linear model, if they had a 0 in every column that remains for industry, they would get the intercept effect for industry, while every industry that had data would be slightly different than the intercept.

  • $\begingroup$ But the problem is I am going to miss a lot of data on a pretty important variable. This is an imbalanced dataset that I am currently modeling by downsampling so I made it so that I have 8000 training data points and over 400000 test data points so it's perfectly logical that many will be missing from the training set. One alternative is to upsample, however, that will decrease the number of rows for which I can make a prediction, and I would like that to be as high as possible. $\endgroup$ – mathlover Aug 2 '18 at 18:45
  • 1
    $\begingroup$ But you can't learn about things that aren't in your training set. So if you're not going to train on it, how could you possibly want to predict using it? Your learning model has never seen it before! In case I wasn't clear, I'm not saying throw out industry as a whole. After the hot-encoding, throw out the columns that didn't appear in your training set. The ones that did can still have an estimated effect. $\endgroup$ – jntrcs Aug 2 '18 at 18:48
  • $\begingroup$ Yes, I know, I essentially currently have it as one mean encoded variable that is leaked from the test set to the training. So I basically just looked at how the variable did on the entire dataset, which I think technically would be fine to do if I were also just training on most of my dataset. The main reason I am not doing that is the big class imbalance and not using a sampling methodology seems to give me worse results. But yes, that is the issue, as of now I am using data leakage for the variable, which as you state is inaccurate $\endgroup$ – mathlover Aug 2 '18 at 18:53
  • 2
    $\begingroup$ Not suggesting there isn't one because I'm definitely not an expert, but I do not know any statistically valid reasons for upsampling and downsampling (this might have more info for you stats.stackexchange.com/questions/122409/why-downsample). You might research alternatives to that because yes, if you want a true train/test split (with accurate measurements of error), extracting means from your test set to use in training is not the way to go. $\endgroup$ – jntrcs Aug 2 '18 at 18:57
  • $\begingroup$ Yea, I am reading that now. However, it makes no sense that there would be no statistically no valid reason as there are specific statistical techniques to doing both, such as SMOTE, Tomek Links etc. So how can statistical techniques for upsampling or downsampling exist if they make no statistical sense? $\endgroup$ – mathlover Aug 3 '18 at 13:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.