1
$\begingroup$

Say you have a variable (in this case industry) that you dummify (one hot encode) hence creating many new features in both the training and test sets for which you are getting ready to run a machine learning model. However, this dummy variable generates by several hundred more variables (features) in the test set then the training set.

What should you do?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

What about just throwing out all the columns that aren't present in the training set? We have learned nothing about those industries, so I don't think they can teach you anything about your response variable.

There might be algorithm specific answers, but thinking about it as a linear model, if they had a 0 in every column that remains for industry, they would get the intercept effect for industry, while every industry that had data would be slightly different than the intercept.

Improve this answer
$\endgroup$
7
  • $\begingroup$ But the problem is I am going to miss a lot of data on a pretty important variable. This is an imbalanced dataset that I am currently modeling by downsampling so I made it so that I have 8000 training data points and over 400000 test data points so it's perfectly logical that many will be missing from the training set. One alternative is to upsample, however, that will decrease the number of rows for which I can make a prediction, and I would like that to be as high as possible. $\endgroup$
    – data_science_math
    Commented Aug 2, 2018 at 18:45
  • 1
    $\begingroup$ But you can't learn about things that aren't in your training set. So if you're not going to train on it, how could you possibly want to predict using it? Your learning model has never seen it before! In case I wasn't clear, I'm not saying throw out industry as a whole. After the hot-encoding, throw out the columns that didn't appear in your training set. The ones that did can still have an estimated effect. $\endgroup$
    – jntrcs
    Commented Aug 2, 2018 at 18:48
  • $\begingroup$ Yes, I know, I essentially currently have it as one mean encoded variable that is leaked from the test set to the training. So I basically just looked at how the variable did on the entire dataset, which I think technically would be fine to do if I were also just training on most of my dataset. The main reason I am not doing that is the big class imbalance and not using a sampling methodology seems to give me worse results. But yes, that is the issue, as of now I am using data leakage for the variable, which as you state is inaccurate $\endgroup$
    – data_science_math
    Commented Aug 2, 2018 at 18:53
  • 2
    $\begingroup$ Not suggesting there isn't one because I'm definitely not an expert, but I do not know any statistically valid reasons for upsampling and downsampling (this might have more info for you stats.stackexchange.com/questions/122409/why-downsample). You might research alternatives to that because yes, if you want a true train/test split (with accurate measurements of error), extracting means from your test set to use in training is not the way to go. $\endgroup$
    – jntrcs
    Commented Aug 2, 2018 at 18:57
  • $\begingroup$ Yea, I am reading that now. However, it makes no sense that there would be no statistically no valid reason as there are specific statistical techniques to doing both, such as SMOTE, Tomek Links etc. So how can statistical techniques for upsampling or downsampling exist if they make no statistical sense? $\endgroup$
    – data_science_math
    Commented Aug 3, 2018 at 13:54
0
$\begingroup$

(This specific question is kinda old now, but the issue is common, and our answers are an enduring resource. So please ignore the untimeliness of this reply.)

It appears from the description and comments that you've created an arbitrary sample of 8,000 observations for the training set, and that's not serving you well.

Obvious things you could do to help with that would be:

  • increase the size of the training set, and/or
  • develop a stratified sampling method to split your data into training vs test, which forces in those predictors (industry flags) you believe ought to be included.

The quote "this dummy variable generates by several hundred more variables (features) in the test set then the training set" also suggests that your training dataset could be relatively wide relative to its height. In other words, you may have insufficient EPV (events per variable) to have a stable & credible model. Another reason to increase the size of the training data.

Personally, I'm always going start with a training set that is As Big As Possible. I'll only cut it down IF the training process chugs along for an intolerable amount of time. If the initial training set had been half of the available data ... i.e., 200k records rather than 8k ... it's likely that no time would have been wasted trying to understand and solve the "some industries aren't represented in the training data" problem that was created by the choice to use a small sample. Computer time is usually much cheaper than people-time. (That said, I don't deal with Amazon/Google-sized datasets.)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.