1
$\begingroup$

For reference, the dataset I'm using is the Kaggle Housing prices dataset.

The train data is (1460 x 80). I split it into a train and test dataset, with 1168 rows in the train set and 292 rows in the test dataset. So I would say it is sufficiently large for our purposes.

Most of the variables here are unordered categorical vars, so I one hot encoded them. Therefore, after preprocessing and creating dummy variables, my X_train shape became 1168 rows x 233 variables (vs. 80 previously). I then used StandardScaler on my X_train, and was prepared to transform my X_test dataset with the same StandardScaler parameters I used on my X_train (fit_transform for the X_train, just transform for X_test).

However, I'm getting an error that says ValueError: X has 208 features, but StandardScaler is expecting 233 features as input, which makes alot of sense. Obviously, in this Kaggle case and in real life scenarios, the test/train split will have categorical variables that may not include all of the categories in each split. Eg. If we have 10 categories in a color variable, and the larger training set gets all 10, but the test set only gets 8. Now the shapes are off due to one hot encoding. So how do we get StandardScaler to work on our test set? Are you just not supposed to StandardScaler on the test data?

In Kaggle, they actually provide you the train and test dataset separately, so this question is even more pertinent as I'm basically using a validation set within my training data, and I will have no idea what "final shape" the Kaggle official test dataset will be after my processing.

$\endgroup$

2 Answers 2

1
$\begingroup$

How to fix StandardScaler would be a programming question, off-topic here, but there's a deeper problem beneath.

If you have a different number of columns in the train vs test set, the additional columns, beyond the common ones, cannot be used. If you trained a model with some feature that is absent in the test set, you cannot validate how it impacts the results. If you trained a model without a feature that is present in the test set, yourthe the model has no way of using it to make predictions.

Notice that even if you replaced the features with something, it wouldn't solve the problem. If you replaced such a column with a constant in a training set, a constant column would not affect the results, so you could as well ignore it. If you did it for the training set, you wouldn't be able to validate its impact because it would always be the same.

The only good solution is to get more data. A slightly worse one is to predict the features with another model first. But training or validating model trained on predicted features is not the same as on the real ones.

Finally, the problem can happen when your data is sparse. For example, if you are dealing with language data and you are counting word occurrences as features, by chance you could not observe some word in either of the sets. If you coded it poorly and in such a case you would record no column instead of a column of zeros, the problem would arise. Notice however that even if such data would be correct, you would still have the problems described above.

$\endgroup$
2
  • $\begingroup$ I'm not repairing variables in this particular case though -- that would be a valid but different discussion. The variables I have are categorical variables. How would you deal with those if you cant one-hot encode them into new columns? $\endgroup$
    – Katsu
    Feb 2, 2023 at 18:40
  • $\begingroup$ @Katsu I'm not sure what you mean. If you don't have the data, you can't use the features. $\endgroup$
    – Tim
    Feb 2, 2023 at 20:08
0
$\begingroup$

For everyone else thats searching for the answer: Balance the 2 datasets by encoding all 0's for the missing columns.

Eg. Train set has Color: Green, Blue, Test set has Color: Red, Green.

You will have 4 OneHotEncoded columns in both Train and Test datasets.

The Blue column will be all 0's in the Test set and the Red column will be all 0's in the Train dataset.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.