For reference, the dataset I'm using is the Kaggle Housing prices dataset.
The train data is (1460 x 80). I split it into a train and test dataset, with 1168 rows in the train set and 292 rows in the test dataset. So I would say it is sufficiently large for our purposes.
Most of the variables here are unordered categorical vars, so I one hot encoded them. Therefore, after preprocessing and creating dummy variables, my X_train
shape became 1168 rows x 233 variables (vs. 80 previously). I then used StandardScaler
on my X_train
, and was prepared to transform my X_test
dataset with the same StandardScaler parameters I used on my X_train (fit_transform
for the X_train
, just transform
for X_test
).
However, I'm getting an error that says ValueError: X has 208 features, but StandardScaler is expecting 233 features as input
, which makes alot of sense. Obviously, in this Kaggle case and in real life scenarios, the test/train split will have categorical variables that may not include all of the categories in each split. Eg. If we have 10 categories in a color
variable, and the larger training set gets all 10, but the test set only gets 8. Now the shapes are off due to one hot encoding. So how do we get StandardScaler to work on our test set? Are you just not supposed to StandardScaler on the test data?
In Kaggle, they actually provide you the train and test dataset separately, so this question is even more pertinent as I'm basically using a validation set within my training data, and I will have no idea what "final shape" the Kaggle official test dataset will be after my processing.