Joining train and test dataset when preprocessing data with encoders

Question

So I've been looking at several beginner Kaggle kernels for the 'Titanic - Machine Learning from Disaster' competition. In these kernels, I noticed that sometimes they combine train and test data, often using pd.concat(), when they are preprocessing data such as using encoders to fit and transform.

(https://www.kaggle.com/jeffd23/scikit-learn-ml-from-start-to-finish) In this case :

data_train = pd.read_csv('../input/train.csv')
data_test = pd.read_csv('../input/test.csv')

def encode_features(df_train, df_test):
    features = ['Fare', 'Cabin', 'Age', 'Sex', 'Lname', 'NamePrefix']
    df_combined = pd.concat([df_train[features], df_test[features]])
    
    
    for feature in features:
        le = preprocessing.LabelEncoder()
        le = le.fit(df_combined[feature])
        df_train[feature] = le.transform(df_train[feature])
        df_test[feature] = le.transform(df_test[feature])
    return df_train, df_test
data_train, data_test = encode_features(data_train, data_test)

Specifically, they would fit the encoder using the combined dataset but transform train/test independently.

In other cases, the encoder would only be fit using train data but transform would still be done independently on both train and test set. In both cases, they would then separate features and labels from the training dataset and use train_test_split().

I am confused what the exact difference these two approaches make, and why and when would one choose a certain method over the other. Some notebook(Titanic Top 4% with ensemble modeling) say they combined test and train set "to obtain the same number of features during categorical conversion," but I have yet to figure it out what that means.

This is my first question here so I'm not sure if I've worded the question properly or have provided enough context, but I would really appreciate it if anyone could help me understand this material easier!

Answer 1

In machine learning, we generally want to learn about a process or system which will be useful to us in a real world scenario. We train our model on a train set and then check our performance on the hold out test set. By checking how our model performs on previously unseen data, we get an idea of how our model might perform in the real world. We can also compare our train and test performance metrics to check for overfitting.

By fitting our preprocessing to both the train and test datasets, we might be leaking some information into our modelling process which should be unseen. If we only use the train set for figuring out which processing to do on our data, the test set truely remains unseen. After fitting our preprocessor to the train set, we can then transform the train and test datasets.