5
$\begingroup$

I've seen quite a lot of conflicting views on if one-hot encoding (dummy variable creation) should be done before/after the training/test split.

Responses seem to state that one-hot encoding before leads to "data leakage".

This example states it's industry norm to do one-hot encoding on the entire data before training/test split:

Industry Example

This example from kaggle states that it should be done after the training/test split to avoid data leakage:

kaggle response - after split

My question is the following;

  1. Do we perform one-hot encoding before or after the Train/Test Split?
  2. Where is the data leakage occuring in the following example?

If we take the following example, we have two columns - web_views and website (non-ordinal categorical feature) (assuming we are one-hot encoding across the entire column, not dropping any dummies)

Our dataframe:

import pandas as pd from sklearn.model_selection import train_test_split import numpy as np

df = pd.DataFrame({'web_views': [100,200,300,400], 
                  'website': ['Youtube','Facebook','Instagram', 'Google']})

Scenario 1: One-Hot Encoding/Dummy Variables before splitting into Train/Test:

np.random.seed(123)


df_before_split = pd.concat([df.drop('website', axis = 1), pd.get_dummies(df['website'])], axis=1) 

# create your X and y dataframes
X_before_split = df_before_split.drop('web_views', axis = 1)
y_before_split = df_before_split['web_views']

# perform train test split
X_train_before_split, X_test_before_split, y_train_before_split, y_test_before_split = train_test_split(X_before_split, y_before_split, test_size = 0.20)

Now viewing the dataframes we have:

# view X train dataset (this is encoding before split)
X_train_before_split

and then for test

# View X test dataset dataset (this is encoding before split)
X_test_before_split

Scenario 2: One-Hot Encoding/Dummy Variables AFTER splitting into Train/Test:

# Perform One Hot encoding after the train/test split instead

X = df.drop('web_views', axis = 1)
y = df['web_views']

# perform data split: 
np.random.seed(123)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.20)

# perform one hot encoding on the train and test dataset datasets: 
X_train = pd.concat([X_train.drop('website', axis = 1), pd.get_dummies(X_train['website'])], axis=1)
X_test = pd.concat([X_test.drop('website', axis = 1), pd.get_dummies(X_test['website'])], axis=1)

Viewing the X_train and X_test dataframes:

# encode after train/test split - train dataframe
X_train

# encode after train/test split - test dataframe
X_test

Performing Linear Regression Modelling

Now that we have split our data to demonstrate we will create a simple linear model:

from sklearn.linear_model import LinearRegression

Before split linear model

regressor_before_split = LinearRegression()  
regressor_before_split.fit(X_train_before_split, y_train_before_split)

y_pred_before_split = regressor_before_split.predict(X_test_before_split)
y_pred_before_split

y_pred_before_split returns a predicting value what we would expect.

After split linear model

regressor_after_split = LinearRegression()  
regressor_after_split.fit(X_train, y_train)

y_pred_after_split = regressor_after_split.predict(X_test)
y_pred_after_split

Error message from Scenario 2:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-92-c63978a198c8> in <module>()
      2 regressor_after_split.fit(X_train, y_train)
      3 
----> 4 y_pred_after_split = regressor_after_split.predict(X_test)
      5 y_pred_after_split

C:\Anaconda3\lib\site-packages\sklearn\linear_model\base.py in predict(self, X)
    254             Returns predicted values.
    255         """
--> 256         return self._decision_function(X)
    257 
    258     _preprocess_data = staticmethod(_preprocess_data)

C:\Anaconda3\lib\site-packages\sklearn\linear_model\base.py in _decision_function(self, X)
    239         X = check_array(X, accept_sparse=['csr', 'csc', 'coo'])
    240         return safe_sparse_dot(X, self.coef_.T,
--> 241                                dense_output=True) + self.intercept_
    242 
    243     def predict(self, X):

C:\Anaconda3\lib\site-packages\sklearn\utils\extmath.py in safe_sparse_dot(a, b, dense_output)
    138         return ret
    139     else:
--> 140         return np.dot(a, b)
    141 
    142 

<__array_function__ internals> in dot(*args, **kwargs)

ValueError: shapes (1,1) and (3,) not aligned: 1 (dim 1) != 3 (dim 0)

My thoughts:

  1. Encoding with dummies before splitting ensures that the test data that we pass in e.g. X_test to perform the predicitions has the same shape as the training data that the model was trained on therefore understands how to predict values when it encounters these features - unlike with encoding after splitting, since the X_test data has only one feature we are using to make predicitions with whereas the X_train has 3 features
  2. Maybe I've introduced data leakage?

I'd be happy for someone to correct me if i've got things wrong or misinterpreted anything, but i'm stuck scratching me head if you encode before or after splitting!

Update as of 04.01.2023

You can perform categorical encoding of the feature column after you split the data into training and test sets. This may help for testing purposes because test data is supposed to be "unseen", at which point if a category is unseen you can iterate the model and add it as a new category to that particular feature etc.

In order to to do this in Python using sci-kit learn there is a parameter from the OneHotEncoder() - titled handle_unknown - when set to ignore it will do the following:

ignore’ : When an unknown category is encountered during transform, the resulting one-hot encoded columns for this feature will be all zeros. In the inverse transform, an unknown category will be denoted as None.

The default is error which is why an error message would be returned when it encounters a new categorical feature during fitting.

More information can be found here: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html

Also to add - when you do want to perform one hot encoding on the training/test split of the data remember to:

  1. Apply both fit and transform (or fit_transform) to the training data
  2. Apply only transform to the test data - this is due to it having been fitted on the training data already!
$\endgroup$
9
  • 2
    $\begingroup$ Welcome to Cross Validated! What do you see as the reasons there could be data leakage? What do you see as the reasons there wouldn’t be data leakage? $\endgroup$
    – Dave
    Commented Dec 19, 2022 at 13:19
  • 1
    $\begingroup$ I don't see any reason for data leakage, the model is not trained at the point of encoding. nor is it tested - the kaggle link i sent above is a very common view across the internet on encoding after splitting, i just want to know which is the correct way? $\endgroup$ Commented Dec 19, 2022 at 13:49
  • 2
    $\begingroup$ It's not unusual to find sloppy code on Kaggle. Here's another example: stats.stackexchange.com/questions/569630/… $\endgroup$
    – Sycorax
    Commented Dec 19, 2022 at 17:56
  • 3
    $\begingroup$ Is the concern here about the possibility that some levels of a factor might not appear within the training dataset? That's not really a coding issue: it's a conceptual one that goes to what you're trying to accomplish and has various solutions, such as using a random effects model instead of a fixed effects model. Or is the problem just inadequate coding in Python? $\endgroup$
    – whuber
    Commented Dec 19, 2022 at 18:22
  • 4
    $\begingroup$ @BeansOnToast The issue whuber alludes to is a bit more subtle. You can collect training data today and dummy encode it, but if there are new categories that appear when you try to apply your model tomorrow, you'll need to take steps to account for that; likewise if 1 or more of today's categories are missing. $\endgroup$
    – Sycorax
    Commented Dec 19, 2022 at 22:21

1 Answer 1

7
$\begingroup$

I can see no reason that one-hot encoding (dummy encoding) could lead to data leakage. This is really a simple data transformation, fully determined by the predictor variables, and in no need of estimation or training. It is comparable to calculating the logarithm of a positive variable, no need of any training. There is no choices involved, no use of the $Y$ variable. So doing this before or after train/test split should give exactly the same result.

But if the transformations are more involved, maybe target encoding Strange encoding for categorical features which uses the response variable $Y$. Then doing this before split would use the response variable also from the test set, so data leakage.

So maybe what to look at is if your encoding uses the response variable, such encodings might be more used in machine learning than in traditional statistics, explaining this strange advice you refer to.

$\endgroup$
1
  • 2
    $\begingroup$ Thanks @kjetil! I agree with your answer, including the case when strange encoding needs to be done - I truly believe that there is a lot of wrong information out there on Kaggle/medium/towards data science articles that can confuse someone, and this is probably a prime example! $\endgroup$ Commented Dec 19, 2022 at 22:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.