Why is my Random Forest Regression performing worse in cross validation than on a baseline?

Question

So I am trying to use a Random Forest Regression on a dataset with a mix of categorical and numeric data types. The predictors are in X_train and X_test. I used a 80/20 split resulting in 256 vs 64 observations. I set up a preprocessing pipeline which imputes missing values with the median and then encodes the categorical variables (I used one hot for a binary variable, ordinal for another and hash encoding for the last since it had about 98 unique values). After that the pipeline fits the Random Forest. After the encoding the result is 19 predictors for the target variable I am trying to predict.

My problem is that when I run this on all of X_train and measure the training accuracy and the performance on X_test to form a baseline I am getting better results than running the cross validation using 5-fold CV. In fact here is my output:

For the baseline where I run the whole pipeline on X_train:

R2 on training data:  0.9770830687502748 
 R2 on test data:  0.8590100930540333 
 RMSE on training data:  0.15177396779032892 
 RMSE on test data:  0.32237641157671765

Where I am using R2 value and the RMSE as performance metrics.

For the cross validation I am using 5-fold and cross validating for max_depth using a range values given by list(range(2,22,2)). I get this from the cross validation:

RF best hyperparameters were:  {'randomforestregressor__max_depth': 2}
R2 on training data:  0.7951554670350791 
 R2 on test data:  0.7737034455273433 
 RMSE on training data:  0.45376526245074367 
 RMSE on test data:  0.40842114225679055

Why is this happening? My understanding would be that it should have performed at least similarly, not significantly worse. I can't seem to pick out what the problem might be. I am using the same random_state parameter for the baseline and for the cross validation are the same so it's probably not by chance either.

I guess my problem is similar to this person's post here? But it didn't seem like he had found an answer.

EDIT: Here is some more code as requested. I had to use some custom transformers because I need the output of the preprocessing to still be a dataframe. Here they are together with the final pipeline

import category_encoders as ce
from sklearn.preprocessing import FunctionTransformer
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.ensemble import RandomForestRegressor

def SimpleImputerDF(df):
    """Impute missing values of with median and return df"""
    return df.fillna(df.median())

def BinariserDF(df):
    """Binarises new_store column in dataframe and drops other column"""
    df_binary = df.copy()
    if ('new_store' in list(df.columns)):
        df_binary = pd.concat([df_binary, pd.get_dummies(df['new_store'])], axis=1)
        df_binary = df_binary.drop(['new_store','no'], axis=1)
    return df_binary

Ordinal = ce.OrdinalEncoder(cols='transport_availability') # ordinal encoding of transport_availability
Hash = ce.HashingEncoder(cols='county',n_components=7) # hash encoding of the county
preprocess = make_pipeline(FunctionTransformer(SimpleImputerDF), FunctionTransformer(BinariserDF), Ordinal, Hash)

rf = RandomForestRegressor(n_estimators=500, random_state=12)
final_pipeline = make_pipeline(preprocess, rf)


clf = GridSearchCV(final_pipeline, hyperparam, cv=crossval, n_jobs=n_jobs) # cross validate
clf = clf.fit(X_train, y_train) # fit model

Note I just reran the code cross validating for max_features too to see if that made a difference. In both cases I am getting something VERY strange - when I try to get the best_score for the cross validated fit I am getting

RF.best_score_

nan

This could be what's causing my problems. Do you know why this could be happening? I checked that there are no missing values after using preprocess on X_train by running preprocess.fit_transform(X_train) and indeed there are none.

EDIT2: A suggestion was made that it may be my custom function BinariserDF that is causing the problem. So I followed the suggestion and instead used make_column_transformer instead using:

numerical_ix = X_train.select_dtypes(include=['int64', 'float64']).columns
Binary = ce.OneHotEncoder() # binary encoding of new_store
Ordinal = ce.OrdinalEncoder() # ordinal encoding of transport_availability
Hash = ce.HashingEncoder(n_components=7) # hash encoding of the county

preprocessor = make_column_transformer((SimpleImputer(missing_values=np.nan, strategy='median'), numerical_ix),
                       (Binary, 'new_store'),
                        (Ordinal, 'transport_availability'),
                        (Hash, 'county')
                       )

Running this with still gives me the strange nan error. Any ideas?

Answer 1

Your function BinariserDF is probably the problem. Since you're using it in a FunctionTransformer, it gets called separately for the training and test folds in the cross-validation, so the number of dummy variables may be different, and the model scoring fails.

Instead, use SimpleImputer and ColumnTransformer with OneHotEncoder. (The encoding is also probably safe to do on the entire dataset, if you know what you're doing; but the imputation shouldn't be done on the entire set nor separately on the train and test sets as you do now.)