So I am trying to use a Random Forest Regression on a dataset with a mix of categorical and numeric data types. The predictors are in X_train and X_test. I used a 80/20 split resulting in 256 vs 64 observations. I set up a preprocessing pipeline which imputes missing values with the median and then encodes the categorical variables (I used one hot for a binary variable, ordinal for another and hash encoding for the last since it had about 98 unique values). After that the pipeline fits the Random Forest. After the encoding the result is 19 predictors for the target variable I am trying to predict.

My problem is that when I run this on all of X_train and measure the training accuracy and the performance on X_test to form a baseline I am getting better results than running the cross validation using 5-fold CV. In fact here is my output:

For the baseline where I run the whole pipeline on X_train:

R2 on training data:  0.9770830687502748 
 R2 on test data:  0.8590100930540333 
 RMSE on training data:  0.15177396779032892 
 RMSE on test data:  0.32237641157671765

Where I am using R2 value and the RMSE as performance metrics.

For the cross validation I am using 5-fold and cross validating for max_depth using a range values given by list(range(2,22,2)). I get this from the cross validation:

RF best hyperparameters were:  {'randomforestregressor__max_depth': 2}
R2 on training data:  0.7951554670350791 
 R2 on test data:  0.7737034455273433 
 RMSE on training data:  0.45376526245074367 
 RMSE on test data:  0.40842114225679055

Why is this happening? My understanding would be that it should have performed at least similarly, not significantly worse. I can't seem to pick out what the problem might be. I am using the same random_state parameter for the baseline and for the cross validation are the same so it's probably not by chance either.

I guess my problem is similar to this person's post here? But it didn't seem like he had found an answer.

EDIT: Here is some more code as requested. I had to use some custom transformers because I need the output of the preprocessing to still be a dataframe. Here they are together with the final pipeline

import category_encoders as ce
from sklearn.preprocessing import FunctionTransformer
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.ensemble import RandomForestRegressor

def SimpleImputerDF(df):
    """Impute missing values of with median and return df"""
    return df.fillna(df.median())

def BinariserDF(df):
    """Binarises new_store column in dataframe and drops other column"""
    df_binary = df.copy()
    if ('new_store' in list(df.columns)):
        df_binary = pd.concat([df_binary, pd.get_dummies(df['new_store'])], axis=1)
        df_binary = df_binary.drop(['new_store','no'], axis=1)
    return df_binary

Ordinal = ce.OrdinalEncoder(cols='transport_availability') # ordinal encoding of transport_availability
Hash = ce.HashingEncoder(cols='county',n_components=7) # hash encoding of the county
preprocess = make_pipeline(FunctionTransformer(SimpleImputerDF), FunctionTransformer(BinariserDF), Ordinal, Hash)

rf = RandomForestRegressor(n_estimators=500, random_state=12)
final_pipeline = make_pipeline(preprocess, rf)

clf = GridSearchCV(final_pipeline, hyperparam, cv=crossval, n_jobs=n_jobs) # cross validate
clf = clf.fit(X_train, y_train) # fit model

Note I just reran the code cross validating for max_features too to see if that made a difference. In both cases I am getting something VERY strange - when I try to get the best_score for the cross validated fit I am getting



This could be what's causing my problems. Do you know why this could be happening? I checked that there are no missing values after using preprocess on X_train by running preprocess.fit_transform(X_train) and indeed there are none.

EDIT2: A suggestion was made that it may be my custom function BinariserDF that is causing the problem. So I followed the suggestion and instead used make_column_transformer instead using:

numerical_ix = X_train.select_dtypes(include=['int64', 'float64']).columns
Binary = ce.OneHotEncoder() # binary encoding of new_store
Ordinal = ce.OrdinalEncoder() # ordinal encoding of transport_availability
Hash = ce.HashingEncoder(n_components=7) # hash encoding of the county

preprocessor = make_column_transformer((SimpleImputer(missing_values=np.nan, strategy='median'), numerical_ix),
                       (Binary, 'new_store'),
                        (Ordinal, 'transport_availability'),
                        (Hash, 'county')

Running this with still gives me the strange nan error. Any ideas?

  • $\begingroup$ A few things to note:A. 320 observation are too few to have a stable training-test split. B. When first testing the pipeline, unless already set, the RF is not constrained to be between 2 and 22 depth. C. Especially because each training sample is so small mean imputation can impose variability. Having said the above can you please: 1. Try and use max_depth equal to 2 when computing the base-line performance? 2. Give a more code as to why the baseline performance is trained/computed? 3. Use repeated cross-validation instead of a single repeat? 4. Potentially not use a train-test split. $\endgroup$ – usεr11852 Sep 27 '20 at 12:02
  • $\begingroup$ @usεr11852 I have added more code as requested. I think it could be something wrong in the pipeline? I don't understand why RF.best_score_ is nan. $\endgroup$ – Ansh Sep 27 '20 at 13:19
  • $\begingroup$ "My understanding would be that it should have performed at least similarly, not significantly worse." This is not correct, even for a correctly used Random Forest. RF is designed to have a large gap between training and testing performance. Check out the chapter on RF in Elements of Statistical Learning (free online). $\endgroup$ – Matthew Drury Sep 27 '20 at 14:59
  • $\begingroup$ @MatthewDrury: I have read Ch.15 from ESEL a couple of times and I have missed that point. Can you please a bit more specific on this? I appreciate that Hastie et al. "might" allude to this by criticizing the idea of "Another claim is that random forests “cannot overfit” the data." but that is a pretty general (and of course valid) criticism. Also, I think (at least) the OP refers to error difference observed on the test data between the baseline and the "tuned" model. For the "tuned" model the difference between training and test performance is "reasonable" ($R^2$ of 79.5% to 77.3%). $\endgroup$ – usεr11852 Sep 27 '20 at 17:35
  • $\begingroup$ re: edit2, do you get any warnings about failed fits? Do all the scores in cv_results_ come out NaN? Can you try removing subsets of the columntransformers to narrow down the problem? $\endgroup$ – Ben Reiniger Sep 28 '20 at 1:31

Your function BinariserDF is probably the problem. Since you're using it in a FunctionTransformer, it gets called separately for the training and test folds in the cross-validation, so the number of dummy variables may be different, and the model scoring fails.

Instead, use SimpleImputer and ColumnTransformer with OneHotEncoder. (The encoding is also probably safe to do on the entire dataset, if you know what you're doing; but the imputation shouldn't be done on the entire set nor separately on the train and test sets as you do now.)

  • $\begingroup$ but then SimpleImputer outputs a numpy array. Do you know of any work around so I can get column names after SimpleImputer or OneHotEncoder is applied? The OneHotEncoder implementation from category_encoders preserves dataframes but SimpleImputer does not. The reason why I am asking this is because eventually I would like to plot feature importance and I would need the column names to see which number in the array corresponds to which features $\endgroup$ – Ansh Sep 27 '20 at 15:07
  • $\begingroup$ please see 'EDIT2' in the question $\endgroup$ – Ansh Sep 27 '20 at 15:33
  • $\begingroup$ Feature name processing is still in the works in sklearn. Since SimpleImputer is relatively simple, you might be able to wrap your own that works directly on dataframes for your use-case, or just add get_feature_names to the sklearn class locally. $\endgroup$ – Ben Reiniger Sep 29 '20 at 1:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.