I have a relatively small, imbalanced data set (~3k datapoints, 12 classes). I want to tune the parameters of a RandomForestClassifier and eventually test the model.

Currently I'm doing it like this, but strangely it yields higher scores on the test set than on the training set (I used cohen_kappa_score and accuracy)

#Split data in training and test set (70/30 stratified split)
x_train, x_test, y_train, y_test = train_test_split(X_Distances, Y, test_size=0.3, random_state=42, stratify=Y)

#Scorings used for parameter tuning evaluation
scoring = {'Accuracy' : make_scorer(accuracy_score), 'Recall' : 'recall_macro', 'Kappa' : make_scorer(cohen_kappa_score)}

#Initializing of parameter ranges
params_randomSearch = {"min_samples_leaf": np.arange(1,30,2),
              "min_samples_split": np.arange(2,20,2),
              "max_depth": np.arange(2, 20, 2),
              "min_weight_fraction_leaf": np.arange(0. ,0.4, 0.1),
              "n_estimators": np.arange(10, 1000, 100),
              "max_features" : ['auto', 'sqrt', 'log2', None],
              "criterion" : ['entropy', 'gini']}

if __name__ == '__main__':
    rs = RandomizedSearchCV(RandomForestClassifier(random_state=42), param_distributions=params_randomSearch, scoring = scoring, cv = 3, refit = 'Kappa', n_iter=60, n_jobs=-1, random_state=42)
    rs.fit(x_train, y_train)
    print('Best Score: ', rs.best_score_, '\nBest parameters: ', rs.best_params_)
    y_predict = rs.best_estimator_.predict(x_test)
    acc = cohen_kappa_score(y_test, y_predict)

Results - Accuracy:

best_score_ = {float64} 0.5103216514642342
best_params_ = {dict} {'n_estimators': 310, 'min_weight_fraction_leaf': 0.0, 'min_samples_split': 12, 'min_samples_leaf': 5, 'max_features': 'auto', 'max_depth': 14, 'criterion': 'entropy'}

# 1. Eval
Accuracy of base (default) classifier on test set:  0.47928331466965285
Accuracy of classifier with best_params of RandomSearchCV on test set:  0.5666293393057111

Same results for the cohen_kappa_score, interestingley I get the exact same model using 'Kappa' as refit score

I don't know if this is an acceptable result or if something is wrong with my approach. Since the dataset is small, maybe the test set is just "easier"?

Another random seed for the data split:

random_state = 1 on data split results with cohen_kappa:

best_params_ = {dict} {'n_estimators': 310, 'min_weight_fraction_leaf': 0.0, 'min_samples_split': 12, 'min_samples_leaf': 5, 'max_features': 'auto', 'max_depth': 14, 'criterion': 'entropy'}
best_score_ = {float64} 0.4302884321273102
best_estimator on test set: 0.45628479220654583
  • $\begingroup$ Are you looking at out-of-bag performance estimates for the training set? $\endgroup$ – Scortchi Apr 12 '18 at 18:54
  • $\begingroup$ I'm looking at the best_score field of the random search object which is a mean kappa/accuracy of the cv $\endgroup$ – Christian Apr 12 '18 at 19:14
  • $\begingroup$ @Scortchi♦ how to I measure out-of-bag performance estimates for the training set during RandomizedSearchCV? $\endgroup$ – Christian Apr 13 '18 at 6:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.