0
$\begingroup$

As a follow up to my previous post (What is the correct procedure for nested cross-validation?) I wanted to see if my following nested cross-validation procedure is valid:

  1. Split my data into train/test (80%/20% split) sets
X_train, X_test, Y_train, Y_test = train_test_split(X_res, Y_res, test_size=0.2, random_state=0)
  1. Set my inner cross-validation, model and parameter search grid:
inner_cv = KFold(n_splits=5, shuffle=True, random_state=seed)
rfc = RandomForestClassifier(random_state=seed)
param_grid = {'bootstrap': [True, False],
              'max_depth': [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, None],
              'max_features': ['auto', 'sqrt', 'log2', None],
              'min_samples_leaf': [1, 2, 4, 25],
              'min_samples_split': [2, 5, 10, 25],
              'criterion': ['gini', 'entropy'],
              'n_estimators': [200, 400, 600, 800, 1000, 1200, 1400, 1600, 1800, 2000]}
  1. I then run a randomised search to tune the hyperparameters on the train data only
rfclf = RandomizedSearchCV(rfc, param_grid, cv=inner_cv, n_iter=1000, n_jobs=-1, scoring='accuracy', verbose=1).fit(X_train, Y_train)
  1. I get the best parameters of grid search:
print(rfclf.best_score_)

Which returns: {'n_estimators': 200, 'min_samples_split': 25, 'min_samples_leaf': 2, 'max_features': 'auto', 'max_depth': 80, 'criterion': 'gini', 'bootstrap': True}

  1. I then define the outer cross-validation and the model with the tuned hyperparameters
outer_cv = KFold(n_splits=5, shuffle=True, random_state=seed)
tunedModel = RandomForestClassifier(n_estimators = 200,
                                    min_samples_split = 25,
                                    min_samples_leaf = 2,
                                    max_features = 'auto',
                                    max_depth = 80,
                                    criterion = 'gini',
                                    bootstrap = True)
  1. I run the outer cross-validation, get the mean accuracy and fit the model:
nested_cv_results = cross_val_score(tunedModel, X_train, Y_train, cv=outer_cv, scoring = 'accuracy', verbose=1)
tunedModel.fit(X_train, Y_train)
  1. Finaly, I calculate the roc_auc score on the unseen test data:
print(roc_auc_score(Y_test, tunedModel.predict_proba(X_test)[:,1]))

I would like to know three things:

  • Is this the correct procedure for nested cross-validation?
  • Is the mean of nested_cv_results a measure of how well the model generalises?
  • Is the roc_auc_score the predictive power of the model?
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.