# Is this the correct procedure for hyperparameter tuning and model evaluation?

As a follow up to my previous post (What is the correct procedure for nested cross-validation?) I wanted to see if my following nested cross-validation procedure is valid:

1. Split my data into train/test (80%/20% split) sets
X_train, X_test, Y_train, Y_test = train_test_split(X_res, Y_res, test_size=0.2, random_state=0)

1. Set my inner cross-validation, model and parameter search grid:
inner_cv = KFold(n_splits=5, shuffle=True, random_state=seed)
rfc = RandomForestClassifier(random_state=seed)
param_grid = {'bootstrap': [True, False],
'max_depth': [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, None],
'max_features': ['auto', 'sqrt', 'log2', None],
'min_samples_leaf': [1, 2, 4, 25],
'min_samples_split': [2, 5, 10, 25],
'criterion': ['gini', 'entropy'],
'n_estimators': [200, 400, 600, 800, 1000, 1200, 1400, 1600, 1800, 2000]}

1. I then run a randomised search to tune the hyperparameters on the train data only
rfclf = RandomizedSearchCV(rfc, param_grid, cv=inner_cv, n_iter=1000, n_jobs=-1, scoring='accuracy', verbose=1).fit(X_train, Y_train)

print(rfclf.best_score_)


Which returns: {'n_estimators': 200, 'min_samples_split': 25, 'min_samples_leaf': 2, 'max_features': 'auto', 'max_depth': 80, 'criterion': 'gini', 'bootstrap': True}

1. I then define the outer cross-validation and the model with the tuned hyperparameters
outer_cv = KFold(n_splits=5, shuffle=True, random_state=seed)
tunedModel = RandomForestClassifier(n_estimators = 200,
min_samples_split = 25,
min_samples_leaf = 2,
max_features = 'auto',
max_depth = 80,
criterion = 'gini',
bootstrap = True)

1. I run the outer cross-validation, get the mean accuracy and fit the model:
nested_cv_results = cross_val_score(tunedModel, X_train, Y_train, cv=outer_cv, scoring = 'accuracy', verbose=1)
tunedModel.fit(X_train, Y_train)

1. Finaly, I calculate the roc_auc score on the unseen test data:
print(roc_auc_score(Y_test, tunedModel.predict_proba(X_test)[:,1]))


I would like to know three things:

• Is this the correct procedure for nested cross-validation?
• Is the mean of nested_cv_results a measure of how well the model generalises?
• Is the roc_auc_score the predictive power of the model?