# Hyperparameter tuning on the whole data set reasonable?

It may be a weird question because I don't fully understand hyperparameter-tuning yet.

Currently I'm using gridSearchCV of sklearn to tune the parameters of a randomForestClassifier like this:

gs = GridSearchCV(RandomForestClassifier(n_estimators=100, random_state=42), param_grid={'max_depth': range(5, 25, 4), 'min_samples_leaf': range(5, 40, 5),'criterion': ['entropy', 'gini']}, scoring=scoring, cv=3, refit='Accuracy', n_jobs=-1)
gs.fit(X_Distances, Y)
results = gs.cv_results_


After that I check the gs object for the best_params and best_score. Now I'm using best_params to instantiate a RandomForestClassifier and use stratified validation again to record metrics and print a confusion matrix:

rf = RandomForestClassifier(n_estimators=1000, min_samples_leaf=7, max_depth=18, criterion='entropy', random_state=42)
accuracy = []
metrics = {'accuracy':[], 'precision':[], 'recall':[], 'fscore':[], 'support':[]}
counter = 0

print('################################################### RandomForest ###################################################')
for train_index, test_index in skf.split(X_Distances,Y):
X_train, X_test = X_Distances[train_index], X_Distances[test_index]
y_train, y_test = Y[train_index], Y[test_index]
rf.fit(X_train, y_train)
y_pred = rf.predict(X_test)

precision, recall, fscore, support = np.round(score(y_test, y_pred), 2)
metrics['accuracy'].append(round(accuracy_score(y_test, y_pred), 2))
metrics['precision'].append(precision)
metrics['recall'].append(recall)
metrics['fscore'].append(fscore)
metrics['support'].append(support)

print(classification_report(y_test, y_pred))
matrix = confusion_matrix(y_test, y_pred)
methods.saveConfusionMatrix(matrix, ('confusion_matrix_randomforest_distances_' + str(counter) +'.png'))
counter = counter+1

meanAcc= round(np.mean(np.asarray(metrics['accuracy'])),2)*100
print('meanAcc: ', meanAcc)


Is this a reasonable approach or do I have something completely wrong?

EDIT:

I just tested the following:

gs = GridSearchCV(RandomForestClassifier(n_estimators=100, random_state=42), param_grid={'max_depth': range(5, 25, 4), 'min_samples_leaf': range(5, 40, 5),'criterion': ['entropy', 'gini']}, scoring=scoring, cv=3, refit='Accuracy', n_jobs=-1)
gs.fit(X_Distances, Y)


This yields best_score = 0.5362903225806451 at best_index = 28. When I check the accuracies in the 3 folds at index 28 I get:

1. split0: 0.5185929648241207
2. split1: 0.526686807653575
3. split2: 0.5637651821862348

Which leads to the mean test accuracy: 0.5362903225806451. best_params: {'criterion': 'entropy', 'max_depth': 21, 'min_samples_leaf': 5}

Now I run this code which is using the mentioned best_params with a stratified 3 fold split (like GridSearchCV):

rf = RandomForestClassifier(n_estimators=100, min_samples_leaf=5, max_depth=21, criterion='entropy', random_state=42)
metrics = {'accuracy':[], 'precision':[], 'recall':[], 'fscore':[], 'support':[]}
counter = 0
print('################################################### RandomForest_Gini ###################################################')
for train_index, test_index in skf.split(X_Distances,Y):
X_train, X_test = X_Distances[train_index], X_Distances[test_index]
y_train, y_test = Y[train_index], Y[test_index]
rf.fit(X_train, y_train)
y_pred = rf.predict(X_test)

precision, recall, fscore, support = np.round(score(y_test, y_pred))
metrics['accuracy'].append(accuracy_score(y_test, y_pred))
metrics['precision'].append(precision)
metrics['recall'].append(recall)
metrics['fscore'].append(fscore)
metrics['support'].append(support)

print(classification_report(y_test, y_pred))
matrix = confusion_matrix(y_test, y_pred)
methods.saveConfusionMatrix(matrix, ('confusion_matrix_randomforest_distances_' + str(counter) +'.png'))
counter = counter+1

meanAcc= np.mean(np.asarray(metrics['accuracy']))
print('meanAcc: ', meanAcc)


The metrics dictionairy yields the exact same accuracies (split0: 0.5185929648241207, split1: 0.526686807653575, split2: 0.5637651821862348)

However the mean calculation is a bit off: 0.5363483182213101. With this approach I get the actual predictions of the best_estimator found by gridSearchCV. Now I can plot a confusion matrix for each fold to analyse. The productive model would be trained with my whole data set.

• Have you heard of overfitting? If you used the whole dataset for fitting the model (hyperparameter tuning included), then during model evaluation you cannot tell anything about the possible future performance of the model, you are overfitting the model to the data that you have. – Tim Apr 11 '18 at 13:36
• Yes I heard of overfitting. I just thought that its reasonable to use all the data I have to find the best parameters for the model. When I then test the model using cross validation I fit the modell only to the training data and test it with my test data. What is the point of only using 80% of my data for parameter tuning? – Christian Apr 11 '18 at 14:17
• What is the purpose of using training set at all? – Tim Apr 11 '18 at 14:28
• Doesn't GridSearchCv() handle the splitting into training/test data? I just want to know if my code makes sense like I posted it, I may have described it not good enough. The option cv=3 performs a 3 fold (stratified) cross validation during parameter tuning. – Christian Apr 11 '18 at 14:31
• You use the same data twice: for finding the hyperparameters and then validating the model. So your error estimates would be overtly optimistic and would not give you clear picture of the future performance – Tim Apr 11 '18 at 14:56