0
$\begingroup$

It may be a weird question because I don't fully understand hyperparameter-tuning yet.

Currently I'm using gridSearchCV of sklearn to tune the parameters of a randomForestClassifier like this:

gs = GridSearchCV(RandomForestClassifier(n_estimators=100, random_state=42), param_grid={'max_depth': range(5, 25, 4), 'min_samples_leaf': range(5, 40, 5),'criterion': ['entropy', 'gini']}, scoring=scoring, cv=3, refit='Accuracy', n_jobs=-1)
gs.fit(X_Distances, Y)
results = gs.cv_results_

After that I check the gs object for the best_params and best_score. Now I'm using best_params to instantiate a RandomForestClassifier and use stratified validation again to record metrics and print a confusion matrix:

rf = RandomForestClassifier(n_estimators=1000, min_samples_leaf=7, max_depth=18, criterion='entropy', random_state=42)
accuracy = []
metrics = {'accuracy':[], 'precision':[], 'recall':[], 'fscore':[], 'support':[]}
counter = 0

print('################################################### RandomForest ###################################################')
for train_index, test_index in skf.split(X_Distances,Y):
    X_train, X_test = X_Distances[train_index], X_Distances[test_index]
    y_train, y_test = Y[train_index], Y[test_index]
    rf.fit(X_train, y_train)
    y_pred = rf.predict(X_test)

    precision, recall, fscore, support = np.round(score(y_test, y_pred), 2)
    metrics['accuracy'].append(round(accuracy_score(y_test, y_pred), 2))
    metrics['precision'].append(precision)
    metrics['recall'].append(recall)
    metrics['fscore'].append(fscore)
    metrics['support'].append(support)

    print(classification_report(y_test, y_pred))
    matrix = confusion_matrix(y_test, y_pred)
    methods.saveConfusionMatrix(matrix, ('confusion_matrix_randomforest_distances_' + str(counter) +'.png'))
    counter = counter+1

meanAcc= round(np.mean(np.asarray(metrics['accuracy'])),2)*100
print('meanAcc: ', meanAcc)

Is this a reasonable approach or do I have something completely wrong?

EDIT:

I just tested the following:

gs = GridSearchCV(RandomForestClassifier(n_estimators=100, random_state=42), param_grid={'max_depth': range(5, 25, 4), 'min_samples_leaf': range(5, 40, 5),'criterion': ['entropy', 'gini']}, scoring=scoring, cv=3, refit='Accuracy', n_jobs=-1)
gs.fit(X_Distances, Y)

This yields best_score = 0.5362903225806451 at best_index = 28. When I check the accuracies in the 3 folds at index 28 I get:

  1. split0: 0.5185929648241207
  2. split1: 0.526686807653575
  3. split2: 0.5637651821862348

Which leads to the mean test accuracy: 0.5362903225806451. best_params: {'criterion': 'entropy', 'max_depth': 21, 'min_samples_leaf': 5}

Now I run this code which is using the mentioned best_params with a stratified 3 fold split (like GridSearchCV):

rf = RandomForestClassifier(n_estimators=100, min_samples_leaf=5, max_depth=21, criterion='entropy', random_state=42)
metrics = {'accuracy':[], 'precision':[], 'recall':[], 'fscore':[], 'support':[]}
counter = 0
print('################################################### RandomForest_Gini ###################################################')
for train_index, test_index in skf.split(X_Distances,Y):
    X_train, X_test = X_Distances[train_index], X_Distances[test_index]
    y_train, y_test = Y[train_index], Y[test_index]
    rf.fit(X_train, y_train)
    y_pred = rf.predict(X_test)

    precision, recall, fscore, support = np.round(score(y_test, y_pred))
    metrics['accuracy'].append(accuracy_score(y_test, y_pred))
    metrics['precision'].append(precision)
    metrics['recall'].append(recall)
    metrics['fscore'].append(fscore)
    metrics['support'].append(support)

    print(classification_report(y_test, y_pred))
    matrix = confusion_matrix(y_test, y_pred)
    methods.saveConfusionMatrix(matrix, ('confusion_matrix_randomforest_distances_' + str(counter) +'.png'))
    counter = counter+1

meanAcc= np.mean(np.asarray(metrics['accuracy']))
print('meanAcc: ', meanAcc)

The metrics dictionairy yields the exact same accuracies (split0: 0.5185929648241207, split1: 0.526686807653575, split2: 0.5637651821862348)

However the mean calculation is a bit off: 0.5363483182213101. With this approach I get the actual predictions of the best_estimator found by gridSearchCV. Now I can plot a confusion matrix for each fold to analyse. The productive model would be trained with my whole data set.

$\endgroup$
  • $\begingroup$ Have you heard of overfitting? If you used the whole dataset for fitting the model (hyperparameter tuning included), then during model evaluation you cannot tell anything about the possible future performance of the model, you are overfitting the model to the data that you have. $\endgroup$ – Tim Apr 11 '18 at 13:36
  • $\begingroup$ Yes I heard of overfitting. I just thought that its reasonable to use all the data I have to find the best parameters for the model. When I then test the model using cross validation I fit the modell only to the training data and test it with my test data. What is the point of only using 80% of my data for parameter tuning? $\endgroup$ – Christian Apr 11 '18 at 14:17
  • $\begingroup$ What is the purpose of using training set at all? $\endgroup$ – Tim Apr 11 '18 at 14:28
  • $\begingroup$ Doesn't GridSearchCv() handle the splitting into training/test data? I just want to know if my code makes sense like I posted it, I may have described it not good enough. The option cv=3 performs a 3 fold (stratified) cross validation during parameter tuning. $\endgroup$ – Christian Apr 11 '18 at 14:31
  • $\begingroup$ You use the same data twice: for finding the hyperparameters and then validating the model. So your error estimates would be overtly optimistic and would not give you clear picture of the future performance $\endgroup$ – Tim Apr 11 '18 at 14:56
2
$\begingroup$

Gridsearch uses crossvalidation, if you take the best parameters you should be able to reproduce the best result, just be carefull to leave aside your test data and use it only at the end.

20-30 % test data is the usual.

$\endgroup$
  • $\begingroup$ So I should split my dataset in train and test data (80/20), then perform gridSearchCV on my training data (inc. cross validation). At the end I use the best best estimator on my test data for testing? So basically it wouldn't make sense to record the confusion matrix during the cross validation in gridsearch cv? $\endgroup$ – Christian Apr 11 '18 at 21:37
  • $\begingroup$ That's what I do, I leave the confusion matrix for test. $\endgroup$ – Germán Alfaro Apr 11 '18 at 21:39
  • $\begingroup$ Ok, but how does it work when I want to test my model in the end with cross validation? If I only have 20% test data there is not much left to train with. $\endgroup$ – Christian Apr 11 '18 at 21:45
  • $\begingroup$ You don't train anymore. The best params are the ones to use in your model. $\endgroup$ – Germán Alfaro Apr 11 '18 at 21:47
  • $\begingroup$ Ah, so I make a stratified split in the beginning (80/20), perform gridSearchCV with my 80% training data and cross validation (e.g. 3 folds). Then I can use the best_estimator (which is already trained) to test on my test data and calculate confusion matrix? I thought I should do a cross validation to test my model because if there is very noisy data in my test set the result is not very good? $\endgroup$ – Christian Apr 11 '18 at 21:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.