1
$\begingroup$

I am running a random forest classifier for binary classification. I have split the dataset into training and testing. From training data, I select 70% of the data for hyperparameter tuning, I used 5 fold cross validation for that. Then I use the hyperparameters that produce high mean cross-validation accuracy (let's call it validation accuracy) and train the model on full training data and check the model on testing data. I get the testing accuracy.

My question is-

i) Should I report the best validation accuracy or the best testing accuracy?

I can use the seed (by proving random_state = xyz in random forest classifier) to fix the samples used for bootstrapping, which gives the best test accuracy.

If I need to report both the accuracies, can I select the seeds that give good validation accuracy and a good testing accuracy? What if I use the best test accuracy I got and report that in the paper?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

If I need to report both the accuracies, can I select the seeds that give good validation accuracy and a good testing accuracy? What if I use the best test accuracy I got and report that in the paper?

No. The idea of held out test set is that you are not allowed to look at the test set till your model is ready, so you can use it to get the final metric, so you can take the model or leave it. If you used the test set performance to tune the model, this would be a straightforward way to overfit to test set. Using different seeds and picking the best result is exactly that: you are choosing the model that best fits the test set, though you have no guarantees whatsoever that this is the best model that would generalize best. You would be cheating yourself and your readers.

On another hand, if you used something like $k$-fold cross-validation to assess the performance of the model (not to tune it), then you should report the average of the metrics, best if accompanied with some measure of variability like standard deviation.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Why we can't report the cv error as an estimate of the generalization error when we use cv for tuning hyperparameters? $\endgroup$
    – ado sar
    Commented Nov 8, 2022 at 10:40
  • $\begingroup$ @adosar not sure what you mean. $\endgroup$
    – Tim
    Commented Nov 8, 2022 at 10:46
  • $\begingroup$ Suppose that we perform k-fold cv to get the optimal hyperparameters and we don't have a separate test set. Are we allowed to report the average value of the k-fold cv error of the optimal model (best hyperparameters found from cv) as an estimate of its performance? $\endgroup$
    – ado sar
    Commented Nov 8, 2022 at 10:48
  • $\begingroup$ @adosar if you used CV for tuning, the validation set in CV would serve as a "training" set for hyperparameter optimization. It's the same problem as reporting training set metrics, just applied to hyperparameter optimization here. $\endgroup$
    – Tim
    Commented Nov 8, 2022 at 10:53
  • $\begingroup$ Lets say someone has exaclty the same training set that we have and he doesn't want to tune hyperparameters. He just chooses a set of hyperparameter values and wants to estimate the performance. He does k-fold cv and reports the average error. Lets say that he selected (maybe by chance) the optimal set we find from our cv procedure. If the k-folds are the same as our k-folds then the average error would be the same for both procedures. Why he can report his error as an estimate of the generalization error while we can't? $\endgroup$
    – ado sar
    Commented Nov 8, 2022 at 11:00
1
$\begingroup$

Typically the test accuracy is reported unless you're performing a particular analysis on your validation routine.

Choosing the best seed for your data is cheating. To stabilize your results, it's better to increase your number of bootstrap iterations so that you get similar results even if you change seeds. The seed should be there only for the purpose of reproducibility.

Improve this answer
$\endgroup$
2
  • $\begingroup$ By increasing the number of bootstrap iterations, do you mean to increase the number of trees? $\endgroup$
    – lsr729
    Commented Feb 5, 2022 at 16:51
  • $\begingroup$ I thought you were implementing a bootstrap layer out of random forest. To decrease variability, yes, I'd suggest increasing the number of trees. $\endgroup$
    – gunes
    Commented Feb 5, 2022 at 17:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.