0
$\begingroup$

I'm tuning the hyperparameters for different Machine Learning models with Optuna package. For ExtraTreesRegressor I got r2 train = 0.938, r2 test = 0.922 and RMSE test = 0.907. In this case the average 5-fold CV on training data was 0.924.

But for HistGradientBoostingRegressor I got r2 train= 0.977, r2 test = 0.941 and RMSE test= 0.793. and average 5-fold CV on training data was 0.919. I'm wondering if the latter model overfit and which model is preferred in this situation?

$\endgroup$

1 Answer 1

1
$\begingroup$

Second model improved results for both training and testing sets, but has more variance so it may not be a good estimator of the pattern. Depending on the context (what is important given the problem and where the data you have sampled stands relative to the rest), you must decide between a possibly overfitting model (second) and a possibly biased model (first.) Why don't you run CV on all of the data you have? (Holdout and CV are seperate methods of model evaluation and it would give a better metric to make a decision here.)

$\endgroup$
2
  • $\begingroup$ Thanks! I tried 5-fold CV on the whole dataset but it is strange that the average value is very low ( it is a negative number!). I'm very confused. Do you know why this happens? $\endgroup$
    – Etemon
    Commented Nov 14, 2023 at 17:59
  • $\begingroup$ What is your evaluation method? I assume the previous CV results where accuracies, then it'd be negative due to an implementation mistake. $\endgroup$
    – tolgarecep
    Commented Nov 16, 2023 at 5:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.