3
$\begingroup$

"Nested CV to testing data" is similar to "CV to validation data".

  • Validation data is used to tune hyperparameters and prevent overfitting.

  • CV lets us get multiple copies of validation data (ideally different). So we our parameter selection process can be more robust.

  • Testing data is to used evaluate the performance of the selected model (refit using whole training data with optimal parameters selected by CV).

  • Outer loop of Nested CV lets us get multiple copies of testing data. Our final model evaluation (such as MSE, AUC) can be more robust.

With above statements, the purpose of Nested CV is to evaluate model performance better, NOT to tune parameters better.

Is anything wrong about my understanding? Thanks!

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ I think you have this correct. Nested CV validates the procedure of selecting hyperparameters via cross validation. The hyperparameter selection thus becomes part of the model, and thus we validate it. $\endgroup$
    – Demetri Pananos
    Mar 1, 2021 at 21:26
  • $\begingroup$ @DemetriPananos Thank you! Does that mean the parameter tuning process is validated if performance on the outer folds are similar (small variance on AUC or MSE)? Otherwise, something might be wrong in the modeling or data splitting. $\endgroup$
    – H.Yuanchen
    Mar 2, 2021 at 2:25

1 Answer 1

Reset to default
3
$\begingroup$

Your understanding is correct, the tuning process doesn't improve from nested CV as it is handed by the "inner loop". What nested CV does is that it helps the final evaluation be more robust and objective, as it is done based on multiple test sets.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.