2
$\begingroup$

Lets assume for example that I am attempting to predict a binary outcome using p predictors in which n>p with methods including a LASSO Regression, a Logistic Regression and SVM with an RBF kernel. Lets also assume that I plan to use nested cross-validation for model selection and generalization error estimation.

My question is:

Should we estimate performance in the outer loop only for the best model? What if I estimate performance in the outer loop for the logistic regression model, the best LASSO model and best SVM model from the inner loop? Will then selecting one of these 3 'best' models for production based on the generalization error of the outer loop lead to bias? It is sometimes the case that if a more interpretable model such as a logistic model has only a slight decrease in performance compared to a more complex model, one might prefer the more interpretable model at the cost of some performance - hence, the motivation to estimate generalization error for more than just the best performing model from the inner loop.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Should we estimate performance in the outer loop only for the best model?

yes. more precisely: in each of teh outer loop's folds, for the model chosen by its inner CV.

What if I estimate performance in the outer loop for the logistic regression model, the best LASSO model and best SVM model from the inner loop?

Then you know that these three modelling Ansätze perform similarly well for your data compared to the uncertainty / instability of the optimization.

This may mean that your optimization failed. OTOH, unlike optimizing model complexity where we expect one global optimum, there really isn't any reason why different modeling heuristics should not do similarly well on the same data. So it may just mean that it doesn't matter whether you do LR or SVM in terms of your figure of merit.

see also my answers to related questions: https://stats.stackexchange.com/a/245169/4598 and https://stats.stackexchange.com/a/65156/4598

Will then selecting one of these 3 'best' models for production based on the generalization error of the outer loop lead to bias?

yes

It is sometimes the case that if a more interpretable model such as a logistic model has only a slight decrease in performance compared to a more complex model, one might prefer the more interpretable model at the cost of some performance

You can build that into your selection heuristic in the inner loop.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Im a bit confused about one thing: A single final model and parameter estimates will be determined after running the inner loop approach on the full dataset in the end which may give us an SVM, LR or LASSO to be used in production. So it is confusing that the final generalization error reported can potentially be made up of entirely different modeling approaches (e.g., LR and SVM and LASSO), depending on which models won in the inner loops. In half, SVM may win, in half, LR may win and the error averaged across the outer loops will be based on entirely different modeling approaches. $\endgroup$
    – pauluccd927
    Nov 1, 2019 at 20:33
  • $\begingroup$ Yes. Each surrogate model may have a different hyperparameter set, and that includes the hyperparameter which model to use. It is up to you to decide whether such instability in the optimization results (which should ideally be caught already in the inner CV loop because that would be expected to be even more unstable) are a dealbreaker indicating that your optimization isn't to be trusted and you need to step back and improve your training heuristic or whether the seemingly different models are really equivalent and it doesn't matter wich one is chosen (e.g. LR and LDA producing the same ... $\endgroup$
    – cbeleites unhappy with SX
    Nov 3, 2019 at 18:35
  • $\begingroup$ ... decision boundary). $\endgroup$
    – cbeleites unhappy with SX
    Nov 3, 2019 at 18:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.