2
$\begingroup$

In my work I'm trying to fit a multinomial logistic regression with the objective of prediction. I am currently applying cross validation with Repeated Stratified K Folds but I still have some questions about the method I haven't seen answered before.

Does it make sense to use cross validation to test the regression, in this case where I am not tuning any hyperparameters? I've seen a lot that cross val is most useful for hyperparameter tuning.

I ran my model (regression with the same predictors) with 10 folds repeated 3 times, and I get really good metrics in each fold (ROC of 0.95, micro average precision-recall of 0.94, and more along those lines), which suggest my model is discriminating appropriately and capable of predicting well. Can I be confident that my regression is not overfitting? That is, that the variables I selected to run as predictors would not overfit the data.

Finally, I am not sure if I can technically end my analysis there, or I can then make a "final model" with all the same predictors and trained in a larger part of (if not all) the data. I assume if the company wants to actually run this model they will need a "final fit" to predict over, right? Should I use another train-test split for this final model?

Your help is very much appreciated!

$\endgroup$
1
$\begingroup$

Cross validation can be used for many tasks: hyperparameter tunning, how stable your out of sample error is, but I would say that it is most useful for comparing different models.

For example, if you have two models, and you run cross validation on both of them, then you can compare the performance of different folds and see if one model outperforms the other. By doing this, say, 10-fold, you get a more robust estimate of the out of sample performance compared to only using one test set (i.e. 1-fold validation).

You might find that a more complex model is able to get an average AUC of 0.97, or maybe if overfits and give you a worse AUC of 0.9. You are only able to say if a model overfits if you actually compare it out of sample with a simpler model.

For your last question: After you have found the best model doing cross-validation, and you have decided that this model is going to be used in production, you should train the model on all data available, so that you get the most accurate estimates possible.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ thank you for your answer! Okay, I did actually try different models on different cross validation stages (eg. one where the variables where chosen with PCA, one where they were centered/standarized, one where the predictors were chosen by intuition). So as long as I use cross validation, I need to be aiming for the highest metrics possible in all folds and not worry about overfitting, right? $\endgroup$ – amestrian Aug 18 at 10:06
  • $\begingroup$ Oh, this phrase keeps ringing "You are only able to say if a model overfits if you actually compare it out of sample with a simpler model." how do you know when a model is "simpler" necessarily? What if this is my benchmark model to do some fancier stuff later? $\endgroup$ – amestrian Aug 18 at 10:07
  • $\begingroup$ Of course, all your models can be overfitting, so make sure you have a simple baseline model. A simpler model can have fewer covariates, only linear terms (instead of splines, higher polynomials etc). The simplest model could also just be an intercept model. If you do more "fancy stuff" later, then this should be part of the cross-validation process. $\endgroup$ – J.C.Wahl Aug 18 at 11:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.