In my work I'm trying to fit a multinomial logistic regression with the objective of prediction. I am currently applying cross validation with Repeated Stratified K Folds but I still have some questions about the method I haven't seen answered before.
Does it make sense to use cross validation to test the regression, in this case where I am not tuning any hyperparameters? I've seen a lot that cross val is most useful for hyperparameter tuning.
I ran my model (regression with the same predictors) with 10 folds repeated 3 times, and I get really good metrics in each fold (ROC of 0.95, micro average precision-recall of 0.94, and more along those lines), which suggest my model is discriminating appropriately and capable of predicting well. Can I be confident that my regression is not overfitting? That is, that the variables I selected to run as predictors would not overfit the data.
Finally, I am not sure if I can technically end my analysis there, or I can then make a "final model" with all the same predictors and trained in a larger part of (if not all) the data. I assume if the company wants to actually run this model they will need a "final fit" to predict over, right? Should I use another train-test split for this final model?
Your help is very much appreciated!