I have a simple model with 3 free parameters to fit, and wanted to try estimating those parameters via grid search. I was wondering -- what is the convention for selecting the best model in this case? I'm thinking it shouldn't be too complicated since the number of parameters are the same, so can I just use R^2? If not, would I need to select the parameters based on smallest MSE from some type of cross validation?

I suppose I don't have intuition about what to expect from cross validation when I have the same model structure, with different parameter estimates and correspondingly R^2 values. Assuming overfitting isn't an issue, would the model with the best R^2 always so better in cross validation?

  • $\begingroup$ Overfitting being an issue is one of the reasons for using cross-validation. Why do you not tune in cross-validation by looking at measures of accuracy on the validation sets? $\endgroup$ – Henry Dec 1 '19 at 22:29
  • $\begingroup$ I suppose it wouldn't hurt. But if I'm confident that overfitting isnt an issue, would a measure of accuracy via cross-validation give me an advantage over just selecting the model with the highest R^2? $\endgroup$ – John Alperto Dec 1 '19 at 22:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.