I have two models that predict a continuous outcome using the same 4 predictor variables.
Currently I am using the RMSE to compare the models so I am comparing the predicted outcome vs. the actual outcome for both models.
Can you advise on this method of model comparison? Are there more popular methods besides RMSe to compare linear vs. bayesian models. I like RMSE because it is interpretable for a lay audience.
Thank you.
Of course (R)MSE is a good metric for evaluating predictive models. If you are interested in how the models would do predicting out-of-sample cases, then I would recommend K-fold cross-validation to estimate out-of-sample MSE.
If one model is a penalized regression model, then it wouldn't be surprising to see it have higher MSE when predicting in-sample and lower MSE out-of-sample than the non-penalized model due to the non-penalized model over-fitting the data.
