I am working on a regression model to predict a target variable in a dataset with over 100 features. Three different regression models are defined and fit in order to compare their performance using $R^2$ and the errors: MAE, MSE, RMSE. E.g.:
R^2: 0.89 MSE: 0.13 RMSE: 0.3
I am working with Pyro and following their documentation where they evaluate the Bayesian model by calculating the posterior on predictive samples; but am not sure I understand how one would compare both models using a metric.
AFAIK it does not make sense to calculate an $R^2$ score in a Bayesian model so the questions are:
- Is it possible to compare non-Bayesian vs. Bayesian regression models using the same metric?
- What is the best way to do that?