I am working on a regression model to predict a target variable in a dataset with over 100 features. Three different regression models are defined and fit in order to compare their performance using $R^2$ and the errors: MAE, MSE, RMSE. E.g.:

R^2: 0.89
MSE: 0.13
RMSE: 0.3

I am working with Pyro and following their documentation where they evaluate the Bayesian model by calculating the posterior on predictive samples; but am not sure I understand how one would compare both models using a metric.

AFAIK it does not make sense to calculate an $R^2$ score in a Bayesian model so the questions are:

  1. Is it possible to compare non-Bayesian vs. Bayesian regression models using the same metric?
  2. What is the best way to do that?
  • 1
    $\begingroup$ Welcome to the site. Questions about coding (as your second question) are off topic here. If you drop that, then I think you have a good question for this site. $\endgroup$ – Peter Flom Feb 26 at 12:11
  • $\begingroup$ Please register your account (you can find information on how to do this in the My Account section of our help center), then you will be able to edit & comment on your own question. $\endgroup$ – gung Feb 26 at 19:20
  • 1
    $\begingroup$ I edited your question for clarity. By "regressor", you seem to mean "regression model" (a regressor is a feature in a regression model). Please ensure that it reflects what you want to ask. $\endgroup$ – gung Feb 26 at 19:25
  • $\begingroup$ You can use an information criterion like AIC to compare the models. $\endgroup$ – Alex Jul 2 at 2:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.