How to compare predictions from MLE-based regression Vs. predictions from bayesian regression?

Question

Say I have two linear regression models that I want to use for predictions.

Linear regression: \begin{equation} \mathbf{y} \sim \mathcal{N}(\mathbf{X^Tb}, \Sigma_y) \end{equation}

Bayesian linear regression: \begin{align} \mathbf{y | b} &\sim \mathcal{N}(\mathbf{X^Tb}, \Sigma_y) \notag\\ \mathbf{b} &\sim \mathcal{N}(0, \Sigma_b) \end{align}

Predictions

For the linear regression, I find the coefficients by MLE: \begin{equation} \mathbf{b}_{\text{MLE}} = \mathbf{(X X^T)^{-1} X y} \end{equation}

The predictions for the the test set are: \begin{equation} \mathbf{y_{test}} = \mathbf{X_{test}^T b_\text{MLE}} \end{equation}

For the bayesian linear regression, I find the predictive posterior: \begin{equation} p(\mathbf{y_{test}} | \mathbf{X_{test}} , \mathbf{y}, \mathbf{b}) = \int p(\mathbf{y_{test}} | \mathbf{X_{test}}, \mathbf{b}) p(\mathbf{b} |\mathbf{y}) \text{d}\mathbf{b} \end{equation} (Thought this integral could be analytically derived, I find it by Gibbs sampling since in my actual model the different $b_i$ come from a mixture of gaussians.)

The predictions are plotted here, where for the bayesian prediction I also plotted the variance of every $y$ (computed from the traces of the Gibbs sampler)

Questions

• How should I compare the two models? Should I use $R^2$ (and then all the bayesian machinery is for nothing?)

• (forgetting about MLE) How to evaluate the performance of the bayesian estimators? I've found a lot about model selection, but what about the accuracy of a single model?

Answer 1

How should I compare the two models?

Form a Bayesian point of view you do not have two models to compare: MLE is just a single 'best' guess. (Instead of 'MAP' I would you the expectation of the model prediction; because MAP also is only a 'best' guess.) To compare the two models in a Bayesian framework, you would need their evidences - which is essentially zero for the MLE 'model'.

How to evaluate the performance of the bayesian estimators? I've found a lot about model selection, but what about the accuracy of a single model?

From a Bayesian viewpoint it makes only sense to compare a model if you have alternative competing models. The 'root means square deviation' (RMSD) is a measure I would refrain from using (the only exception being if everything is nice and Gaussian). By definition, the likelihood defines how your model relates to the data. How can you justify to switch to the RMSD to assess the 'quality of your model'?

Of course, you can use the tools mentioned. However, you should not call it Bayesian then.

Answer 2

How should I compare the two models?

Rather than think of these two approaches as two distinct "models", I suggest you look at the ML and MAP results as estimates (or predictions) derived from models that are characterizing randomness/uncertainty (depending on the perspective). Consider this: the ML estimates you are using for the points $\mathbf{y}_\text{test}$ have a sampling distribution, the expectation of which is what you have taken as your prediction for $\mathbf{y}_\text{test}$; this is just $\mathbf{X}_\text{test}^T \mathbf{b}_\text{MLE}$. Observe that you can construct a confidence interval with this estimate/prediction using frequentist theory. This characterization of randomness is analogous to the Bayesian characterization of uncertainty in the posterior predictive. For example, in your posted plot, you could have also given the blue line "error bars" to denote your characterization of randomness as it relates to your estimates of $\mathbf{y}_\text{test}$.

So in that sense, your plot is playing favorites: it did not present the frequentist measure of "variance" in the MLE predictions. It's not as if the Bayesian approach to regression offers substantially useful information that the ML approach does not.

Should I use $R^2$ (and then all the bayesian machinery is for nothing?)

You can use $R^2$ to compare two (point estimate) predictions, which you did for the MLE and MAP predictions. There are pros/cons and caveats to using that particular metric which have been answered in other questions on this site.

(forgetting about MLE) How to evaluate the performance of the bayesian estimators? I've found a lot about model selection, but what about the accuracy of a single model?

You can specify a loss function that has some valuable meaning and then calculate it. For example, for a set of actual values $\{y_i\}_{i=1}^n$ and a set of estimators $\{\hat y_i\}_{i=1}^n$, the root mean square deviation (RMSD; a.k.a. RMSE) is given by $$ \text{RMSD}(y_{i=1,\dots,n}, \hat y_{i=1,\dots,n}) = \sqrt{\frac{1}{n} \sum_{i=1}^n (y_i - \hat y_i)^2} $$ and is equivalent to the sample standard deviation of the difference between the estimator and its actual value --- which has a particularly useful meaning.

This approach of evaluating loss functions on estimators rings well with formal Bayesian theory, where the choice of which specific estimator to use can only be determined once a loss function, which we are wanting to minimize, has been specified. For example, the posterior mean estimate is the Bayesian estimator implied when we specify squared error loss. (Although, in practice, we often choose which estimator to use and then justify the loss function implying that estimator as needed..)