Should my residuals have the same distribution as my likelihood function in a Bayesian linear regression?

Question

If I have a standard linear regression model: $$ y \sim \mathcal{N}(\beta_0 + \beta_1X + ..., \sigma) $$ With whatever priors on $\beta$'s and $\sigma$. Does that imply that my residuals $r = y - (\beta_0 + \beta_1X + ...)$ should be $r \sim \mathcal{N}(0, \sigma)$? I know that in a traditional least-squares setting, this is a common check. Is it relevant in a Bayesian setting? Further, in traditional least-squares, not meeting the check means prediction intervals would be off as well as confidence intervals for parameters. I can understand how the prediction interval issue would be relevant in the Bayesian context as well, but does the parameter issue apply as well? Are the posterior estimates for the parameters potentially off because my data doesn't match my assumed likelihood distribution?

Ultimately, I'm curious about the relationship between residuals and the likelihood function in a Bayesian context.

Answer 1

The probabilistic model for linear regression is

$$\begin{align} \mu &= \beta_0 + \beta_1 X_1 + \dots + \beta_k X_k \\ y &\sim \mathcal{N}(\mu, \sigma) \end{align}$$

which is the same as saying

$$ y - \mu = \varepsilon \sim \mathcal{N}(0, \sigma) $$

This follows from the properties of a normal distribution if $Y \sim \mathcal{N}(\mu, \sigma)$, then $Y + c$ would follow a normal distribution with mean $\mu + c$.

Notice that this does not hold for all distributions. For example, the difference between $y$ and the mean predicted by Poisson regression could result in negative values of the residuals, so they cannot follow a Poisson distribution that has non-negative support. The same is tue with logistic regression: if you subtract the predicted mean (probabilities) from zeros and ones, the residuals could not be zeros or ones, so won't follow the Bernoulli distribution assumed by logistic regression. As you can see, there is no one-to-one correspondence between the distribution of residuals and the likelihood function. Here you can read about the distribution of residuals for logistic regression. For models other than linear regression, the distribution of residuals is not equally elegant.

But yes, if your data is inconsistent with the distribution you assumed by the model, then your results will be flawed. The estimates of the parameters may be invalid; the intervals for the parameters and predictions would be incorrect as well; if you conducted hypothesis testing, tests would not work properly. This would be true for both Bayesian and frequentist models.

Answer 2

In principle what Tim says; however, there are two subtleties

a) Given that your parameter estimates are distributions, what's your definition of "residual"? To get a "point" residual, you could take the model predictions at the MAP or the posterior mean, or else you evaluate the entire posterior (then, your residual is a distribution though). Alternatively, you can calculate quantile residuals from posterior predictive simulations, which creates a point-residual accounting for the posterior uncertainty.

b) The second point is the influence of the priors - if priors dominate, you could force your model to have pretty much any residual distribution.

See comments in https://cran.r-project.org/web/packages/DHARMa/vignettes/DHARMaForBayesians.html