Let's say Im trying to do a bayesian linear regression and I have some strong prior on the parameters x1,...,xn.

However, after collecting my data I can clearly see the residuals are not normally distributed, and thus can't calculate the likelihood correctly.

What should I do? Use a transformation?



So you are confusing two sources of randomness in Bayesian linear regression. (1) Randomness over the parameter space, and (2) Randomness in the input noise

Assuming your aim is to model something like $Y = \beta X + \varepsilon$, you can have $\varepsilon \sim \mathcal{N}(0,\sigma^2)$, so the noise is normally distribution, and then also $\beta \sim \mathcal{N}(\mu_{\beta},\sigma_{\beta}^2)$, so then the parameters are distributed according to a normal distribution also.

Thus your prior over your parameters $p(\beta)$, and your distribution of residuals $p(\varepsilon)$ are two different and independent modelling assumptions, and the choice of one model, in most cases does not effect your choice of the other model.

Now, onto the other part of your question. In the framework of Bayesian linear regression, it is possible to for your noise to look like any distribution you want. If your noise is not normal you can reflect that in $p(\varepsilon)$.

However, in general a normality assumption in the noise (and parameters) just makes things much easier to calculate, and evaluate in post processing. It is a very well-understood system. So much so, that in general if your noise does not look Gaussian it would be advisable to apply a transformation to it so it appears Gaussian. Look at logarithmic transformations, and Box Cox transformations for this.

I strongly looking at the following post also, in regards to noise:

Regression when the OLS residuals are not normally distributed


| cite | improve this answer | |
  • $\begingroup$ Thanks for your answer! But my problem is that if I decide the p(epsilon) after looking at the data, I can't choose "unbiased" prior, and the ones that I did choose beforehand are no longer relevant (because the residuals are distributed differently). What should I do? $\endgroup$ – StavO Aug 28 '19 at 17:26
  • $\begingroup$ You should be normally deciding your error model by looking at the data. Looking at the data to help modeling does not introduce "bias". Bias is to do with how underfit your model is. If you say that new input data has a different looking residual then either (1) You should be training your regression on more data i.e. your original dataset was not a complete representation of your populatoin, or (2) Train another regression model on the new system. $\endgroup$ – tisPrimeTime Aug 28 '19 at 17:34
  • $\begingroup$ Also if you don't mind, you can accept my answer as the "accepted ansewr" by clicking on the tick please =) $\endgroup$ – tisPrimeTime Aug 28 '19 at 17:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.