# Bayesian inference of parameters: residuals are independent but not normally distributed

I would like to compute belief intervals (confidence intervals; CI) for the parameters of an environmental dynamic model within the Bayes' theorem. The measurement model of the data is $$d(t)=y(\theta,t)+\epsilon,$$ where $\epsilon$~Norm(0, $\sigma$) is the assumed measurement error, $y$ is the dynamic system output dependent on parameter $\theta$ and $d(t)$ is the observed data at time t. Because the measurment error is assumed normally distributed, the likelihood function is a Gaussian distribution. The prior for $\theta$ is non-informative (i.e. uniform distribution). The aim is to compare CI estimated from Bayes’ formula and the classical linear approximation (covariance matrix estimated from the Hessian inversion): I would like to maintain the “least-square” objective function in order to perform this comparison.

The measurement model assumes that errors are i.i.d. (independent and identically distributed). What is the meaning of the CI if the residuals are not normally distributed but are still "independent" (without auto-correlation)? In my case the distribution of residuals has long tails (e.g. Laplace distribution).

In the reference book of Jaynes et al. (2003), pp.592–593, it is said that:

Measurement errors in physical experiments are often modeled by a normal distribution. This use of a normal distribution does not imply that one is assuming the measurement errors are normally distributed; rather using the normal distribution produces the most conservative predictions possible given only knowledge about the mean and variance of the errors.

I understand that Jaynes' statement refers to the prediction uncertainty. What about CI of $\theta$, apart of $\sigma$? Should $CI_{\theta}$ be considered overestimated and conservative? Or the opposite since we do not consider a statistical model for residuals with long tails (i.e. the likelihood is proportional to the posterior)? Is the CI estimation totally unreliable because normality of residuals is not guaranteed or is it "unreliable" only to the “degree” the normality assumption is violated?

Thanks for a long and frank explanation of your worries! Having non-normal errors is not an issue if you incorporate the true distribution of the errors in your Bayesian calculation. If you mis-specify and use another family of distributions than the "true" one, you have to assess the impact of this error either analytically or by simulation. I am quite skeptical of the "most conservative" label endowed by Jaynes on the normal distribution. I would think it applies only to the case when you replace a normal error with a $t$ error, since the normal distribution had lighter tails than any $t$ distribution. But if the error can be double-exponential or anything else with even lighter tails, I do not see why the normal should be "most conservative".

• What do you mean by "assess the impact of this error either analytically or by simulation"? Should I compare the posteriors under different statistical models for the residuals? For example when using a normal or a t-Student (or any other)? – Zivko Juznic-Zonta Jan 30 '12 at 13:50
• ...Or should I follow some authors that estimate additional hyper-parameters associated with some “flexible-enough” distribution that can fit the errors’ histogram? How can I reply to a peer-reviewer that questions my CI evaluation based on the fact that the errors are non-normal? It is difficult to replay that I have just used the "true" distribution of the errors. – Zivko Juznic-Zonta Jan 30 '12 at 14:14
• @ZivkoJuznic-Zonta: sorry, it was not clear from your original question. Now, if you cannot fix a specific family of distributions from knowledge about the problem at hand, you should indeed demonstrate by simulation that the fact that the errors are not normal is not very influential on the resulting CIs if the errors are from another family, e.g. $t$. Or Laplace. – Xi'an Jan 30 '12 at 18:03

I think one needs to distingish between measurement uncertainty and the variability of the process generating the data. For instance convective rainfall can be modelled as having a gamma distribution for the amount that it rains (when it does rain). This uncertainty has nothing to do with the measurements of any quantity, but is caused by the chaotic nature of the physics involved. So if you have some knowledge of the data generating process to suggest that the variability around the mean will be long-tailed, then you should use a long-tailed distribution (e.g. Laplace or t) instead of a Gaussian.

By "most conservative" I think Jaynes simply means that if all you know about the variability is its mean and variance, then the maximum entropy choice of distribution is the Gaussian. If you know more about the noise process, e.g. for convective rainfall), then you don't need to fall back on maxent to justify the choice of distribution.

• Lets say that a priori I have no knowledge over the noise process. So I just assume the Gaussian distribution (maxent). After "calibration", I realize that the errors are not normally distributed. They seems like-t/Laplace distributed. Is it fair to repeat the the calibration, assuming that this time the noise model is another (only for this particular case "better")? The "a priori" information becomes conditional on the posterior result, which seems to me like cheating since from the begging it was clear that I do not know the "true" error model. – Zivko Juznic-Zonta Jan 30 '12 at 14:25
• One thing you could do would be to compare the evidence for each model, using e.g. Bayes factor. I would say that would provide sufficient justification if the Bayes Factor were non-negligible. While it may seem impure to re-do the analysis in hindsight, it is also wrong to use a model where you have good reason to suspect the model assumptions are wrong. As an engineer I would suggest that there will always be a point where a degree of pragmatism is required as no framework is bullettproof. – Dikran Marsupial Jan 30 '12 at 15:53
• I agree with Dikran Marsupial that you cannot be that of a purist! You actually start your analysis by considering that the normal distribution is not the "true" one. The idea of using a Bayes factor is then relevant: you could run in parallel similar analyses for different families and let the Bayes factor pick the most appropriate for your data. That you cannot run this for all possible families and that you only consider the most likely cases is a constraint of time and pragmatism. No sin of using the data twice there! – Xi'an Jan 31 '12 at 13:52
• I think that the problem with my view is that I'm working with a "causal" structural equation model and thus, I imagine that the measurement model should be coherent with some phenomenological explanation too. I like the Dikran Marsupial's proposal to use Bayes’ factor in order to decide which measurement model is more appropriate. Unfortunately, I have no experience on how to estimate this factor from MCMC samples that I have at hand. As proposed by Xi'an, I will try to verify if posteriors estimated within different/reasonable measurement models are approximately the same. – Zivko Juznic-Zonta Jan 31 '12 at 18:04