Bayesian inference of parameters: residuals are independent but not normally distributed

Question

I would like to compute belief intervals (confidence intervals; CI) for the parameters of an environmental dynamic model within the Bayes' theorem. The measurement model of the data is $$ d(t)=y(\theta,t)+\epsilon, $$ where $\epsilon$~Norm(0, $\sigma$) is the assumed measurement error, $y$ is the dynamic system output dependent on parameter $\theta$ and $d(t)$ is the observed data at time t. Because the measurment error is assumed normally distributed, the likelihood function is a Gaussian distribution. The prior for $\theta$ is non-informative (i.e. uniform distribution). The aim is to compare CI estimated from Bayes’ formula and the classical linear approximation (covariance matrix estimated from the Hessian inversion): I would like to maintain the “least-square” objective function in order to perform this comparison.

The measurement model assumes that errors are i.i.d. (independent and identically distributed). What is the meaning of the CI if the residuals are not normally distributed but are still "independent" (without auto-correlation)? In my case the distribution of residuals has long tails (e.g. Laplace distribution).

In the reference book of Jaynes et al. (2003), pp.592–593, it is said that:

Measurement errors in physical experiments are often modeled by a normal distribution. This use of a normal distribution does not imply that one is assuming the measurement errors are normally distributed; rather using the normal distribution produces the most conservative predictions possible given only knowledge about the mean and variance of the errors.

I understand that Jaynes' statement refers to the prediction uncertainty. What about CI of $\theta$, apart of $\sigma$? Should $CI_{\theta}$ be considered overestimated and conservative? Or the opposite since we do not consider a statistical model for residuals with long tails (i.e. the likelihood is proportional to the posterior)? Is the CI estimation totally unreliable because normality of residuals is not guaranteed or is it "unreliable" only to the “degree” the normality assumption is violated?

Answer 1

Thanks for a long and frank explanation of your worries! Having non-normal errors is not an issue if you incorporate the true distribution of the errors in your Bayesian calculation. If you mis-specify and use another family of distributions than the "true" one, you have to assess the impact of this error either analytically or by simulation. I am quite skeptical of the "most conservative" label endowed by Jaynes on the normal distribution. I would think it applies only to the case when you replace a normal error with a $t$ error, since the normal distribution had lighter tails than any $t$ distribution. But if the error can be double-exponential or anything else with even lighter tails, I do not see why the normal should be "most conservative".

Answer 2

I think one needs to distingish between measurement uncertainty and the variability of the process generating the data. For instance convective rainfall can be modelled as having a gamma distribution for the amount that it rains (when it does rain). This uncertainty has nothing to do with the measurements of any quantity, but is caused by the chaotic nature of the physics involved. So if you have some knowledge of the data generating process to suggest that the variability around the mean will be long-tailed, then you should use a long-tailed distribution (e.g. Laplace or t) instead of a Gaussian.

By "most conservative" I think Jaynes simply means that if all you know about the variability is its mean and variance, then the maximum entropy choice of distribution is the Gaussian. If you know more about the noise process, e.g. for convective rainfall), then you don't need to fall back on maxent to justify the choice of distribution.