I would like to compute belief intervals (confidence intervals; CI) for the parameters of an environmental dynamic model within the Bayes' theorem. The measurement model of the data is $$ d(t)=y(\theta,t)+\epsilon, $$ where $\epsilon$~Norm(0, $\sigma$) is the assumed measurement error, $y$ is the dynamic system output dependent on parameter $\theta$ and $d(t)$ is the observed data at time t. Because the measurment error is assumed normally distributed, the likelihood function is a Gaussian distribution. The prior for $\theta$ is non-informative (i.e. uniform distribution). The aim is to compare CI estimated from Bayes’ formula and the classical linear approximation (covariance matrix estimated from the Hessian inversion): I would like to maintain the “least-square” objective function in order to perform this comparison.
The measurement model assumes that errors are i.i.d. (independent and identically distributed). What is the meaning of the CI if the residuals are not normally distributed but are still "independent" (without auto-correlation)? In my case the distribution of residuals has long tails (e.g. Laplace distribution).
In the reference book of Jaynes et al. (2003), pp.592–593, it is said that:
Measurement errors in physical experiments are often modeled by a normal distribution. This use of a normal distribution does not imply that one is assuming the measurement errors are normally distributed; rather using the normal distribution produces the most conservative predictions possible given only knowledge about the mean and variance of the errors.
I understand that Jaynes' statement refers to the prediction uncertainty. What about CI of $\theta$, apart of $\sigma$? Should $CI_{\theta}$ be considered overestimated and conservative? Or the opposite since we do not consider a statistical model for residuals with long tails (i.e. the likelihood is proportional to the posterior)? Is the CI estimation totally unreliable because normality of residuals is not guaranteed or is it "unreliable" only to the “degree” the normality assumption is violated?