I have a nonlinear physical model for which I'm trying to determine parameter uncertainties using Monte Carlo. Instead of describing the nitty-gritty details, I will use a series of figures:

enter image description here

enter image description here


The first image is the experimental data; the second is the best fit simulation. To the eye, they seem to match quite well. But this impression vanishes with one looks at the residuals (shown in the third image). They are clearly correlated, and their magnitude dwarfs the genuine white noise that is present. This leads to a $\chi^2_\text{red}$, with 60 degrees of freedom (64 bins, 3 model parameters), of 86.5.

The issue here seems to be that the experimental data was acquired with such high signal to noise that features of the full physical model that are normally ignorable in the simplified three-parameter version of the model I'm using are now introducing systematic bias. Additionally, the uncertainties that have been determined by the Monte Carlo routine are laughably minuscule. In other words,

  • High S/N --> Low uncertainties reported; CI about best fit parameters do not overlap their true values.

  • Low S/N --> High uncertainties reported; CI's do overlap true value, even though the reported best fit value may in fact be farther away from the true value than in the high S/N case!

    Ironically, the latter scenario seems preferable!

In retrospect, all this should have been obvious: correlated residuals indicate model failure. It's like trying to report the uncertainty on the slope of a line you fit through data obeying a power law! Nevertheless, my current model captures all the important big picture stuff. So is there still hope in determining realistic uncertainties for my parameters? I'm thinking about adding white noise to the spectrum and redoing the fit such that the best fit produces a $\chi^2_\text{red}$ of about 1 or 2, but that just seems so perverse, lack of rigor notwithstanding.

  • $\begingroup$ The two votes tells me this is at least interesting to some, so I'm going to bump. So as to not leave a completely vapid comment, let me add that as expected, adding white noise to the spectrum increased the size of the CI's. Example: For one parameter, the 95% CI went from (222.00,222.22) to (220.11,224.16). I have strong reason to believe from other data that the true value of this parameter is around 220. $\endgroup$ – AFineTransform May 13 '13 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.