2
$\begingroup$

I am doing a grid search for model fits to data. At each point of the grid (let's say a 2d parameter space) I have a value for the statistic

$\chi^{2} = \sum_{i}\frac{(O_{i}-E_{i})^{2}}{dO_{i}^{2}}$

where $O_i$ are observed data points, $E_i$ the model values and $dO_i$ the (generally different) errors on the observed values.

How do I find a confidence interval (let's say the 68% interval) around the minimum value that I find?

To my understanding, standard $\chi^{2}$ only works if the errors are approximately Poissonian.

Improve this question
$\endgroup$
6
  • $\begingroup$ I think that the assumption is that $O_i$ is Poisson and $\chi^2=\sum_i \frac{(O_i-E_i)^2}{E_i}$? $\endgroup$ – user83346 May 29 '17 at 13:49
  • $\begingroup$ Right. But what if that condition is not met? I'm supposing one has to resort to sampling methods. $\endgroup$ – user1991 May 29 '17 at 14:26
  • $\begingroup$ Your formula for $\chi^2$ is very different from the one in my comment, the denominator is very ''strange'', it should be $E_i$. Usually, if the $E_i$ are all over 5 then the variable can be approximated well by $\chi^2$. $\endgroup$ – user83346 May 29 '17 at 16:19
  • $\begingroup$ If the denominator is replaced by $\mathrm{Var}(O_i)$ it would be more general. My question is whose confidence interval you want to get? $\endgroup$ – user158565 May 29 '17 at 18:18
  • $\begingroup$ Ultimately a confidence interval for the two estimated parameters. This should follow from a contour in the $\chi^{2}$-space as in the following paper, the only difference being that the errors in the paper are manifestly Poissonian and mine are not: articles.adsabs.harvard.edu/cgi-bin/… $\endgroup$ – user1991 May 30 '17 at 5:30
1
$\begingroup$

$\chi^2$ is a valid statistic for any data where you expect the errors to be normally distributed. It is commonly used for counting data, where you expect poisson errors of order $\sqrt{O_i}$ (which become gaussian in the large count limit), but as @a_statistician notes the more general formula is $$\chi^2 = \sum_i\frac{(O_i-E_i)^2}{Var(O_i)}$$.

Restating your problem, it sounds like you have some model parameters, let's call them $\alpha$ and $\beta$, which are used to predict $E_i(\alpha,\beta)$, and some observed values $O_i$, along with some prediction about the size of errors, $Var(O_i)$. Therefore you can calculate $\chi^2(\alpha,\beta)$ as a function of $\alpha$ and $\beta$.

You have then (correctly) estimated $\alpha$ and $\beta$ by minimizing this function, and now want to obtain the confidence intervals for these estimates. As the paper you linked explains, one way to do this is to consider the parameters themselves to be free, and lookup to corresponding cumulative-probability function for a chi-squared distribution with that number of degrees of freedom (using one of the many handy tables available online).

For the example you gave, with two parameters, the 95% cumulative probability point is 5.991 (corresponding to (2,0.05) on the linked table). Therefore, one would call the 95% confidence region the set of all points $(\alpha, \beta)$ in parameter space such that

$$\chi^2(\alpha,\beta) - \chi^2_{min} < 5.991$$

For a good, well-constrained model, this should give you something fairly nice, like an ellipse. This ellipse corresponds to the covariance matrix for your parameters, as this article nicely explains, and therefore standard errors and other commonly used measures of uncertainty.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.