4
$\begingroup$

I am doing a grid search for model fits to data. At each point of the grid (let's say a 2d parameter space) I have a value for the statistic

$\chi^{2} = \sum_{i}\frac{(O_{i}-E_{i})^{2}}{dO_{i}^{2}}$

where $O_i$ are observed data points, $E_i$ the model values and $dO_i$ the (generally different) errors on the observed values.

How do I find a confidence interval (let's say the 68% interval) around the minimum value that I find?

To my understanding, standard $\chi^{2}$ only works if the errors are approximately Poissonian.

$\endgroup$
6
  • $\begingroup$ I think that the assumption is that $O_i$ is Poisson and $\chi^2=\sum_i \frac{(O_i-E_i)^2}{E_i}$? $\endgroup$
    – user83346
    May 29, 2017 at 13:49
  • $\begingroup$ Right. But what if that condition is not met? I'm supposing one has to resort to sampling methods. $\endgroup$
    – user1991
    May 29, 2017 at 14:26
  • $\begingroup$ Your formula for $\chi^2$ is very different from the one in my comment, the denominator is very ''strange'', it should be $E_i$. Usually, if the $E_i$ are all over 5 then the variable can be approximated well by $\chi^2$. $\endgroup$
    – user83346
    May 29, 2017 at 16:19
  • $\begingroup$ If the denominator is replaced by $\mathrm{Var}(O_i)$ it would be more general. My question is whose confidence interval you want to get? $\endgroup$
    – user158565
    May 29, 2017 at 18:18
  • $\begingroup$ Ultimately a confidence interval for the two estimated parameters. This should follow from a contour in the $\chi^{2}$-space as in the following paper, the only difference being that the errors in the paper are manifestly Poissonian and mine are not: articles.adsabs.harvard.edu/cgi-bin/… $\endgroup$
    – user1991
    May 30, 2017 at 5:30

1 Answer 1

4
$\begingroup$

$\chi^2$ is a valid statistic for any data where you expect the errors to be normally distributed. It is commonly used for counting data, where you expect poisson errors of order $\sqrt{O_i}$ (which become gaussian in the large count limit), but as @a_statistician notes the more general formula is $$\chi^2 = \sum_i\frac{(O_i-E_i)^2}{Var(O_i)}$$.

Restating your problem, it sounds like you have some model parameters, let's call them $\alpha$ and $\beta$, which are used to predict $E_i(\alpha,\beta)$, and some observed values $O_i$, along with some prediction about the size of errors, $Var(O_i)$. Therefore you can calculate $\chi^2(\alpha,\beta)$ as a function of $\alpha$ and $\beta$.

You have then (correctly) estimated $\alpha$ and $\beta$ by minimizing this function, and now want to obtain the confidence intervals for these estimates. As the paper you linked explains, one way to do this is to consider the parameters themselves to be free, and lookup to corresponding cumulative-probability function for a chi-squared distribution with that number of degrees of freedom (using one of the many handy tables available online).

For the example you gave, with two parameters, the 95% cumulative probability point is 5.991 (corresponding to (2,0.05) on the linked table). Therefore, one would call the 95% confidence region the set of all points $(\alpha, \beta)$ in parameter space such that

$$\chi^2(\alpha,\beta) - \chi^2_{min} < 5.991$$

For a good, well-constrained model, this should give you something fairly nice, like an ellipse. This ellipse corresponds to the covariance matrix for your parameters, as this article nicely explains, and therefore standard errors and other commonly used measures of uncertainty.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.