Goodness of fit --$\chi^2$ Statistics I have an elementary question about $\chi^2$ statistic.  Let us say that an experiment
makes a set of measurements $E_i \pm \delta_i$, with $i=1, \ldots, n$. $\delta_i$ are the
uncertainties in experimental measurements. Let us assume that a theory predicts  the values to be $T_i$, again with $i=1, \ldots, n$.  We need to determine if the theory is a good fit for the experimental data.
Does one regard the $E_i-T_i \sim N(0, \delta_i^2)$?  That is, are the errors $E_i-T_i$ assumed to be normally distributed with variance $\delta_i^2$? If the answer is yes, then assuming that the errors are independent, we can take $\sum_{i=1}^n \frac{(E_i-T_i)^2}{\delta_i^2}$
to be a $\chi^2$ random variable with $n$ degrees of freedom.  My question is,
what is the justification for taking $\delta_i$ to be the estimator of $\sigma_i$, the actual standard deviation of the error variable $E_i-T_i$?
Independently, what is the widely used method for gauging the goodness of fit in the above scenario?
 A: There is, of course, no way to say for certain that the $E_i - T_i$ are normally distributed (they could theoretically obey any probability distribution, subject to the experimental setup and underlying theory). However, due to the Central Limit Theorem and many uncontrolled random fluctuations present in any given measurement, it is often a very good approximation to treat them as normally distributed.
With this concession, then yes you are correct. You may treat $\sum_{i=1}^n (E_i-T_i)^2/\delta_i^2$ to be a $\chi^2$ random variable with $n$ degrees of freedom. But note that the $\chi^2$ distribution is defined as the sum of $n$ statistically independent normalized Gaussians, so additionally you must make sure your measurements $E_i$ are independent from one another. Of course, you must also ensure 0 systematic error for your measurements.
One potentially relevant empirical fact that is worth noting is that in nature, distributions tend to have larger tails than predicted by treating everything as a Gaussian. That is, the probability of outliers is higher than predicted by the Central Limit Theorem. Specifically, distributions tend to be closer to log-normal distributions.
But again, note that basically all of statistics operates under the assumption that everything is a Gaussian, so don't get too worried about such technicalities.
As for a standard way to measure the goodness of fit, either the goodness of fit parameter Q (discussed in my reference below; it's probably on Wikipedia somewhere but I couldn't easily find it), or the $\chi^2$ per degree of freedom are reasonable metrics.
I learned all of this from Everything you wanted to know about Data Analysis and Fitting but were afraid to ask, by Peter Young. It's admittedly slightly skewed toward techniques in Lattice QCD (namely bootstrap and jackknife), but it also answers all the questions you had above in a pedagogical manner.
