Comparison of two error distributions to determine "goodness of fit" I am a physicist who is a few years out of doing his last course in statistics, so I am hoping to get some advice when comparing some data I recently generated. 
The context is as follows. I have two slightly different theoretical models which I used to generate two sets of data (the numbers are of radiative strengths in multi-electron atoms for various transitions, but not exactly relevant for the mathematics). I also have a corresponding set of "accepted" or "literature" values for these transitions, and I would like to see which of my two models produces results that are closer overall/on average to the "accepted" model. 
My current metric of comparison is relative error for each transition (i.e. $ \delta_{rel} = \frac{|T_{i,j}-A_j|}{A_j} $ where $T_{i,j}$ is my theoretical result for transition $j$ and theoretical model $i$ and $A_j$ is the accepted result for transition $j$). Seemed like the most obvious place to start. I am now stuck as to how to get some sort of meaning from these relative errors. Currently, I have just taken the arithmetic mean of the relative errors, but my intuition tells me this isn't a rigorous or correct way about this. Would a difference of mean, t-test be appropriate? My hypothesis testing knowledge is a bit rusty, so any guidance here would be appreciated.
This seems like it should be a fairly basic question, but I haven't been able to find anything reasonable through my own searches.
 A: First of all, in my opinion, you are defining the relative error incorrectly.  Presumably, given the way that you have posed the question, the $A_{j}$ are experimentally measured values; as such, each $A_{j}$ should have a published error estimate $\sigma_{j}$ associated with it.  This represents the experimenter's best estimate of the one sigma uncertainty in his result (i.e., the magnitude of one "standard deviation"), under the assumption that the uncertainty is Gaussian distributed, which is frequently a good assumption in laboratory physics experiments, because the experimental uncertainties often arise from a combination of many independent random factors, leading to a situation where the key assumptions of the Central Limit Theorem are valid.  If for some reason your literature doesn't list the $\sigma_{j}$ values, or if the experimenter has instead chosen to quantify his experimental uncertainty using some alternative measure, such as a 90% central confidence limit or something like that, then you may stop reading right here, as the rest of what I have to say won't be useful to you.
Anyway, assuming you do have the experimental uncertainty $\sigma_{j}$ associated with each measured transition $A_{j}$, then you may define a revised relative error $$\delta_{ij} = \frac{|T_{ij}-A_{j}|}{\sigma_{j}}$$  Once you've calculated all of the $\delta_{ij}$ values, you may use a variant of Pearson's $\chi^{2}$ test to establish goodness-of-fit between the data and the two alternative theoretical models $T_{1}$ and $T_{2}$.  A good textbook explanation of how to apply this type of test to laboratory physics data may be found in section 7.5, "Testing Goodness-of-Fit with $\chi^{2}$" in "Statistical Data Analysis" by Glen Cowan.  You'll have to actually find and read the reference in order to really understand why the following is true, but essentially the upshot is that if the $\delta_{ij}$ are normally distributed (which in most cases we usually expect they should be), and if the theoretical model $T_{i}$ accurately captures the underlying data, then the sum $$S_{i} = \sum_{j=1}^{N} \delta_{ij}^{2}$$ is expected to be distributed as a $\chi^{2}$ distribution with $N$ degrees of freedom.
You need to understand a couple of key points in order to correctly apply this result.  First of all, in most texts, including the reference text that I listed, the model $T_{i}$ is usually presumed to have $m$ adjustable parameters; i.e., the model may be written more explicitly as $T_{i}(\theta_{1}, \theta_{2}, ..., \theta_{m})$ and the experimenter typically searches for "optimal" values for the $\theta_{m}$ adjustable parameters until he finds a set of values which minimizes $S_{i}$.  In this more usual case, the pdf of $S_{i}$ is actually expected to be a $\chi^{2}$ distribution with $(N-m)$ degrees of freedom.  In your case, since you presumably derived the functional form for the models $T_{i}$ by writing down some quantum mechanics on a piece of paper, the only parameters in your model should be fundamental physical constants such as $\hbar$ and $c$ whose values are already well-known, therefore I'd guess you probably don't have any adjustable parameters, and $m=0$.
Once you have tabulated actual values for $S_{1}$ and $S_{2}$, you may quantify the goodness of fit, or "success" of each model by quantifying how far out into the tails of a $\chi^{2}$ distribution each one falls.  Typically this is done by calculating a p-value, which you can obtain by calculating the integral under the $\chi^{2}$ curve to the right of the actual observation (this graphic gives a pretty good visual summary of what I mean).  This p-value gives you a quasi-apples-to-apples way of comparing different models in order to see which one is better, even for models with slightly different numbers of adjustable parameters (it breaks down however if $m$ becomes very large, on the order of $N$).  Effectively, the $S_{i}$ which results in more of the $\chi^{2}$ integral lying to the right, (i.e., a larger p-value) corresponds to smaller deviations $\delta_{ij}$ on average, and indicates a better goodness of fit.
One last minor point: as $(N-m)$ becomes large, the mean of a $\chi^{2}$ distribution with $(N-m)$ degrees of freedom approaches $(N-m)$; many experimental physicists will often therefore define a "reduced $\chi^{2}$", $$\chi_{(N-m)}^{2} = \frac{S_{i}}{(N-m)}$$ as a sort of shorthand.  Typically, if you have estimated your errors $\sigma_{j}$ correctly, and if the model $T_{i}$ is a good fit to the data, then the reduced $\chi^{2}$ should typically come out to a value somewhere around 1, give or take.  In practice, this reduced $\chi^{2}$ is often used as an approximate measure of goodness-of-fit, because it's slightly easier to calculate than a p-value.  You may find some references to it in the literature, but don't become confused by it; the p-value is actually the more fundamental and technically correct measure to use for your purpose.
