P-value of alternative hypothesis - Can you simply take the difference of the chi2 goodness-of-fit tests?

Question

I guess this is the kind of question extremely hard to Google if you're lacking just the right word.

My situation is: I have some experimental data points, and I have two models: the "simple model" (null hypothesis, H0) and the more complex model (H1), with two more degrees of freedom. I try and fit each model separately to my data, and to assess how good the fit is, I compute a chi2 goodness-of-fit test.

I notice that chi2(H0) > chi2(H1), and I want to compute the p-value (a 3-sigmas significance allowing me to reject H0).

A colleague tells me that I can compute chi2' = chi2(H0) - chi2(H1) and the corresponding degrees of freedom: ndof' = ndof(H1) - ndof(H0) = 2. And then, I can integrate the chi2 distribution (with 2 d.o.f.) from chi2' to infinity, which yields the p-value. The colleague says that taking the difference of chi2 values "comes from some theorem", which they ofc cannot find.

So far I have failed to find references to such method (nor help from anyone knowledgeable in statistics). Is that the right way of computing the significance?

If not, what would be a correct way of assessing by how much H1 is favoured over H0 ?

Answer 1

Regarding the first part of your question, try writing out the $\chi^2$ and likelihood $\cal L$ (product of probabilities at each data point), and see if $-2 \ln {\cal L }= \chi^2$. The result your colleague is referring to sounds like Wilk's theorem for the distribution of the ratio of two nested likelihoods (certain restrictions apply).

Regarding your second question, in any case, you should be able to run simulations of H0, calculate test statistic $\chi2'$ for each simulation, and with that distribution of $\chi2'$ see where your experimental value falls to determine a p-value.

Lastly, regarding what you noticed earlier, as you add degrees of freedom, the $\chi^2$ can only remain constant or decrease.