I've been using the approach from Appendix A from this paper. It states that if I have two fits with $\chi^2$ values $\chi^2_1(\nu_1)$ and $\chi^2_2(\nu_2)$ ($\nu$ - degree of freedom), then their difference $\Delta\chi^2=\chi^2_1-\chi^2_2>0$ is distributed like $\chi^2$ with $\Delta\nu=\nu_1-\nu_2$ degrees of freedom. Then, a small $p$-value indicates a small probability that the model with $\chi^2_1$ describes the data compared with the other model. Obviously, both $\Delta\chi^2$ and $\Delta\nu$ have to be positive (if both are negative, then you just have to compare the models the other way around). This have worked well until now - because now I have some fits that have $\Delta\chi^2>0$ and $\Delta\nu<0$ (or vice versa). This means that some models with less parameters are better than those with more parameters. I want to quantify what "better" means. The questions that arise are the following:
- Can I still use this approach and if yes, how?
- If not, can I use the F-test for it?
- If not, what's the proper approach to compare the models?
I've been using the above approach because it's common in the field. Now I'm doing some new things, but would like to use the same tool as they are most known in the community (if it's possible, of course).
For example, I have two pairs I want to test:
- $\chi^2_1=16.6125$, $\nu_1=13$; $\chi^2_2=14.2513$, $\nu_2=11$; for this I get a $p$-value of 0.307094.
- $\chi^2_1=14.0735$, $\nu_1=10$; $\chi^2_2=15.0182$, $\nu_2=7$.
EDIT: I see that not paying attention to whether the models are nested or not may be a serious issue. Browsing the web and SE I learnt that the AIC method should do (for nested as well as non-nested models). Still, I'm curious about whether my models are nested or not; they are standard normal (Gaussian) dist., skew normal (SN) dist. and sinh-arcsinh (SAS) dist. As far as I know, the latter two contain standard Gaussians in them. I have to verify whether the SAS contains SN in it. Or, may anyone shed some light on this issue?
EDIT2: I got $AIC=153.519$ for a mixture of two standard Gaussians (with $\chi^2=16.6125,\,\nu=13$) and $AIC=157.24$ for a mixture of three Gaussians (with $\chi^2=14.0735,\,\nu=10$). These two models are obviously nested, and it comes from the $\chi^2$ (and reduced $\chi^2$, and corresponding $p$-values) that a three-Gaussian is better than a two-Gaussian. Contrary, AIC gives the opposite conclusion (supported by BIC). Why?
Mathematica's built-in functionality to compute AIC (shown to be correct here).