4
$\begingroup$

I've been using the approach from Appendix A from this paper. It states that if I have two fits with $\chi^2$ values $\chi^2_1(\nu_1)$ and $\chi^2_2(\nu_2)$ ($\nu$ - degree of freedom), then their difference $\Delta\chi^2=\chi^2_1-\chi^2_2>0$ is distributed like $\chi^2$ with $\Delta\nu=\nu_1-\nu_2$ degrees of freedom. Then, a small $p$-value indicates a small probability that the model with $\chi^2_1$ describes the data compared with the other model. Obviously, both $\Delta\chi^2$ and $\Delta\nu$ have to be positive (if both are negative, then you just have to compare the models the other way around). This have worked well until now - because now I have some fits that have $\Delta\chi^2>0$ and $\Delta\nu<0$ (or vice versa). This means that some models with less parameters are better than those with more parameters. I want to quantify what "better" means. The questions that arise are the following:

  1. Can I still use this approach and if yes, how?
  2. If not, can I use the F-test for it?
  3. If not, what's the proper approach to compare the models?

I've been using the above approach because it's common in the field. Now I'm doing some new things, but would like to use the same tool as they are most known in the community (if it's possible, of course).

For example, I have two pairs I want to test:

  1. $\chi^2_1=16.6125$, $\nu_1=13$; $\chi^2_2=14.2513$, $\nu_2=11$; for this I get a $p$-value of 0.307094.
  2. $\chi^2_1=14.0735$, $\nu_1=10$; $\chi^2_2=15.0182$, $\nu_2=7$.

EDIT: I see that not paying attention to whether the models are nested or not may be a serious issue. Browsing the web and SE I learnt that the AIC method should do (for nested as well as non-nested models). Still, I'm curious about whether my models are nested or not; they are standard normal (Gaussian) dist., skew normal (SN) dist. and sinh-arcsinh (SAS) dist. As far as I know, the latter two contain standard Gaussians in them. I have to verify whether the SAS contains SN in it. Or, may anyone shed some light on this issue?

EDIT2: I got $AIC=153.519$ for a mixture of two standard Gaussians (with $\chi^2=16.6125,\,\nu=13$) and $AIC=157.24$ for a mixture of three Gaussians (with $\chi^2=14.0735,\,\nu=10$). These two models are obviously nested, and it comes from the $\chi^2$ (and reduced $\chi^2$, and corresponding $p$-values) that a three-Gaussian is better than a two-Gaussian. Contrary, AIC gives the opposite conclusion (supported by BIC). Why?

I'm using Mathematica's built-in functionality to compute AIC (shown to be correct here).

Improve this question
$\endgroup$
2
  • 4
    $\begingroup$ Are your models nested? That is the basic requirement of these tests. So a model with more parameters can only be better. $\endgroup$
    – Aniko
    Commented May 22, 2015 at 20:27
  • $\begingroup$ @Aniko Good question; at first I wanted to say "yes, of course", but now I'm not quite sure. In my example 1. I have a mixture of 2 standard Gaussians (1) and a mixture of 2 skew normal distributions (2). In my example 2. I have a mixture of 3 standard Gaussians (1) and a mixture of 3 skew normal dist. In the paper I cite it's said that models are nested if one is a subset of the other; so, is this the case in my examples? When you fix the skewness parameter in the skew normal dist. to zero it reduces to a standard Gaussian. $\endgroup$
    – corey979
    Commented May 22, 2015 at 20:55

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.