3
$\begingroup$

I was recently at a seminar where a statistician used chi-squared divided by the degrees of freedom (DF) as an assessment of model performance in a logistic regression. They suggested that having a $\frac{\chi^2}{DF}$ close to 1 suggested good performance in the model.

I am having some difficulty googling more information around this measure of goodness of fit and was hoping to get some guidance or resources.

Once source I found stated the following:

"A $\chi^2$ statistic with k degrees of freedom, d.f., is the sum of the squares of k random unit-normal deviates. Therefore its expected value is k, and its model variance is 2k. This provides the convenient feature that the expected value of a mean-square statistic, i.e., a $\chi^2$ statistic divided by its d.f. is 1." ~ https://www.rasch.org/rmt/rmt171n.htm

And there is some mentioned of dividing test statistics by degrees of freedom in SAS documentation guidelines. But I am having difficulty finding a more thorough and satisfactory answer.

Why does having $\frac{\chi^2}{DF}$ close to 1 suggest decent good model performance (goodness-of-fit)?

Related but different questions on $\frac{\chi^2}{DF}$ are found here and here.

$\endgroup$
2
  • 5
    $\begingroup$ Model performance isn't the same as goodness of fit. Did they say the former or the latter? Dividing a chi-squared test statistic in a model by the statistic's degrees of freedom is an occasionally used measure of effect size. It can be used to assess one sense of variable importance, eg. This would only make sense within a single model, though, not between different models from different datasets, etc. $\endgroup$ Mar 7, 2023 at 19:57
  • $\begingroup$ Yes, it was actually goodness-of-fit. - I was also able to touch base with the presenter via email who was gracious enough is recommend Vonesh 1996 as a good starting point. $\endgroup$
    – Jbnimble
    Mar 7, 2023 at 21:27

1 Answer 1

2
$\begingroup$

A couple preliminary points:

First, as gung notes, model performance is not the same as goodness of fit. The latter is one aspect of the former - all else being equal, a model that fits the data better is probably a better-performing model - but not the whole picture.

Second, when people say $\chi^2/DF \approx 1$ is an indicator of good model fit, what does $\chi^2$ mean and where did it come from? The answer is that, to assess goodness of fit, we often look at the residual sum of squares (RSS): $$RSS = \sum_{i=1}^n r_i^2 = \sum_{i=1}^n (\hat{y}_i - y_i)^2$$ where $y_i$ is the actual and $\hat{y}_i$ is the model predicted value. Under certain conditions, the RSS is exactly or approximately $\chi^2$ distributed. This property of the RSS allows us to assess the quality of the model fit against a theoretical expectation.

Linear Model with Gaussian Errors

To see how we can use the RSS and $\chi^2$ to assess goodness of fit, let's now ask: if we were to happen upon a good model, how big would we expect the RSS to be? For example, suppose we have $n = 1,000,000$ data points that fit the standard linear model perfectly:

$$y_i = ax_i + b + e_i$$

with $e_i$ i.i.d. standard normal. Suppose we're very lucky to have an "oracle", a magic algorithm which always finds the exact true values of $a$ and $b$ regardless of the noise in the data. Then the residuals of our oracular "fit" would be precisely

$$r_i = y_i - ax_i - b = e_i$$

and the RSS would be precisely $\sum_i e_i^2$. Since $\chi^2$ is by definition the sum of squared i.i.d. standard normal random variables, our RSS is $\chi^2$-distributed with $k = n = 1,000,000$ degrees of freedom.* Because $k$ is so large in our example, $\chi^2$ is also very close to normally distributed with mean $k$ and variance $2k$. This means the RSS is approximately normal with mean $k=1,000,000$ and standard deviation $(2k)^{1/2} \approx 1414$, and RSS/k is also approximately normal with mean 1 and standard deviation $(2k)^{1/2}/k \approx 0.0014$. With RSS/k having such a small standard deviation, for it to deviate even 0.01 from its expected value of 1 would be more than seven standard deviations away from the expected value, and therefore extremely unlikely. Thus, the realized value of RSS/k is almost certainly within the interval 0.99-1.01.

This example suggests that, if we have lots of data and are able to fit a perfect model, RSS/k should be very close to 1. In the real world, models rarely fit perfectly, and a model that fails this test is not necessarily bad. But it can be a useful diagnostic - if it's not close to 1, it may be worth plotting the residuals or doing other forms of digging in.

*In this thought experiment, we have an oracle which gives us the true value of $a$ and $b$ regardless of the noise in the data. In practice, one must make an adjustment for the fact that the estimated $a$ and $b$ are not exactly right and are influenced by the noise in the data, which makes the RSS lower than it would be if we had the oracle. That's why when we do a real fit, we have to subtract off the number of parameters to obtain the degrees of freedom, and we get $k = n-2$ rather than $k = n$ for a two-parameter linear fit like this example.

Logistic and other models

The theoretical results used in the argument above are available in more general situations. Actually, any time you (1) fit a model to i.i.d. data with maximum likelihood, (2) have a lot of data, and (3) you know the structure of the true model, the residual sum of squares will have an approximate $\chi^2$ distribution. This follows from more general results for likelihood ratio tests.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.