goodness of fit statistic independant of the fixed effects

Question

I am using glm model to fit data.

I want to check if my fit is good. My model is usually the classical model with only fixed effects. In order to do so, I use goodness of fit tools like residual deviance.

However, I have been told by a professor that:

"goodness of fit statistic is expected to be independant of the fixed effects".

I don't understand that sentence, on the theoretical level. Can someone please explain me, why for instance, if the residual deviance depends on the fixed effect, one can't use the residual deviance in order to know the quality of the fit?

I found this, Why is it futile to use the deviance as a goodness-of-fit measure for Bernoulli data? if anyone has another example.

Answer 1

By goodness of fit in this context we mean to evaluate if the model used for the analysis is reasonable, that is, it the underlying assumptions of the analysis is not to badly broken. For that we cannot just look at the parameter estimates---the fixed effects---the estimation algorithm will give us some estimates, however broken is the model assumptions. Then, any statistic that is just a function of the parameter estimates cannot give information independent of the estimates, it is only repeating the same information.

That at least indicates that a good statistic for evaluating goodness of fit should be independent of the parameter estimates, or at least not too heavily dependent. In itself that can only be a heuristic, a statistic independent of the parameter estimates is not necessarily a good test for goodness of fit. For the Bernoulli case you mention, the deviance is a function of the parameter estimates, so does not give a good test for goodness of fit. This is discussed in this classical book in section 4.4.5 Sparseness, which in this context means count data with small $n$. Bernoulli data is the extreme example, but binomial data with small $n$ or likewise Poisson data is similar.

In normal linear models we use the residuals for evaluating goodness of fit. The residuals are not independent from the parameter estimates, but each individual residual has correlation zero with the estimates, so the dependence is weak (unless $n$ is very small, in which case testing goodness of fit do not make much sense.)