Should one use the same overdispersion parameter when comparing Binomial models?

Question

McCullagh & Nelder, 2nd edition, p 91 claim that to make comparisons "fair", it's best to use a single estimate of overdispersion parameter, usually derived from the most complex model. I noticed that the same thing is done in this SAS example. Burnham and Anderson (2nd edition, p 68) are also adamant about using just the global model to estimate the dispersion parameter and then plugging it into smaller models.

Does anyone know what is meant by "fair" and why re-estimating a variance parameter is unfair? If (which is very likely) the most complex model is overfitted then its deviance is too small, meaning that the dispersion parameter is underestimated. Is it still a good idea to use the biased estimate for all models?

McCullagh & Nelder (p 127) admit that estimating the dispersion parameter is similar to estimating $\sigma^2$ in linear regression. Correspondingly, I recalled a similar reasoning in Neter et al (4th edition, p 342) where $C_p$ criterion for linear models is described. We are suggested that the largest model delivers "an unbiased estimator of $\sigma^2$" that we are supposed to use to assess the bias-variance tradeoff for smaller models.

For both linear and binomial setups, all the authors above agree that one must get the structural part of the model right before estimating the dispersion parameter. However, if we know what the correct structural part is, it is not clear why we should proceed with model selection instead of just using the global model.

Another contradiction is that, if we want to use AIC and similar criteria for linear model selection, then I have never heard of a recommendation to estimate $\sigma^2$ using the global model and then plug it into the smaller ones. Likewise, I haven't heard of such recommendation for the dispersion parameter in Negative Binomial regression.

Some comments below suggested that the problem is that in Binomial regression $\phi$ is not estimated via MLE, but in Negative Binomial regression it is. So, as long as there is a way to estimate dispersion via MLE, no one has a problem with allowing the dispersion parameter to vary across models, and vice-versa? For instance, if we use Williams variance function

$Var[Y_i] = m_i\pi_i(1-\pi_i)[1 + \phi(m_i-1)]$

where $\phi$ is estimated by IWLS, then we must use the same $\phi$ for all the models in the pool. However, if we apply Beta-binomial regression that can be estimated by MLE given that $\alpha_i + \beta_i = c$, then it's fine to allow $c$ to vary across the models. The problem with such reasoning is that Beta-binomial is a particular case of Williams variance function, where $\phi = 1/(c+1)$. It doesn't make sense to disallow $\phi$ to vary simply because it was not estimated by "pure" MLE. IWLS and MLE estimates of $\phi$ are probably close anyway, and may be even identical because in practice MLE is often implemented via IWLS (e.g. Negative Binomial regression in R).

Answer 1

On page 90 of McCullagh & Nelder, they state that many covariate selection procedures, including AIC minimization and tests using the F statistic, are equivalent to minimizing $Q = D + \alpha q \phi$. Here $D$ is the deviance, $\alpha$ is a function of the number of data points, $q$ is the number of covariates, and $\phi$ is the dispersion parameter.

They cite a paper of Atkinson which I can't access for this statement. From the introduction, it seems Atkinson's actual statement is that the quantity to be minimized is $-L + \alpha q/2$, where $L$ is the maximized log likelihood of the model. Note the relationship $D/\phi = -2L + C$, where $C$ is a constant, from pages 33 and 34 of McCullagh and Nelder. It seems to me that in order to obtain their formulation from Atkinson's they already assume $\phi$ is the same across all candidate models.

I think there are basically two issues here. First, if you are estimating a dispersion parameter in some way other than MLE, it's not even clear that Atkinson's quantity is defined, since $L$ should be the maximized log likelihood. If you are estimating a dispersion parameter for a binomial distribution, as McCullagh and Nelder do, you are already not using MLE to fit all the parameters of the model.

Second, you may still want to use a criterion of this kind anyway by minimizing $D/(2\phi) + \alpha q/2$ among your candidate models. $D$ can be calculated for the model even if the regression parameters weren't obtained by maximizing the likelihood. If $\phi$ is constant across all candidate models this has heuristic value because it measures a trade off between model fit (with decreasing $D/\phi$) and complexity (with increasing $\alpha q$). This seems to be what McCullagh and Nelder are suggesting.

However if the estimate for $\phi$ is not constant across all candidate models, then even this heuristic value is lost. $D/ \phi$ is no longer solely a measure of how well each candidate model fits the data but is also affected by changes in each model's dispersion parameter estimate, and the nature of the complexity-fit trade off becomes less clear. In fact if the differences in your scale parameter estimates are sufficiently large, minimizing this quantity amounts to minimizing the estimated scale parameter.

Of course, the scale parameter estimate for each model will depend both on the number of parameters and the fit of the model. So I think the best answer to this question is: If you believe the complexity-fit trade off implied by your method of estimating $\phi$ will produce a better model than the quantity above, then allow it to vary for each model, or just use it directly. I would be skeptical of this belief, since any method of estimating $\phi$ was likely not intended to be a good procedure for model selection.

Answer 2

I suspect the reason for the recommendation is that, in the old days, first the model was fit, then the dispersion parameter was calculated, and then the likelihood was adjusted for overdispersion. Deriving suitable test statistics for a LRT with adjusted dispersion parameters seems difficult, so it may be that people just said: whatever, we'll just develop a test conditional on a fixed dispersion, and that's it.

Still, keeping the dispersion fixed seems weird to me. As you say, the most complex model likely fits tighter to the data, so using its dispersion parameter also for the simpler models should lead to suboptimal likelihoods, which would seem to create a bias towards larger complexity.

A more sensible approach that makes use of modern computing power seems to me to fit both models including the dispersion with a full likelihood, and then do a simulated LRT for the comparison.