3
$\begingroup$

I am an MBA Student that is taking courses in Statistics.

Up until now, we had only encountered "Chi-Square" in the context of Contingency Tables. That is, how to find if the difference between different groups of subjects is statistically significant or non-significant (e.g. https://web.pdx.edu/~newsomj/uvclass/ho_chisq.pdf).

On the other hand, we have been learning about the concept of "Overdispersion" (related to variance). For example, we are learning about Regression Models for Discrete Count Data (e.g. Poisson GLM) in which an "Overdispersion Parameter" is required to be estimated, and this parameter will serve to "correct" the confidence intervals on the beta-coefficients of the regression model.

I would have thought that this "Overdispersion Parameter" would have been estimated through the Maximum Likelihood Equations - but our prof mentioned that the "Overdispersion Parameter" is estimated independently of the Maximum Likelihood Equations and is in fact calculated using a very similar formula as the "Chi-Square Statistic".

This makes me wonder:

  • Is there some special relationship between "Overdispersion" (i.e. variance) and "Chi-Square" - and why isn't the Overdispersion Parameter estimated from the Maximum Likelihood Equations (as is done for the regression parameters)?
$\endgroup$
2
  • $\begingroup$ Of possible interest $\endgroup$
    – Dave
    Sep 30 at 3:31
  • $\begingroup$ "the "Overdispersion Parameter" is estimated independently of the Maximum Likelihood Equations and is in fact calculated using a very similar formula as the "Chi-Square Statistic"." Can you explain better how this formula looks like? It may just be the same as maximizing the (quasi) likelihood function. (In a similar way as the mean of a sample is the maximum likelihood estimate of the mean of a normal distributed population) $\endgroup$ Oct 1 at 7:50

2 Answers 2

3
+25
$\begingroup$

There will be better authorities on CV with respect to this topic, and what follows is mostly what I recall from my actuarial studies.

In the context of a GLM with Poisson link, if there is a need to adjust for overdispersion (which is a contradiction of the assumptions of a Poisson GLM) then often a simple approach is to change the specification of the variance.

$$V(Y_{i})=\phi\mu_{i}\quad\quad\quad(1)$$ where $\phi$ is the dispersion parameter.

I would have thought that this "Overdispersion Parameter" would have been estimated through the Maximum Likelihood Equations

The log-likelihood function for this Poisson GLM cannot incorporate this new information into its form. The Poisson distribution is a single parameter distribution, how can it? We cannot change the variance specification and preserve the mean. By definition, $\mathbb{E}[Y_{i}]=V(Y_{i})$ for a Poisson.

In the case of overdispersion, the general estimation method is Iterative Reweighted Least Squares (IRLS) and not vanilla MLE. The estimates obtained from this are generally referred to as Maximum Quasi-likelihood estimates (MLQE).

In the case of the variance being proportional to the mean in $(1)$, it turns out that the $\hat{\mu}^{\text{MLE}}_{i}$ estimate coincides with the MQLE estimate due to cancellation of the $\phi$ term so it's good-to-go in that regard.

However, we still need to estimate $\hat{\phi}$ but we do this separately because as we just found out, $\hat{\mu}_{i}$ does not depend on $\phi$.

The variance of the estimates is $$\text{Var}(\hat{\beta})=\phi(X'WX)^{-1}$$ which means we need to know $\phi$ in order to make correct inference about estimated parameters i.e. the standard errors are wrong. It's important to note that we only care about the overdispersion issue because of its effect on the standard errors.

I believe there are a few ways to go about getting $\hat{\phi}$ but the one you noted is to use the Pearson $\chi^{2}$ statistic. I think the general reasoning is that it's a moment-matching exercise. If we define $$P=\sum_{i=1}^{n}\frac{(y_{i}-\mu_{i})^2}{\text{Var}(y_{i})}=\sum_{i=1}^{n}\frac{(y_{i}-\mu_{i})^2}{\phi\mu_{i}}$$

If the assumptions of the model were correct, then $$\phi=1\quad\quad\Rightarrow\quad\quad P\sim\chi^{2}_{n-p}\quad\quad\Rightarrow\quad\quad\mathbb{E}[P]=n-p$$

Then we simply solve for the $\hat{\phi}$ that gives us this expectation $$\hat{\phi}=\sum_{i=1}^{n}\frac{(y_{i}-\mu_{i})^2}{\text{Var}(y_{i})}\bigg/(n-p)$$

So hopefully this answers your other question:

Is there some special relationship between "Overdispersion" (i.e. variance) and "Chi-Square"

$\endgroup$
2
  • $\begingroup$ Thank you so much for replying! Can you please explain what is meant by "Moment Matching"? Thank you so much! $\endgroup$ Sep 28 at 4:40
  • $\begingroup$ By moment-matching I was referring to a Method of Moments (MoM) approach, which is an estimation method (much like MLE is). In MoM, we use the empirically observed moments (sample mean, etc.) to solve for the parameters of the assumed model. $\endgroup$ Sep 28 at 4:45
2
$\begingroup$

When we incorporate a dispersion parameter, then we are not working with real likelihood functions anymore.

Motivation, where does it come from

For several one parameter distributions (more precisely from the exponential family) it is possible to describe the derivative of the likelihood function as

$$\frac{\partial \log\mathcal{L}(x,\mu)} {\partial \mu}= \frac{x-\mu}{V(\mu)}$$

Where $V(\mu)$ is the variance of the distribution as function of the mean $\mu$.

  • For instance for the Poisson distribution with mean $\mu$ and variance $V(\mu) = \mu$ the logarithm of the density is

    $$\log[f(x,\mu)] = x \log(\mu) - \mu - \log(x!)$$

    and the derivative with respect to location parameter $\mu$ is

    $$\frac{\partial \log[f(x,\mu)]}{\partial \mu} = x/\mu - 1 = \frac{x-\mu}{\mu}$$

Including a dispersion parameter

This way to describe the likelihood function can be extended and we add a parameter

$$\frac{\partial \log\mathcal{L}(x,\mu,\phi)} {\partial \mu}= \frac{x-\mu}{\phi V(\mu)}$$

The effect of this parameter $\phi$ is that it changes the curvature of the likelihood function and this relates to more/less precise knowledge about the parameter $\mu$ depending on whether the likelihood is made sharper/blunt.

Finding the dispersion parameter (how it won't work with likelihood)

In this way of expressing the likelihood function, as the derivative with respect to $\mu$, the dispersion parameter does not play a role in the minimum.

So when you are minimizing this quasi likelihood function (quasi because it may not need to be the likelihood function of an actual existing distribution density) then the solution for $\mu$ is independent of $\phi$ which is only a scale factor.

We show the Gaussian distribution as an example to make more clear why this dispersion parameter disappear in the expression of the quasi likelihood function disappears

  • For the Gaussian distribution with mean $\mu$ and variance $V(\mu) = \sigma^2$ (thus a constant function) the logarithm of the density is

    $$\log[f(x,\mu, \sigma)] = -0.5 \log[\sigma^22 \pi] - 0.5 \frac{(\mu-x)^2}{\sigma^2}$$

    and the derivative with respect to location parameter $\mu$ is

    $$\frac{\partial \log[f(x,\mu, \sigma)]}{\partial \mu} = \frac{x-\mu}{\sigma^2} $$

    This derivative is only expressing the second $ 0.5 \frac{(\mu-x)^2}{\sigma^2}$ term of the likelihood function. The part with $ -0.5 \log[\sigma^22 \pi]$, which depends on the dispersion $\sigma^2$ but not on the mean parameter $\mu$, is eliminated.

Finding the dispersion parameter (alternative)

So to find the dispersion parameter, one has to use a different trick. In Wedderburn 1974 this is done by using the method of moments. The observed variance and the dispersion parameter is chosen such that the expected variance equals the observed variance.

$$E\left[\sum_{i=1}^n \frac{(x_i-\mu_i)^2}{V(\mu_i)} \right] \approx \phi({n-m})$$

(The approximation is based on assuming that $V(\mu_i)$ is approximately linear and that the distribution is $\chi^2$ distributed)

Then using the estimates $\hat{\mu_i}$ in place of the true values $\mu_i$

$$\phi = \frac{1}{n-m} \sum_{i=1}^n \frac{(x_i-\hat\mu_i)^2}{V(\hat\mu_i)}$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.