Where are the residuals in a GLM?

Question

I am just now moving on to GLMs after the standard models.

In the standard model,

y = Xb + epsilon

and epsilon is assumed to be normally distributed. That means we can write

y - Xb = epsilon

and then we can minimize the lhs using some suitable norm given the normality assumption.

In a GLM, these residuals are nowhere to be seen, so what are the residual assumptions? That is, when you fit a GLM and determine the residuals, how do you check your distribution assumption? A qqplot? Against what? The normal quantiles? Or the quantiles of the distribution you chose?

The GLM as I understand it:

mu = Xb, mu = Ey, y follows some non-Gaussian distribution.

Answer 1

In addition to @DavidSmith's answer, some more formal terminology follows:

Generalized linear models invoke a mean-variance relationship as a consequence of the link function. There are no residuals in a GLM because the variance is just a function of the mean. So when we write a GLM it is of the form:

$$g(E[Y|X]) = \beta X$$

Where $g$ is a link function, the terms $\beta X$ are the linear predictors $\nu$ and the transformed values $g^{-1}(\beta X)$ are the fitted values. In general, the case is that $E[Y] = g^{-1}(\beta X)$ implies $var(Y) = \frac{\partial}{\partial \beta} g^{-1}(\beta X)$. For instance, with logistic regression, the inverse logit link $g^{-1}(x) = \log(\frac{X}{1-X})$ has $g^{ \prime -1}(X) = \log(\frac{1}{1-X}) = g^{-1}(X)(1-g^{-1}(X))$ with the second expression easily recognized as the binomial variance.

When you write out the estimating equations for common probability models, like binomial, poisson, or exponential, you actually observe that the information (or variance) depends on the mean and nothing else. These one parameter models, as the name suggests, use only one parameter (like a log odds or log relative rate) to relate the expected outcome to a linear combination of predictors and a corresponding link function. The influence function (gradient or derivative) of the link relates the mean to the variance.

Gaussian probability models differ from binomial (logistic) models in that they are 2 parameter models including a dispersion term (sigma, or the residual variance). A Gaussian model is also different from other 2 parameter models (like negative binomial or Gamma) because you can write the residual variance as a separate term in a model.

Basically the ordinary least squares with normal, independent error is the only case I know of where we can actually write: $y = \beta X + \epsilon$ meaningfully.

The bigger question of how you relate expected outcomes to observed outcomes is complicated. In a normal model, this is a simple difference of expected and observed to obtain a residual. In GLMs, the variance is heteroscedastic because the mean changes as a function of $X$, so you can standardize each residual by dividing by the expected standard error to obtain Pearsonized residuals.

Answer 2

The specific residuals depend on the distribution used and on the characteristics of the dependent variable. Sometimes these are not very informative and sometimes they can't be computed easily.

The utility of residuals varies greatly as well, in evaluating how well the model works. Logistic regression of a binary variable is a good example. All the residuals can be computed, but making sense of them is difficult without a summary such as calibration and a Hosmer-Lemeshow test. Summaries of other kinds, eg, by another categorical variable, can also be useful. Sometimes you can learn from comparing the estimated probabilities from two different models.

• For ordinal or nominal logistic regression with several categories, you can compute a set of probabilities for each observation. These can be useful but are difficult to interpret with straightforward graphical methods or summary statistics.

• Residuals for censored survival data are not uniquely defined. Estimated survival time might be longer or shorter than the time of censorship.

• Residuals for highly skewed dependent variables, eg, exponential, negative binomial, Poisson, etc, can be misleading in graphical displays since models do not reduce or remove the skewness. They leave you with the impression of many large outliers. Sometimes it is better to examine these on a transformed scale, such as logs.

So, there is no general purpose answer to your question. The use of residuals depends on the model.

For Gaussian residuals the story is easier. Unfortunately, we often find out that there are problems with a linear model that are not resolvable in simplistic, algorithmic ways.