What useful properties does the canonical link function have?

Question

So here I am studying generalized linear models. I know this question is quite naive and simple, but I do not exactly know why the link canonical function is so useful. Could someone provide me an intuition on this problem?

Answer 1

I know this question is quite naive and simple, but I do not exactly know why the link canonical function is so useful

Is it really so useful? A link function being canonical is mostly a mathematical property. It simplifies the mathematics somewhat, but in modeling you should anyhow use the link function that is scientifically meaningful.

So what extra properties does a canonical link function have?

1. It leads to existence of sufficient statistics. That could imply somewhat more efficient estimation, maybe, but modern software (such as glm in R) do not seem to treat canonical links differently from other links.

2. It simplifies some formulas, so theoretical developments are eased. Many nice mathematical properties, see What is the difference between a "link function" and a "canonical link function" for GLM.

So advantages seem to be mostly mathematical and algorithmical, not really statistical.

Some more details: Let $Y_1, \dotsc, Y_n$ be independent observations from the exponential dispersion family model $$ f_Y(y;\theta,\phi)=\exp\left\{(y\theta-b(\theta))/a(\phi) + c(y,\phi)\right\} $$ with expectation $\DeclareMathOperator{\E}{\mathbb{E}} \E Y_i=\mu_i$ and linear predictor $\eta_i = x_i^T \beta$ with covariate vector $x_i$. The link function is canonical if $\eta_i=\theta_i$. In this case the likelihood function can be written as $$ \mathcal{L}(\beta; \phi)=\exp\left\{ \sum_i \frac{y_i x_i^T \beta -b(x_i^T \beta)}{a(\phi)}+\sum_i c(y_i,\phi)\right\} $$ and by the factorization theorem we can conclude that $\sum_i x_i y_i$ is sufficient for $\beta$.

Without going into details, the equations needed for IRLS will be simplified. Likewise, this google search mostly seems to find canonical links mentioned in the context of simplifications, and not any more statistical reasons.

Answer 2

The canonical link function describes the mean-variance relationship in a GLM. For instance, a binomial random variable has link function $\mu = \exp( \nu) /(1-\exp(\nu))$ where $\nu$ is a linear predictor $\mathbf{X}^T\beta$. Note that $\frac{\partial }{\partial \nu} \mu = \mu(1-\mu)$ which is the appropriate mean-variance relationship for a Bernoulli random variable. The same is true of Poisson random variables, where the inverse link function is $\mu = \exp(\nu)$ and $\frac{\partial }{\partial \nu} \mu = \mu$ where in a Poisson random variable, the variance is the mean.

The generalized linear model solves an estimating equation of the form:

$$ U(\beta) = D V^{-1} (Y - g(\mathbf{X}^T\beta))$$

where $D = \frac{\partial}{\partial \beta} g(\mathbf{X}^T\beta)$ and $V=\text{var}(Y)$. When the link is canonical, therefore, $D = V$ and the estimating function is the score function, i.e. the derivative of the log likelihood:

$$ S(\beta) = \mathbf{X}^{T}(Y - g(\mathbf{X}^T\beta))$$

As was noted in Wedderburn's 1976 paper on quasilikelihood, the canonical link has the advantage that expected and observed information are the same and that iteratively reweighted least squares is equivalent to Newton-Raphson, so this simplifies estimating procedures and variance estimation.