# What is the difference between a "link function" and a "canonical link function" for GLM

What's the difference between terms 'link function' and 'canonical link function'? Also, are there any (theoretical) advantages of using one over the other?

For example, a binary response variable can be modeled using many link functions such as logit, probit, etc. But, logit here is considered the "canonical" link function.

• I discuss link functions extensively here: Difference between logit and probit models, focusing on regression for a binary response variable. Although only a little of that discussion focuses on the meaning of a link function's being 'canonical', it may nonetheless be helpful to read. Note that to understand the distinction b/t & advantages of a canonical vs non-canonical link function requires going fairly deep into the math underlying the GLiM. Oct 21 '12 at 14:37

The above answers are more intuitive, so I try more rigor.

What is a GLM?

Let $Y=(y,\mathbf{x})$ denote a set of a response $y$ and $p$-dimensional covariate vector $\mathbf{x}=(x_1,\dots,x_p)$ with expected value $E(y)=\mu$. For $i=1,\dots,n$ independent observations, the distribution of each $y_i$ is an exponential family with density $$f(y_i;\theta_i,\phi)=\exp\{[y_i\theta_i-\gamma(\theta_i)]/\phi+\tau(y_i,\phi)\}$$ Here, the parameter of interest (natural or canonical parameter) is $\theta_i$, $\phi$ is a scale parameter (known or seen as a nuisance) and $\gamma$ and $\tau$ are known functions. The $n$-dimensional vectors of fixed input values for the $p$ explanatory variables are denoted by $\mathbf{x}_1,\dots,\mathbf{x}_p$. We assume that the input vectors influence (1) only via a linear function, the linear predictor, $$\eta_i=\beta_0+\beta_1x_{i1}+\dots+\beta_px_{ip}$$ upon which $\theta_i$ depends. As it can be shown that $\theta=(\gamma')^{-1}(\mu)$, this dependency is established by connecting the linear predictor $\eta$ and $\theta$ via the mean. More specifically, the mean $\mu$ is seen as an invertible and smooth function of the linear predictor, i.e. $$g(\mu)=\eta\ \textrm{or}\ \mu=g^{-1}(\eta)$$ Now to answer your question:

The function $g(\cdot)$ is called the link function. If the function connects $\mu$, $\eta$ and $\theta$ such that $\eta \equiv\theta$, then this link is called canonical and has the form $g=(\gamma')^{-1}$.

That's it. Then there are a number of desirable statistical properties of using the canonical link, e.g., the sufficient statistic is $X'y$ with components $\sum_i x_{ij} y_i$ for $j = 1, \dots, p$, the Newton Method and Fisher scoring for finding the ML estimator coincide, these links simplify the derivation of the MLE, they ensure that some properties of linear regression (e.g., the sum of the residuals is 0) hold up or they ensure that $\mu$ stays within the range of the outcome variable.

Hence they tend to be used by default. Note however, that there is no a priori reason why the effects in the model should be additive on the scale given by this or any other link.

• +1, this is a really nice answer, @Momo. I did find some of the equations harder to read when they were buried in the paragraphs, so I 'blocked' them out by using double dollar-signs (ie \$$). I hope that's OK (if not, you can rollback, w/ my apologies). Oct 21 '12 at 15:13 • @Momo the original question here does, however, include what Wei asked about, so it's worth pointing out that hasn't been clearly answered yet. Feb 22 '14 at 9:23 • I hope I understand your confusion correctly: In the exponential family you talk about, the canonical parameter is \theta and the canonical link is when \eta=\theta which is when g(\mu)=\theta. As also \theta=(\gamma')^{-1}(\mu) (if you calculate the expected value of the first derivative with respect to \theta of the likelihood function) the only case when \theta \equiv \mu appears when g(.)=(\gamma')^{-1}(.). – Momo Mar 14 '14 at 10:10 • Thank you very much. Using the previous example, we've that \gamma'(\theta) = \pi = \frac{exp(\theta)}{1+exp(\theta)}. Hence (\gamma')^{-1}(.) = \text{logit(.)}. As you said (I just rephrase), we only have \eta = \theta if g(.) is the canonical link, which is the logit. Then we will have \theta = logit(\pi) = \eta. So the equality between \theta and the predictor \eta only exists, if we use the canonical link function. Mar 14 '14 at 15:22 • It seems that there's a typo in key sentence of the answer: shouldn't it read "if the function connects \mu and \theta s.t. \eta \equiv \theta"? Jan 28 '15 at 9:20 Here is a little diagram inspired from MIT's 18.650 class which I find quite useful as it helps visualizing the relationships between these functions. I have used the same notation as in @momo's post: • \gamma(\theta) is the cumulant moment generating function • g(\mu) is the link function So the link function g relates the linear predictor to the mean and is required to be monotone increasing, continuously differentiable and invertible. The diagram allows to easily go from one direction to the other, for example:$$ \eta = g \left( \gamma(\theta)\right) \theta = \gamma'^{-1}\left( g^{-1}(\eta)\right)$$### Canonical link function Another way of seing what Momo has described rigorously is that when g is the canonical link function, then the function composition$$\gamma^{-1} \circ g^{-1}= \left( g \circ \gamma' \right)^{-1} = I$$is the identity and so we get$$\theta = \eta $$• should there be a prime on the gamma? i.e.$$\gamma'^{-1} \circ g^{-1}= \left( g \circ \gamma' \right)^{-1} = I Jan 21 '20 at 21:23
• Yes most likely ! Jan 22 '20 at 16:36

gung's quoted a good explanation: the canonical link possesses special theoretical properties of minimal sufficiency. This means that you can define a conditional logit model (which economists call a fixed effect model) by conditioning on the number of outcomes, but you cannot define a conditional probit model, because there is no sufficient statistics to use with the probit link.

• Can you elaborate a bit on the minimal sufficiency? By the above explanation we can still define a probit model, right? It won't be the canonical link function for sure but what's the harm in using a non-canonical link function. Jun 7 '17 at 3:02

The answers above have already covered what I want to say. Just to clarify a few points as a researcher of machine learning:

1. link function is nothing but the inverse of the activation function. For example, logit is the inverse of sigmoid, probit is the inverse of the cumulative distribution function of Gaussian.

2. If we take the parameter of the generalized linear model to only depend on $w^T x$, with $w$ being the weight vector and $x$ as the input, then the link function is called canonical.

The discussion above has nothing to do with exponential family, but a nice discussion can be found in Christopher Bishop's PRML book Chapter 4.3.6.