100
$\begingroup$

What's the difference between terms 'link function' and 'canonical link function'? Also, are there any (theoretical) advantages of using one over the other?

For example, a binary response variable can be modeled using many link functions such as logit, probit, etc. But, logit here is considered the "canonical" link function.

Improve this question
$\endgroup$
1
  • 12
    $\begingroup$ I discuss link functions extensively here: Difference between logit and probit models, focusing on regression for a binary response variable. Although only a little of that discussion focuses on the meaning of a link function's being 'canonical', it may nonetheless be helpful to read. Note that to understand the distinction b/t & advantages of a canonical vs non-canonical link function requires going fairly deep into the math underlying the GLiM. $\endgroup$
    – gung - Reinstate Monica
    Commented Oct 21, 2012 at 14:37

4 Answers 4

Reset to default
99
$\begingroup$

The above answers are more intuitive, so I try more rigor.

What is a GLM?

Let $Y=(y,\mathbf{x})$ denote a set of a response $y$ and $p$-dimensional covariate vector $\mathbf{x}=(x_1,\dots,x_p)$ with expected value $E(y)=\mu$. For $i=1,\dots,n$ independent observations, the distribution of each $y_i$ is an exponential family with density $$ f(y_i;\theta_i,\phi)=\exp\left(\frac{y_i\theta_i-\gamma(\theta_i)}{\phi}+\tau(y_i,\phi)\right) = \alpha(y_i, \phi)\exp\left(\frac{y_i\theta_i-\gamma(\theta_i)}{\phi}\right) $$ Here, the parameter of interest (natural or canonical parameter) is $\theta_i$, $\phi$ is a scale parameter (known or seen as a nuisance) and $\gamma$ and $\tau$ are known functions. The $n$-dimensional vectors of fixed input values for the $p$ explanatory variables are denoted by $\mathbf{x}_1,\dots,\mathbf{x}_p$. We assume that the input vectors influence (1) only via a linear function, the linear predictor, $$ \eta_i=\beta_0+\beta_1x_{i1}+\dots+\beta_px_{ip} $$ upon which $\theta_i$ depends. As it can be shown that $\theta=(\gamma')^{-1}(\mu)$, this dependency is established by connecting the linear predictor $\eta$ and $\theta$ via the mean. More specifically, the mean $\mu$ is seen as an invertible and smooth function of the linear predictor, i.e. $$ g(\mu)=\eta\ \textrm{or}\ \mu=g^{-1}(\eta) $$ Now to answer your question:

The function $g(\cdot)$ is called the link function. If the function connects $\mu$, $\eta$ and $\theta$ such that $\eta \equiv\theta$, then this link is called canonical and has the form $g=(\gamma')^{-1}$.

That's it. Then there are a number of desirable statistical properties of using the canonical link, e.g., the sufficient statistic is $X'y$ with components $\sum_i x_{ij} y_i$ for $j = 1, \dots, p$, the Newton Method and Fisher scoring for finding the ML estimator coincide, these links simplify the derivation of the MLE, they ensure that some properties of linear regression (e.g., the sum of the residuals is 0) hold up or they ensure that $\mu$ stays within the range of the outcome variable.

Hence they tend to be used by default. Note however, that there is no a priori reason why the effects in the model should be additive on the scale given by this or any other link.

Improve this answer
$\endgroup$
8
  • 8
    $\begingroup$ +1, this is a really nice answer, @Momo. I did find some of the equations harder to read when they were buried in the paragraphs, so I 'blocked' them out by using double dollar-signs (ie \$$). I hope that's OK (if not, you can rollback, w/ my apologies). $\endgroup$
    – gung - Reinstate Monica
    Commented Oct 21, 2012 at 15:13
  • $\begingroup$ Does a canonical link function always exist in GLM? What are the necessary conditions for it to exist? Thanks. $\endgroup$
    – Wei
    Commented Feb 21, 2014 at 12:18
  • 1
    $\begingroup$ I hope I understand your confusion correctly: In the exponential family you talk about, the canonical parameter is $\theta$ and the canonical link is when $\eta=\theta$ which is when $g(\mu)=\theta$. As also $\theta=(\gamma')^{-1}(\mu)$ (if you calculate the expected value of the first derivative with respect to $\theta$ of the likelihood function) the only case when $\theta \equiv \mu$ appears when $g(.)=(\gamma')^{-1}(.)$. $\endgroup$
    – Momo
    Commented Mar 14, 2014 at 10:10
  • 2
    $\begingroup$ Thank you very much. Using the previous example, we've that $\gamma'(\theta) = \pi = \frac{exp(\theta)}{1+exp(\theta)}$. Hence $(\gamma')^{-1}(.) = \text{logit(.)}$. As you said (I just rephrase), we only have $\eta = \theta$ if $g(.)$ is the canonical link, which is the logit. Then we will have $\theta = logit(\pi) = \eta$. So the equality between $\theta$ and the predictor $\eta$ only exists, if we use the canonical link function. $\endgroup$
    – Druss2k
    Commented Mar 14, 2014 at 15:22
  • 2
    $\begingroup$ It seems that there's a typo in key sentence of the answer: shouldn't it read "if the function connects $\mu$ and $\theta$ s.t. $\eta \equiv \theta$"? $\endgroup$
    – Leo Alekseyev
    Commented Jan 28, 2015 at 9:20
29
$\begingroup$

Here is a little diagram inspired from MIT's 18.650 class which I find quite useful as it helps visualizing the relationships between these functions. I have used the same notation as in @momo's post:

how are parameter, mean and linear predictor related

  • $\gamma'(\theta)$ is the cumulant moment generating function
  • $g(\mu)$ is the link function

So the link function $g$ relates the linear predictor to the mean and is required to be monotone increasing, continuously differentiable and invertible.

The diagram allows to easily go from one direction to the other, for example:

$$ \eta = g \left( \gamma'(\theta)\right)$$ $$ \theta = \gamma'^{-1}\left( g^{-1}(\eta)\right)$$

Canonical link function

Another way of seing what Momo has described rigorously is that when $g$ is the canonical link function, then the function composition $$\gamma'^{-1} \circ g^{-1}= \left( g \circ \gamma' \right)^{-1} = I$$ is the identity and so we get $$\theta = \eta $$

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ should there be a prime on the gamma? i.e. $$\gamma'^{-1} \circ g^{-1}= \left( g \circ \gamma' \right)^{-1} = I$$ $\endgroup$
    – Stan Shunpike
    Commented Jan 21, 2020 at 21:23
  • 1
    $\begingroup$ Yes most likely ! $\endgroup$
    – Xavier Bourret Sicotte
    Commented Jan 22, 2020 at 16:36
19
$\begingroup$

gung's quoted a good explanation: the canonical link possesses special theoretical properties of minimal sufficiency. This means that you can define a conditional logit model (which economists call a fixed effect model) by conditioning on the number of outcomes, but you cannot define a conditional probit model, because there is no sufficient statistics to use with the probit link.

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ Can you elaborate a bit on the minimal sufficiency? By the above explanation we can still define a probit model, right? It won't be the canonical link function for sure but what's the harm in using a non-canonical link function. $\endgroup$
    – pikachuchameleon
    Commented Jun 7, 2017 at 3:02
2
$\begingroup$

The answers above have already covered what I want to say. Just to clarify a few points as a researcher of machine learning:

  1. link function is nothing but the inverse of the activation function. For example, logit is the inverse of sigmoid, probit is the inverse of the cumulative distribution function of Gaussian.

  2. If we take the parameter of the generalized linear model to only depend on $w^T x$, with $w$ being the weight vector and $x$ as the input, then the link function is called canonical.

The discussion above has nothing to do with exponential family, but a nice discussion can be found in Christopher Bishop's PRML book Chapter 4.3.6.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.