What is the difference between a "link function" and a "canonical link function" for GLM What's the difference between terms 'link function' and 'canonical link function'? Also, are there any (theoretical) advantages of using one over the other?
For example, a binary response variable can be modeled using many link functions such as logit, probit, etc. But, logit here is considered the "canonical" link function.
 A: The above answers are more intuitive, so I try more rigor.
What is a GLM?
Let $Y=(y,\mathbf{x})$ denote a set of a response $y$ and $p$-dimensional covariate vector $\mathbf{x}=(x_1,\dots,x_p)$ with expected value $E(y)=\mu$. For $i=1,\dots,n$ independent observations, the distribution of each $y_i$ is an exponential family with density
$$
f(y_i;\theta_i,\phi)=\exp\left(\frac{y_i\theta_i-\gamma(\theta_i)}{\phi}+\tau(y_i,\phi)\right) = \alpha(y_i, \phi)\exp\left(\frac{y_i\theta_i-\gamma(\theta_i)}{\phi}\right)
$$
Here, the parameter of interest (natural or canonical parameter) is $\theta_i$, $\phi$ is a scale parameter (known or seen as a nuisance) and $\gamma$ and $\tau$ are known functions. The $n$-dimensional vectors of fixed input values for the $p$ explanatory variables are denoted by $\mathbf{x}_1,\dots,\mathbf{x}_p$. We assume that the input vectors influence (1) only via a linear function, the linear predictor,
$$
\eta_i=\beta_0+\beta_1x_{i1}+\dots+\beta_px_{ip}
$$
upon which $\theta_i$ depends. As it can be shown that $\theta=(\gamma')^{-1}(\mu)$, this dependency is established by connecting the linear predictor $\eta$ and $\theta$ via the mean. More specifically, the mean $\mu$ is seen as an invertible and smooth function of the linear predictor, i.e.
$$
g(\mu)=\eta\ \textrm{or}\ \mu=g^{-1}(\eta)
$$
Now to answer your question:
The function $g(\cdot)$ is called the link function. If the function connects $\mu$, $\eta$ and $\theta$ such that $\eta \equiv\theta$, then this link is called canonical and has the form $g=(\gamma')^{-1}$.
That's it. Then there are a number of desirable statistical properties of using the canonical link, e.g., the sufficient statistic is $X'y$ with
components $\sum_i x_{ij} y_i$ for $j = 1, \dots, p$, the Newton Method and Fisher scoring for finding the ML estimator coincide, these links simplify the derivation of the MLE, they ensure that some properties of linear regression (e.g., the sum of the residuals is 0) hold up or they ensure that $\mu$ stays within the range of the outcome variable.
Hence they tend to be used by default. Note however, that there is no a priori reason why the effects in the model should be additive on the scale given by this or any other link.
A: Here is a little diagram inspired from MIT's 18.650 class which I find quite useful as it helps visualizing the relationships between these functions. I have used the same notation as in @momo's post: 



*

*$\gamma(\theta)$ is the cumulant moment generating function

*$g(\mu)$ is the link function


So the link function $g$ relates the linear predictor to the mean and is required to be monotone increasing, continuously differentiable and invertible. 
The diagram allows to easily go from one direction to the other, for example: 
$$ \eta = g \left( \gamma(\theta)\right)$$
$$ \theta = \gamma'^{-1}\left( g^{-1}(\eta)\right)$$  
Canonical link function
Another way of seing what Momo has described rigorously is that when $g$ is the canonical link function, then the function composition 
$$\gamma^{-1} \circ g^{-1}= \left( g \circ \gamma' \right)^{-1} = I$$ is the identity and so we get 
$$\theta = \eta  $$
A: gung's quoted a good explanation: the canonical link possesses special theoretical properties of minimal sufficiency. This means that you can define a conditional logit model (which economists call a fixed effect model) by conditioning on the number of outcomes, but you cannot define a conditional probit model, because there is no sufficient statistics to use with the probit link.
A: The answers above have already covered what I want to say. Just to clarify a few points as a researcher of machine learning: 


*

*link function is nothing but the inverse of the activation function. For example, logit is the inverse of sigmoid, probit is the inverse of the cumulative distribution function of Gaussian.

*If we take the parameter of the generalized linear model to only depend on $w^T x$, with $w$ being the weight vector and $x$ as the input, then the link function is called canonical.
The discussion above has nothing to do with exponential family, but a nice discussion can be found in Christopher Bishop's PRML book Chapter 4.3.6.
