What's the intuition behind the canonical link function in GLM?

Question

I have already read the answer from What is the difference between a “link function” and a “canonical link function” for GLM but I think my question is different from this one. I am watching the MIT open course on GLM and the explanation of the link function is wonderful and crystal clear. What I am still confused about is why the canonical link function, canonical to the particular distribution of the canonical exponential family is defined as the inverse of $b'$?

The exponential family can always be written in the following form: $$f_{\theta}(y)=\exp \left(\frac{y \theta-b(\theta)}{\phi}+c(y, \phi)\right)$$

In the lecture, the professor explains a canonical link function $g$ is canonical to the distribution which belongs to the canonical exponential family, and the canonical exponential family is completely characterized by $b$, hence we only need to know the connection between $g$ and $b$. Then the professor moves onto say $g=(b')^{-1}$

But just why the canonical link is connected to $b$ through the inverse of the first derivative of $b$ is unclear. Why not $g=b^{-1}$?

Besides, by defining $g=(b')^{-1}$, we are able to substitute $\theta$ by $X^T\beta$ when modeling, and it is much easier to solve the optimization problem of the log-likelihood. But that means the fact that link is called canonical is because it will result in the function which links $\theta$ and $X^T\beta$ being an identity. How does making the function between $\theta$ and $X^T\beta$ an identity has anything to do with the link being canonical? i.e. canonical in what sense?

Answer 1

It is important to distinguish the distribution part and the model part (also called structural part or something).

The distribution part. Consider a distribution of the form you presented. Note that not every exponential family distribution is of this form, but this form defines a proper sub-family of the exponential family. For any exp. fam. distribution the following relation holds: $E[s(y)]=b'(\theta)$, where $s(\cdot)$ is the sufficient statistic, $E$ denotes expectation, $b(\cdot)$ is the log-partition function, and $\theta$ is the natural/canonical parameter. For our distribution, we have $s(y)=y$, and although $\theta$ and $c(\theta)$ are not exactly the natural parameter and the partition function (but their scaled versions), we still have $E[y]=b'(\theta)$. This establishes the relation $E_\theta[y]=g^{-1}(\theta)$, where we have introduced an invertible function $g$. We don't give function $g$ any name at this point, we only note that it is determined by the distribution we chose.

The model part. Now we want to build a regression or classification model, in which we want to predict outputs $y(\boldsymbol{x})$ based on inputs $\boldsymbol{x}$. We decide that $y(\boldsymbol{x})$ is distributed with the pdf $f_\theta(y)$, which implicitly depends on $x$. We decide that this dependence should be such that $\mu(\boldsymbol{x}):=E[y(\boldsymbol{x})]=h(\boldsymbol{\beta}^T \boldsymbol{x})$, where $h$ is an invertible function of our choice and is called mean function; and its inverse $h^{-1}(\cdot)$ is called link function. To make the implicit dependence of $f_\theta(y)$ on $x$ explicit, we write $\theta(\boldsymbol{x}) = g(\mu(\boldsymbol{x})) = g(h(\boldsymbol{\beta}^T \boldsymbol{x}))$.

The pointwise dependence of $\theta$ on $\boldsymbol{x}$ is the simplest when $g\equiv h^{-1}$. To highlight this exceptional property, we call $h^{-1}$ the canonical link function (as opposed to just a link function). Oh! But then $g$ , which is the same function, can also be called the canonical link function.

Update. The above should remove the confusion you express in

But just why the canonical link is connected to $b$ through the inverse of the first derivative of $b$ is unclear. Why not $g=b^{-1}$?

As to the word canonical, I think, it is not related to the "canonical form" or the "canonical parameter"; it is just the same word used in a different situation and in a different sense. You can replace the term "canonical link" with "organic link function". And among possible choices of the link function there will the "organic link function", which is organic to a given distribution, because it makes $\theta$ of this distribution equal $\boldsymbol{\beta}^T\boldsymbol{x}$.