3
$\begingroup$

I have already read the answer from What is the difference between a “link function” and a “canonical link function” for GLM but I think my question is different from this one. I am watching the MIT open course on GLM and the explanation of the link function is wonderful and crystal clear. What I am still confused about is why the canonical link function, canonical to the particular distribution of the canonical exponential family is defined as the inverse of $b'$?

The exponential family can always be written in the following form: $$f_{\theta}(y)=\exp \left(\frac{y \theta-b(\theta)}{\phi}+c(y, \phi)\right)$$

In the lecture, the professor explains a canonical link function $g$ is canonical to the distribution which belongs to the canonical exponential family, and the canonical exponential family is completely characterized by $b$, hence we only need to know the connection between $g$ and $b$. Then the professor moves onto say $g=(b')^{-1}$

But just why the canonical link is connected to $b$ through the inverse of the first derivative of $b$ is unclear. Why not $g=b^{-1}$?

Besides, by defining $g=(b')^{-1}$, we are able to substitute $\theta$ by $X^T\beta$ when modeling, and it is much easier to solve the optimization problem of the log-likelihood. But that means the fact that link is called canonical is because it will result in the function which links $\theta$ and $X^T\beta$ being an identity. How does making the function between $\theta$ and $X^T\beta$ an identity has anything to do with the link being canonical? i.e. canonical in what sense?

$\endgroup$
3
  • $\begingroup$ Please correct the typo: $c(y,\phi) \mapsto c(\theta,\phi)$. $\endgroup$ Jul 14, 2021 at 21:23
  • 1
    $\begingroup$ @paperskilltrees Hi, I think it is not a typo. $\phi$ is assumed to be known in the canonical exponential family form and it needs some function of $y$ to make sure the integral sums up to 1. $\endgroup$
    – JoZ
    Jul 14, 2021 at 21:29
  • $\begingroup$ Oh, sorry. You're right. $\endgroup$ Jul 14, 2021 at 21:30

1 Answer 1

1
$\begingroup$

It is important to distinguish the distribution part and the model part (also called structural part or something).

The distribution part. Consider a distribution of the form you presented. Note that not every exponential family distribution is of this form, but this form defines a proper sub-family of the exponential family. For any exp. fam. distribution the following relation holds: $E[s(y)]=b'(\theta)$, where $s(\cdot)$ is the sufficient statistic, $E$ denotes expectation, $b(\cdot)$ is the log-partition function, and $\theta$ is the natural/canonical parameter. For our distribution, we have $s(y)=y$, and although $\theta$ and $c(\theta)$ are not exactly the natural parameter and the partition function (but their scaled versions), we still have $E[y]=b'(\theta)$. This establishes the relation $E_\theta[y]=g^{-1}(\theta)$, where we have introduced an invertible function $g$. We don't give function $g$ any name at this point, we only note that it is determined by the distribution we chose.

The model part. Now we want to build a regression or classification model, in which we want to predict outputs $y(\boldsymbol{x})$ based on inputs $\boldsymbol{x}$. We decide that $y(\boldsymbol{x})$ is distributed with the pdf $f_\theta(y)$, which implicitly depends on $x$. We decide that this dependence should be such that $\mu(\boldsymbol{x}):=E[y(\boldsymbol{x})]=h(\boldsymbol{\beta}^T \boldsymbol{x})$, where $h$ is an invertible function of our choice and is called mean function; and its inverse $h^{-1}(\cdot)$ is called link function. To make the implicit dependence of $f_\theta(y)$ on $x$ explicit, we write $\theta(\boldsymbol{x}) = g(\mu(\boldsymbol{x})) = g(h(\boldsymbol{\beta}^T \boldsymbol{x}))$.

The pointwise dependence of $\theta$ on $\boldsymbol{x}$ is the simplest when $g\equiv h^{-1}$. To highlight this exceptional property, we call $h^{-1}$ the canonical link function (as opposed to just a link function). Oh! But then $g$ , which is the same function, can also be called the canonical link function.

Update. The above should remove the confusion you express in

But just why the canonical link is connected to $b$ through the inverse of the first derivative of $b$ is unclear. Why not $g=b^{-1}$?

As to the word canonical, I think, it is not related to the "canonical form" or the "canonical parameter"; it is just the same word used in a different situation and in a different sense. You can replace the term "canonical link" with "organic link function". And among possible choices of the link function there will the "organic link function", which is organic to a given distribution, because it makes $\theta$ of this distribution equal $\boldsymbol{\beta}^T\boldsymbol{x}$.

$\endgroup$
5
  • $\begingroup$ Hi, thanks for your answer! Yes, you indeed answered my question of why the canonical link is $g^{-1}$. A minor question on your explanation, why do you change $b'$ into $g^{-1}$, aren't they supposed to mean the same thing? $\endgroup$
    – JoZ
    Jul 14, 2021 at 23:15
  • $\begingroup$ @JoZ They are. I am simply following authors I read. I guess, the point of introducing this additional notation is to abstract from the details of the distribution and to take away only that there is a relation between $\theta$ and $\mu$. Also, $b'^{(-1)}$ does not look nice. You see different notation in different texts, but several books that I came across do that. $\endgroup$ Jul 14, 2021 at 23:30
  • 1
    $\begingroup$ @paperskilltress Thank you so much for being patient. I am just personally easily get confused with a lot of inverses get involved in, but I think I get the idea from your answer! $\endgroup$
    – JoZ
    Jul 14, 2021 at 23:32
  • $\begingroup$ The usual alternative term to "canonical link" would be "natural link" rather then "organic link" $\endgroup$
    – Glen_b
    Jul 15, 2021 at 12:43
  • $\begingroup$ @Glen_b I intentionally chose an unused adjective to demonstrate that it is a separate unrelated notion: "natural" is already used in "natural parameter" and "natural family". If you are saying that "natural link" is used interchangeably with "canonical link" (just as "natural parameter"="canonical parameter"), then thank you, it is good to know! $\endgroup$ Jul 15, 2021 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.