Is this GLM for the Poisson distribution correct?

Question

I'm currently taking a machine learning class, and one of the problem set questions is to construct a GLM that models the Poisson distribution, defined as

$$P(y;\lambda) = e^{-\lambda}\frac{\lambda^y}{y!}$$

According to the class lectures, a GLM takes the form of $$P(y;\eta) = b(y)\exp(\eta^TT(y) - a(\eta))$$

Here was my attempt to derive that form from the given Poisson distribution.

$$\begin{align} P(y; \lambda) & = \exp(-\lambda)\frac{\lambda^y}{y!} \\ & = \exp(-\lambda)\exp(\log(\frac{\lambda^y}{y!})) \\ & = \exp(-\lambda)\exp(\log(\lambda^y) - \log(y!)) \\ & = \exp(-\lambda)\exp(y\log(\lambda) - \log(y!)) \end{align}$$

From this, I got the following results:

$$\begin{align} b(y) & = \exp(-\lambda) \\ \eta & = \log(\lambda) \\ T(y) & = y \\ a(\eta) & = \log(y!) \end{align}$$

Do these results look correct for the Poisson GLM? I've seen some varying other answers online, such as one source saying that $b(y)$ in this case is actually $\frac{1}{y!}$.

Also, how would you find the canonical link function from this? I know from experience that the function is $\log(y)$, but I'm interested in how that's found.

Answer 1

According to the class lectures, a GLM takes the form of $$P(y;\eta) = b(y)\exp(\eta^TT(y) - a(\eta))$$

That is not right. If anything, that is an attempt to characterize the canonical GLM form. A GLM will not always give a probability.

A GLM is just a mean-variance relationship $V(\mu)$ and a link function $g$. The GLM is estimated by solving the following estimating equation:

$$ 0 = \mathbf{D}^T V(\mu)^{-1} \left( Y - g^{-1}(\mathbf{X}^T\beta)\right) $$

where $$ \mathbf{D} = \frac{\partial}{\partial \beta}g^{-1}(\mathbf{X}^T\beta) $$

Note that you can use any old variance function you might like and any old link function. That comprises a GLM which may not in fact maximize any likelihood at all.

With a regular exponential family, the score equations (derivative of log likelihood) are of the form:

$$ S(\beta) = \mathbf{X}^T \left( Y - g^{-1}(\mathbf{X}^T\beta)\right)$$

By choosing the canonical-link $g$, you find that $\mathbf{D}^T V(\mu)^{-1} = \mathbf{I}$ . For instance, with the Poisson density, the mean-variance relationship is that the mean is equal to the variance. You can solve the differential equation to find that the link is the log. In other words:

$$\mathbf{D} = \mathbf{X^T}g^{-1}(\mathbf{X^T}\beta) \text{ and } V(\mu) = \text{diag}(\mathbf{X^T}\beta)$$

So returning to the Poisson density, observe:

$$\begin{eqnarray} L(Y, \lambda) &=& \exp(-\lambda) \lambda^y / y! \\ \ell (Y, \lambda) &=& -\lambda + y \log( \lambda) - \log(y!) \\ \end{eqnarray}$$

at which point it's apparent the log-factorial term is an ancillary parameter

$$\begin{eqnarray} \dot{\ell} (Y, \lambda) &=& 1 + y / \lambda \\ &=& \frac{1}{\lambda} (y - \lambda) \\ \end{eqnarray}$$

which is the canonical representation of precision times the difference of response and expectation. (Precision is inverse of variance). And again, if you solve the differential equation, the log is found as the canonical link. Having identified log as the canonical link $g$, replace $\lambda$ with $g^{-1}(\mathbf{X}^T\beta)$ and you now understand how the maximum likelihood is a special case of a GLM and vice versa which is a powerful result of exponential families and canonical forms.