# How do we apply derivatives to the vector version of canonical exponential-family log-likelihood?

Background (sorry, a bit long):

The log-likelihood of observations from a (simplified) exponential-family distrib look like:

$$l(y) = \sum{(y_i\theta_i - b(\theta_i))}$$

and suppose that the linear predictor is canonically linked to our model and that **we have just a single parameter $\beta$ ** - i.e. $\beta$ is a single parameter, not a vector :

$$\theta_i = x_i\beta$$

and then, since its known that $\frac{db(\theta_i)}{d\theta_i} = \mu_i$:

$$\frac{dl(y)}{d\beta} = \sum{(y_ix_i - y_i\mu_i)}$$

How do you do the same when $\beta$ is a vector of $p$ parameters? Princeton claim that then its a very elegant matrix equation:

$$\frac{dl(y)}{d\beta} = X'(y-\mu)$$

How do they do that???

(the reason its important is because that means that when the link is canonical then at $\beta_{MLE}$ the vector of means $\mu$ is the same as the vector of $y$.)

Be careful with notation. It should be:

$$\frac{dl(y)}{d\beta} = \sum_{i=1}^{n}(y_ix_i - x_i\mu_i) = \sum_{i=1}^{n}x_i(y_i - \mu_i)$$ In the multivariate case,

$$\theta_i = {\bf X}^{T}{\bf \beta}$$ where ${\bf X}$ is $p \times N$ and ${\bf \beta}$ is $p \times 1$. So that each observation, i, still only has one $\theta_i$ and one $\mu_i$ associated with it. Therefore nothing in your $\frac{dl(y)}{d\beta}$ equation changes.

If we write it out in matrix notation:

$$\frac{\partial l({\bf y})}{\partial {\bf \beta}} = \sum_{i=1}^{n}x_i(y_i - \mu_i) = {\bf X}^{T}({\bf y - \mu})$$

where ${\bf y}$ is a $N \times 1$ vector of responses and ${\bf \mu}$ is a $N \times 1$ vector of means.

Note that it's interesting to see in the vectorized form that under the MLE, $\frac{l({\bf y})}{\partial {\bf \beta}} = 0$ and therefore the residuals are perpendicular to the column space of X, i.e. ${\bf y - \mu} \perp col({\bf X})$.