Background (sorry, a bit long):

The log-likelihood of observations from a (simplified) exponential-family distrib look like:

$$l(y) = \sum{(y_i\theta_i - b(\theta_i))}$$

and suppose that the linear predictor is canonically linked to our model and that **we have just a single parameter $\beta$ ** - i.e. $\beta$ is a single parameter, not a vector :

$$ \theta_i = x_i\beta $$

and then, since its known that $\frac{db(\theta_i)}{d\theta_i} = \mu_i$:

$$ \frac{dl(y)}{d\beta} = \sum{(y_ix_i - y_i\mu_i)} $$

How do you do the same when $\beta$ is a vector of $p$ parameters? Princeton claim that then its a very elegant matrix equation:

$$ \frac{dl(y)}{d\beta} = X'(y-\mu) $$

How do they do that???

(the reason its important is because that means that when the link is canonical then at $\beta_{MLE}$ the vector of means $\mu$ is the same as the vector of $y$.)


Be careful with notation. It should be:

$$\frac{dl(y)}{d\beta} = \sum_{i=1}^{n}(y_ix_i - x_i\mu_i) = \sum_{i=1}^{n}x_i(y_i - \mu_i) $$ In the multivariate case,

$$\theta_i = {\bf X}^{T}{\bf \beta}$$ where ${\bf X}$ is $p \times N$ and ${\bf \beta}$ is $p \times 1$. So that each observation, i, still only has one $\theta_i$ and one $\mu_i$ associated with it. Therefore nothing in your $\frac{dl(y)}{d\beta}$ equation changes.

If we write it out in matrix notation:

$$\frac{\partial l({\bf y})}{\partial {\bf \beta}} = \sum_{i=1}^{n}x_i(y_i - \mu_i) = {\bf X}^{T}({\bf y - \mu})$$

where ${\bf y}$ is a $N \times 1$ vector of responses and ${\bf \mu}$ is a $N \times 1$ vector of means.

Note that it's interesting to see in the vectorized form that under the MLE, $\frac{l({\bf y})}{\partial {\bf \beta}} = 0$ and therefore the residuals are perpendicular to the column space of X, i.e. ${\bf y - \mu} \perp col({\bf X})$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.