5
$\begingroup$

Here's a question from my problem sheet.

For the normal linear model, verify that the MLEs $\boldsymbol{\hat{\beta}}$ and $\tilde{\sigma}^2$ are maximal values for $\ell(\beta, \sigma^2;\mathbf{y})$ with respect to $\beta$ and $\sigma$, where $\ell$ denotes the log likelihood. What is the maximum value of the likelihood $L(\beta, \sigma^2,y)$? That is: Compute $\max_{\beta,\sigma^2}L(\boldsymbol{\beta},\sigma^2;\mathbf{y})$, where $\mathbf{y}$ is the vector of observations.

I have tried to solve this question but I am confused at the solution, mostly the Hessian.

The Hessian $H(\boldsymbol{\beta},\sigma^2)$ gives \begin{pmatrix} \dfrac{\partial}{\partial \boldsymbol{\beta}^T} \left[ \dfrac{\partial \ell(\boldsymbol{\beta},\sigma^2;\mathbf{y})}{\partial \boldsymbol{\beta}} \right] & \dfrac{\partial}{\partial \sigma^2} \left[ \dfrac{\partial \ell(\boldsymbol{\beta},\sigma^2;\mathbf{y})}{\partial \boldsymbol{\beta}}\right] \\ \dfrac{\partial}{\partial \boldsymbol{\beta}^T} \left[ \dfrac{\partial \ell(\boldsymbol{\beta},\sigma^2;\mathbf{y})}{\partial \sigma^2} \right] & \dfrac{\partial}{\partial \boldsymbol{\sigma}^2} \left[ \dfrac{\partial \ell(\boldsymbol{\beta},\sigma^2;\mathbf{y})}{\partial \sigma^2} \right] \\ \end{pmatrix} according to the answer.

I have two questions:

  1. How do I know when I need to use the transpose? e.g. why isn't the 1,1th element of the Hessian matrix just $\dfrac{\partial}{\partial \boldsymbol{\beta}}\left[\dfrac{\partial \ell(\boldsymbol{\beta},\sigma^2;\mathbf{y})}{\partial \boldsymbol{\beta}}\right]$?

  2. Why does the 2,1th element of the Hessian have to have the partial differential with respect to $\boldsymbol{\beta}^{T}$ on the outside, not just $\boldsymbol{\beta}$?

$\endgroup$
4
$\begingroup$

You have to remember that since $\pmb{\beta} \in \Re^{n \times 1}$ is a vector, partial derivatives you described are vectors, and matrices. Especially the hessian

$$H(\pmb{\beta}, \sigma^2) = \begin{pmatrix} \frac{\partial}{\partial \boldsymbol{\beta}^{T}}[\frac{\partial l(\boldsymbol{\beta},\sigma^{2};\mathbf{y})}{\partial \boldsymbol{\beta}}] & \frac{\partial}{\partial \sigma^{2}}[\frac{\partial l(\boldsymbol{\beta},\sigma^{2};\mathbf{y})}{\partial \boldsymbol{\beta}}]\\ \frac{\partial}{\partial \boldsymbol{\beta}^{T}}[\frac{\partial l(\boldsymbol{\beta},\sigma^{2};\mathbf{y})}{\partial \sigma^{2}}] & \frac{\partial}{\partial \boldsymbol{\sigma}^{2}}[\frac{\partial l(\boldsymbol{\beta},\sigma^{2};\mathbf{y})}{\partial \sigma^{2}}] \\ \end{pmatrix} \in \Re^{(n+1) \times (n+1)}$$

and since $\pmb{\beta}$ is a column vector, we have

$$ \frac{\partial}{\partial \boldsymbol{\beta}^{T}}\Big[\frac{\partial l(\boldsymbol{\beta},\sigma^{2};\mathbf{y})}{\partial \boldsymbol{\beta}}\Big] \in \Re^{n \times n} \text{, is a matrix}$$

$$\frac{\partial}{\partial \sigma^{2}}\Big[\frac{\partial l(\boldsymbol{\beta},\sigma^{2};\mathbf{y})}{\partial \boldsymbol{\beta}}\Big] \in \Re^{n \times 1} \text{, is a column vector}$$

$$\frac{\partial}{\partial \boldsymbol{\beta}^{T}}\Big[\frac{\partial l(\boldsymbol{\beta},\sigma^{2};\mathbf{y})}{\partial \sigma^{2}}\Big] \in \Re^{1 \times n} \text{, is a row vector}$$

So we simply take partial derivates w.r.t $\pmb{\beta}$ or $\pmb{\beta^T}$ so that they "fit" in a matrix. To visualize it better

$$\frac{\partial}{\partial \boldsymbol{\beta}^{T}}\Big[\frac{\partial l(\boldsymbol{\beta},\sigma^{2};\mathbf{y})}{\partial \boldsymbol{\beta}}\Big] = \frac{\partial}{\partial \boldsymbol{\beta}^{T}} \begin{pmatrix} \frac{\partial l(\boldsymbol{\beta},\sigma^{2};\mathbf{y})}{\partial \boldsymbol{\beta_1}} \\ \vdots \\ \frac{\partial l(\boldsymbol{\beta},\sigma^{2};\mathbf{y})}{\partial \boldsymbol{\beta_n}} \end{pmatrix} = \begin{pmatrix} \frac{\partial l(\boldsymbol{\beta},\sigma^{2};\mathbf{y})}{\partial \boldsymbol{\beta_1}\partial \boldsymbol{\beta_1}}& \frac{\partial l(\boldsymbol{\beta},\sigma^{2};\mathbf{y})}{\partial \boldsymbol{\beta_1}\partial \boldsymbol{\beta_2}}& \dots & \frac{\partial l(\boldsymbol{\beta},\sigma^{2};\mathbf{y})}{\partial \boldsymbol{\beta_1}\partial \boldsymbol{\beta_n}} \\ \frac{\partial l(\boldsymbol{\beta},\sigma^{2};\mathbf{y})}{\partial \boldsymbol{\beta_2}\partial \boldsymbol{\beta_1}} & \ddots & & \vdots \\ \vdots& \\ \frac{\partial l(\boldsymbol{\beta},\sigma^{2};\mathbf{y})}{\partial \boldsymbol{\beta_n}\partial \boldsymbol{\beta_1}} & \dots & & \frac{\partial l(\boldsymbol{\beta},\sigma^{2};\mathbf{y})}{\partial \boldsymbol{\beta_n}\partial \boldsymbol{\beta_n}} \end{pmatrix}$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.