1
$\begingroup$

As part of a proof, I need to take the first derivative of the log of the following multivariate normal density: $(2\pi)^{-k/2} |\Sigma|^{-1/2} \exp\left(\frac{-1}{2} x'\Sigma^{-1}x\right)$.

In this case, the mean is zero and $\Sigma$ has an exchangeable correlation structure, thus $\Sigma = D A(\rho)D$, where $D$ is a diagonal matrix with entries $d = (\sigma_1, \ldots , \sigma_k)$ and $A(\rho) = (1-\rho)I + \rho jj^\top$, where $I$ is the $k\times k$ identity matrix and $j = (1,\ldots,1)^\top$ is a vector of $1$s (of length $k$). I need to take derivatives with respect to $\sigma_j^2$, where $j \in \{1,\ldots,k\}$.

Clearly, the first algebraic steps are:

$$\frac{\partial}{\partial \sigma_j^2} \log \left[(2\pi)^{-k/2} |\Sigma|^{-1/2} \exp\left( \frac{-1}{2} x^{\top} \Sigma^{-1} x \right) \right]$$

$$= \frac{\partial}{\partial \sigma_j^2} \left[ \frac{-1}{2} \log |\Sigma| - \frac{-1}{2} x^\top \Sigma^{-1} x \right],$$

but I am getting stuck at the point where the derivative actually needs to be taken. How can I take the derivative with respect to $\sigma_j^2$?

I would appreciate any help on this! Eventually working towards showing that the first derivative (with respect to $\sigma^2$) is bounded, but for now, just need to derive a proper expression for the first derivative.

$\endgroup$
1
  • 2
    $\begingroup$ Although it might look daunting, in this case it's actually easy to express both $\log|\Sigma|$ and $\Sigma^{-1}$ explicitly in terms of the $\sigma_j.$ The problem reduces to a univariate calculation, so try it with $k=1$ variable first. $\endgroup$
    – whuber
    Commented Jun 1, 2023 at 21:35

2 Answers 2

2
$\begingroup$

Denote $\begin{bmatrix} \sigma_1 & \cdots & \sigma_k\end{bmatrix}'$ by $d$, $\operatorname{diag}(x_1, \ldots, x_k)$ by $X$. Since $|\Sigma| = |D|^2|A(\rho)| = \prod\limits_{i = 1}^k\sigma_i^2|A(\rho)|$ and $x'D = d'X$, the log-likelihood function is \begin{align} l(\sigma_1^2, \ldots, \sigma_k^2) &= -\frac{1}{2}\log|\Sigma| - \frac{1}{2}x'DA(\rho)^{-1}Dx \\ &= -\frac{1}{2}\sum_{i = 1}^k\log\sigma_i^2 - \frac{1}{2}d'XA(\rho)^{-1}Xd + \text{constant}. \end{align} It thus follows by the quadratic form differentiation formula $\partial z'Mz/\partial z = 2Mz$ and the chain rule that \begin{align} \frac{\partial l(\sigma_1^2, \ldots, \sigma_k^2)}{\partial \sigma_j^2} &= -\frac{1}{2\sigma_j^2} - \frac{1}{\sigma_j}e_j'XA(\rho)^{-1}Xd \\ &= -\frac{1}{2\sigma_j^2} - \frac{1}{\sigma_j} \begin{bmatrix}0 & \cdots & x_j & \cdots & 0\end{bmatrix} A(\rho)^{-1}\begin{bmatrix} x_1\sigma_1 \\ \vdots \\ x_k\sigma_k \end{bmatrix}, \tag{1} \end{align} where $e_j$ is the $k$-long column vector with all entries $0$ but the $j$-th entry $1$.

As $A(\rho) = (1 - \rho)I + \rho jj'$, the Sherman-Morrison formula gives \begin{align} A(\rho)^{-1} &= (1 - \rho)^{-1}I - (1 - \rho)^{-2}\frac{\rho jj'}{1 + (1 - \rho)^{-1}\rho k} \\ &= (1 - \rho)^{-1}\left[I - \frac{\rho jj'}{1 + (k - 1)\rho}\right]. \tag{2} \end{align} Substituting $(2)$ into $(1)$ yields \begin{align} \frac{\partial l(\sigma_1^2, \ldots, \sigma_k^2)}{\partial \sigma_j^2} = -\frac{1}{2\sigma_j^2} - \frac{x_j^2}{1 - \rho} + \frac{x_j\rho}{(1 - \rho)\sigma_j}\frac{\sum_i x_i\rho_i}{1 + (k - 1)\rho}. \end{align}

$\endgroup$
3
  • $\begingroup$ this is really helpful, thank you! I think you forgot the $\log$ when you go from $-\frac{1}{2} log \left| \Sigma \right|$ to $-\frac{1}{2} \sum_{i=1}^k \sigma_i^2$, right? I think it should be $-\frac{1}{2} \sum_{i=1}^k \log(\sigma_i^2)$, whose derivative is $-\frac{1}{2\sigma_i^2}$ instead of $\frac{1}{2}$. $\endgroup$
    – bob
    Commented Jun 4, 2023 at 19:27
  • $\begingroup$ Also, if I'm not mistaken, I believe the result of the Sherman-Morrison formula (inside the brackets $[]$)should have a $-$ instead of a $+$, correct? $\endgroup$
    – bob
    Commented Jun 4, 2023 at 20:03
  • $\begingroup$ Yes, I will edit later. $\endgroup$
    – Zhanxiong
    Commented Jun 4, 2023 at 22:57
3
$\begingroup$

Not a full solution, but a few hints to help you on your way:

$$\log|\Sigma| =2\log|D|+\log|A|$$

$$\Sigma^{-1}=D^{-1}A^{-1}D^{-1}$$

$$A^{-1}=\frac{1}{1-\rho}\left(I-\frac{\rho}{1+(k-1)\rho}jj^\top\right)$$

$$x^\top D^{-1} j j^\top D^{-1} x = \left(j^\top D^{-1} x\right)^2 =\left( \sum_a \sigma_a^{-1}x_a\right)^2.$$

$\endgroup$
3
  • $\begingroup$ Thanks! I working with this and it is helpful. Could you point me to a source providing these identities, or something that can be cited? (especially the inverse formula) $\endgroup$
    – bob
    Commented Jun 2, 2023 at 16:29
  • 1
    $\begingroup$ Any basic linear algebra book will state and prove three of these. The fourth, about $A^{-1},$ can be derived in many ways and follows from various considerations (consider the eigenspaces and eigenvalues of $A,$ for instance, which can be found by inspection) -- but the acid test of any matrix inversion formula is to multiply the putative inverse $A^{-1}$ by the original matrix $A$ and check that the identity matrix results. That doesn't even require a citation, really. If you want one, check out the Sherman-Morrison formula. $\endgroup$
    – whuber
    Commented Jun 2, 2023 at 16:45
  • 1
    $\begingroup$ Thanks @whuber! $\endgroup$
    – bob
    Commented Jun 12, 2023 at 16:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.