# First derivative of multivariate normal density with exchangeable correlation structure

As part of a proof, I need to take the first derivative of the log of the following multivariate normal density: $$(2\pi)^{-k/2} |\Sigma|^{-1/2} \exp\left(\frac{-1}{2} x'\Sigma^{-1}x\right)$$.

In this case, the mean is zero and $$\Sigma$$ has an exchangeable correlation structure, thus $$\Sigma = D A(\rho)D$$, where $$D$$ is a diagonal matrix with entries $$d = (\sigma_1, \ldots , \sigma_k)$$ and $$A(\rho) = (1-\rho)I + \rho jj^\top$$, where $$I$$ is the $$k\times k$$ identity matrix and $$j = (1,\ldots,1)^\top$$ is a vector of $$1$$s (of length $$k$$). I need to take derivatives with respect to $$\sigma_j^2$$, where $$j \in \{1,\ldots,k\}$$.

Clearly, the first algebraic steps are:

$$\frac{\partial}{\partial \sigma_j^2} \log \left[(2\pi)^{-k/2} |\Sigma|^{-1/2} \exp\left( \frac{-1}{2} x^{\top} \Sigma^{-1} x \right) \right]$$

$$= \frac{\partial}{\partial \sigma_j^2} \left[ \frac{-1}{2} \log |\Sigma| - \frac{-1}{2} x^\top \Sigma^{-1} x \right],$$

but I am getting stuck at the point where the derivative actually needs to be taken. How can I take the derivative with respect to $$\sigma_j^2$$?

I would appreciate any help on this! Eventually working towards showing that the first derivative (with respect to $$\sigma^2$$) is bounded, but for now, just need to derive a proper expression for the first derivative.

• Although it might look daunting, in this case it's actually easy to express both $\log|\Sigma|$ and $\Sigma^{-1}$ explicitly in terms of the $\sigma_j.$ The problem reduces to a univariate calculation, so try it with $k=1$ variable first.
– whuber
Commented Jun 1, 2023 at 21:35

Denote $$\begin{bmatrix} \sigma_1 & \cdots & \sigma_k\end{bmatrix}'$$ by $$d$$, $$\operatorname{diag}(x_1, \ldots, x_k)$$ by $$X$$. Since $$|\Sigma| = |D|^2|A(\rho)| = \prod\limits_{i = 1}^k\sigma_i^2|A(\rho)|$$ and $$x'D = d'X$$, the log-likelihood function is \begin{align} l(\sigma_1^2, \ldots, \sigma_k^2) &= -\frac{1}{2}\log|\Sigma| - \frac{1}{2}x'DA(\rho)^{-1}Dx \\ &= -\frac{1}{2}\sum_{i = 1}^k\log\sigma_i^2 - \frac{1}{2}d'XA(\rho)^{-1}Xd + \text{constant}. \end{align} It thus follows by the quadratic form differentiation formula $$\partial z'Mz/\partial z = 2Mz$$ and the chain rule that \begin{align} \frac{\partial l(\sigma_1^2, \ldots, \sigma_k^2)}{\partial \sigma_j^2} &= -\frac{1}{2\sigma_j^2} - \frac{1}{\sigma_j}e_j'XA(\rho)^{-1}Xd \\ &= -\frac{1}{2\sigma_j^2} - \frac{1}{\sigma_j} \begin{bmatrix}0 & \cdots & x_j & \cdots & 0\end{bmatrix} A(\rho)^{-1}\begin{bmatrix} x_1\sigma_1 \\ \vdots \\ x_k\sigma_k \end{bmatrix}, \tag{1} \end{align} where $$e_j$$ is the $$k$$-long column vector with all entries $$0$$ but the $$j$$-th entry $$1$$.

As $$A(\rho) = (1 - \rho)I + \rho jj'$$, the Sherman-Morrison formula gives \begin{align} A(\rho)^{-1} &= (1 - \rho)^{-1}I - (1 - \rho)^{-2}\frac{\rho jj'}{1 + (1 - \rho)^{-1}\rho k} \\ &= (1 - \rho)^{-1}\left[I - \frac{\rho jj'}{1 + (k - 1)\rho}\right]. \tag{2} \end{align} Substituting $$(2)$$ into $$(1)$$ yields \begin{align} \frac{\partial l(\sigma_1^2, \ldots, \sigma_k^2)}{\partial \sigma_j^2} = -\frac{1}{2\sigma_j^2} - \frac{x_j^2}{1 - \rho} + \frac{x_j\rho}{(1 - \rho)\sigma_j}\frac{\sum_i x_i\rho_i}{1 + (k - 1)\rho}. \end{align}

• this is really helpful, thank you! I think you forgot the $\log$ when you go from $-\frac{1}{2} log \left| \Sigma \right|$ to $-\frac{1}{2} \sum_{i=1}^k \sigma_i^2$, right? I think it should be $-\frac{1}{2} \sum_{i=1}^k \log(\sigma_i^2)$, whose derivative is $-\frac{1}{2\sigma_i^2}$ instead of $\frac{1}{2}$.
– bob
Commented Jun 4, 2023 at 19:27
• Also, if I'm not mistaken, I believe the result of the Sherman-Morrison formula (inside the brackets $[]$)should have a $-$ instead of a $+$, correct?
– bob
Commented Jun 4, 2023 at 20:03
• Yes, I will edit later. Commented Jun 4, 2023 at 22:57

$$\log|\Sigma| =2\log|D|+\log|A|$$

$$\Sigma^{-1}=D^{-1}A^{-1}D^{-1}$$

$$A^{-1}=\frac{1}{1-\rho}\left(I-\frac{\rho}{1+(k-1)\rho}jj^\top\right)$$

$$x^\top D^{-1} j j^\top D^{-1} x = \left(j^\top D^{-1} x\right)^2 =\left( \sum_a \sigma_a^{-1}x_a\right)^2.$$

• Thanks! I working with this and it is helpful. Could you point me to a source providing these identities, or something that can be cited? (especially the inverse formula)
– bob
Commented Jun 2, 2023 at 16:29
• Any basic linear algebra book will state and prove three of these. The fourth, about $A^{-1},$ can be derived in many ways and follows from various considerations (consider the eigenspaces and eigenvalues of $A,$ for instance, which can be found by inspection) -- but the acid test of any matrix inversion formula is to multiply the putative inverse $A^{-1}$ by the original matrix $A$ and check that the identity matrix results. That doesn't even require a citation, really. If you want one, check out the Sherman-Morrison formula.
– whuber
Commented Jun 2, 2023 at 16:45
• Thanks @whuber!
– bob
Commented Jun 12, 2023 at 16:57