The Problem

Although it seems to be straight forward I am struggling to prove the following statement. Assume, we have $p$-variate Gaussian observations $\left\{x_1, \ldots, x_N \right\} \subset \mathbb{R}^p$ with known mean $\mu = 0$. We encounter the sufficient statistic $$S = \frac{1}{N} \sum_{l=1}^N x_l x_l^t.$$ We are looking for the solution of the following optimization problem, which is the constrained log-likelihood estimator for the normal distribution $$\hat{\Sigma} = argmax_{\Sigma_{ii} = 1 \forall i = 1, \ldots, p} -\log \left(|\Sigma|\right) - tr\left(\Sigma^{-1} S\right). $$ and additionally $\hat{\Sigma}$ shall be positiv-semidefinit (p.s.d). It seems to be clear that $\hat{\Sigma} = \Sigma_0 := D_S^{-\frac{1}{2}} S D_S^{-\frac{1}{2}}$, where $D_S$ is the diagonal matrix with the diagonal elements of $S$. However, I can not prove this.

What I have tried so far

In my oppinion the problem can be tackled from two sides.

  • We know that the maximum likelihood estimator for the transformed observations $x_l' = D_S^{-1}x_l, l = 1, \ldots, N$ is equal to $\Sigma_0$. However, it is not obvious to me, that the problem is invariant to a scaling of the observations.
  • By defining $\Omega = \Sigma^{-1}$, we can transform the problem to a convex optimization problem, where the search space is also convex. I.e. we are then looking for $$\hat{\Omega} = argmin_{\left(\Omega^{-1}\right)_{ii} = 1 \forall i = 1, \ldots, p} -\log \left(|\Omega|\right) + tr\left(\Omega S\right).$$ The Lagrange function is then $$L(\Omega, \lambda) = -\log \left(|\Omega|\right) + tr\left(\Omega S\right) + \sum_{j=1}^p \lambda_j\left(tr\left(\Omega^{-1} e_j e_j^T\right) - 1\right),$$ where $e_j$ is the $j-th$ unit column vector. Setting its derivative to $0$, I ended up with $$ \frac{\partial L}{\partial \Omega} = -\Omega^{-1} + S -\Omega^{-1}\left(\sum_{j=1}^{p} \lambda_j \left( e_j e_j^{T} \right) \right) \Omega^{-1} \overset{!}{=} 0,$$ where $\lambda_1, \ldots, \lambda_p$ can be chosen arbitrarily. Unfortunately, I am not able to prove that this holds for $\Omega = \Sigma_0^{-1}$. Also, I do not use that $\Omega$ is p.s.d.. Optimizing with respect to the square root of $A := \Omega^\frac{1}{2}$ (which can be done to ensure the search space is p.s.d.), I can't write down the derivative with respect to $tr\left(A^{-1} A^{{T}^{-1}} e_j e_j^T\right) - 1$.

Can somebody give a rigorous argument why $\Sigma_0$ solves the problem? Thank you very much! I am looking desperately for it!

  • 1
    $\begingroup$ It turns out, that $\Sigma_0$ is in fact not the maximizer. Apparently, there is also no closed form solution (as indicated for two dimensions in this paper on page 969). However, I would be curious if that is all we can say about the problem. $\endgroup$
    – mvakert
    Sep 15 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.