6
$\begingroup$

I saw the following statement in this paper and wanted to understand its meaning:

enter image description here

$\endgroup$

1 Answer 1

7
$\begingroup$

$\DeclareMathOperator{\diag}{diag}$ $\DeclareMathOperator{\tr}{tr}$

A great question. Most standard multivariate analysis texts only treated the $N > m$ case to give the MLE of $\mu$ and $X$ (or more frequently, $\Sigma = X^{-1}$). The $N \leq m$ case seems always overlooked (or merely qualitatively mentioned).

In short, "the MLE problem is unbounded" means that the objective function \begin{align} l(\mu, X) := -\log\det(X) + \frac{1}{N}\sum_{i = 1}^N(\hat{\xi}_i - \mu)^TX(\hat{\xi}_i - \mu), \quad \mu \in \mathbb{R}^m, X \in S_+^m \end{align} does not have a fixed lower bound (which depends on $\hat{\xi}_1, \ldots, \hat{\xi}_N$ only) when $m \geq N$ (by contrast, when $N > m$, almost every multivariate analysis textbook will show you $l(\mu, X)$ is bounded below). To show that $l(\mu, X)$ can be arbitrarily small, first introduce the following notations:
\begin{align} & Z = \begin{bmatrix} \hat{\xi}_1^T \\ \vdots \\ \hat{\xi}_N^T \end{bmatrix} \in \mathbb{R}^{N \times m}, \; e = \begin{bmatrix} 1 \\ \vdots \\ 1 \end{bmatrix} \in \mathbb{R}^{N \times 1}, \; \bar{\xi} = N^{-1}\sum_{i = 1}^N\hat{\xi}_i, \\ & A = \sum_{i = 1}^N(\hat{\xi}_i - \bar{\xi})(\hat{\xi}_i - \bar{\xi})^T = Z^T(I_{(N)} - N^{-1}ee^T)Z := Z^TPZ \in \mathbb{R}^{m \times m}. \end{align} It can be seen that $P$ is idempotent, whence $A = (PZ)^TPZ$ is positive semi-definite, and \begin{align} r:= \operatorname{rank}(A) = \operatorname{rank}(PZ) \leq \operatorname{rank}(P) = N - 1 < m. \end{align}
Therefore, the spectral decomposition of $A$ can be written as \begin{align} A = O\diag(\Lambda, 0_{(m - r)})O^T, \end{align} where $O$ is an order $m$ orthogonal matrix, $\Lambda = \diag(\lambda_1, \ldots, \lambda_r)$, $\lambda_i (1 \leq i \leq r)$ are positive eigenvalues of $A$.

Using these notations, $l(\mu, X)$ can be rewritten as: \begin{align} l(\mu, X) = -\log\det(X)+\frac{1}{N}\tr(XA) + (\bar{\xi} - \mu)^TX(\bar{\xi} - \mu). \end{align}

Let $X^* = O\diag(\Lambda^{-1}, kI_{(m - r)})O^T$ (where $k > 0$ is to be determined), $\mu^* = \bar{\xi}$, it then follows that \begin{align} & l(\mu^*, X^*) = \sum_{i = 1}^r\log\lambda_i - (m - r)\log k+ \frac{r}{N}, \end{align} whence $l(\mu^*, X^*) \to -\infty$ as $k \to \infty$, i.e., $l$ cannot be bounded from below.

If the above matrix operations are too complicated to understand, you can get some sense by considering the case $m = N = 1$ (univariate normal distribution with only one observation), for which the objective function is \begin{align} l(\mu, X) = -\log X + (\hat{\xi}_1 - \mu)^2X. \end{align} Therefore $l(\hat{\xi}_1, X) = -\log X \to -\infty$ as $X \to +\infty$, i.e., the objective function is unbounded.

$\endgroup$
3
  • $\begingroup$ Overwhelming? It's rather precise. +1. $\endgroup$ Commented Dec 23, 2022 at 5:32
  • 2
    $\begingroup$ @User1865345 Thank you. It takes me a while to achieve this precision. :) $\endgroup$
    – Zhanxiong
    Commented Dec 23, 2022 at 5:39
  • $\begingroup$ Changing to $X^\ast, \mu^\ast$ was witty. Keep answering. Appreciate your posts. $\endgroup$ Commented Dec 23, 2022 at 5:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.