Let $X_1, \ldots, X_n$ be a random sample from $N_d(\mu, \Sigma)$ and let: $$S=\frac{1}{n}\sum_{i=1}^n(X_i-\bar{X})(X_i-\bar{X})'\,,$$ denote the sample covariance matrix.

If $S$ is invertible, it can be shown that $\bar{X}$ and $S$ are the maximum likelihood estimators for the mean $\mu$ and covariance $\Sigma$ respectively.

What would happen when $S$ is not invertible?

  • 4
    $\begingroup$ Then your sample lies on a lower dimensional subspace. You can still do normal inference on the subspace. $\endgroup$
    – user603
    Dec 13 '15 at 13:46

In Srivastava, M. S., & von Rosen, D. (2002). Regression models with unknown singular covariance matrix. Linear algebra and its applications, 354(1), 255-273., from Theorem 2.1 we learn:

Assume a $d$-variate normal $\mathbf N (\mathbf \mu, \Sigma)$ with Covariance matrix $\Sigma$ of rank $0<r < d$. Assume an i.i.d. sample of size $n$ and denote the whole sample by the $d \times n$ matrix $\mathbf Y$ (each column is an observation). Let $\mathbf P= I_n - \frac 1n\cdot \mathbf i \mathbf i'$, (where $\mathbf i$ is a column vector of ones). $\mathbf P$ centers a vector around its mean, is idempotent and symmetric. Let $\mathbf S = \mathbf Y\mathbf P \mathbf Y' = ( \mathbf Y\mathbf P)( \mathbf Y\mathbf P)'$. This is the outer product of the de-meaned sample. Denote $\mathbf H$ the $d \times r$ matrix containing the eigenvectors corresponding to the $r$ largest eigenvalues of $\mathbf S$. Let $\mathbf L$ be an $ r \times r$ diagonal matrix containing in its diagonal those $r$ largest eigenvalues. Then :

1) The MLE for the mean-vector $\mathbf \mu$ is the sample mean vector, as usual

2) The MLE for the covariance matrix is

$$\hat \Sigma_{MLE} = \frac 1n \mathbf H \mathbf L \mathbf H'$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.