# Inference for multivariate normal when the sample covariance matrix is not invertible

Let $X_1, \ldots, X_n$ be a random sample from $N_d(\mu, \Sigma)$ and let: $$S=\frac{1}{n}\sum_{i=1}^n(X_i-\bar{X})(X_i-\bar{X})'\,,$$ denote the sample covariance matrix.

If $S$ is invertible, it can be shown that $\bar{X}$ and $S$ are the maximum likelihood estimators for the mean $\mu$ and covariance $\Sigma$ respectively.

What would happen when $S$ is not invertible?

• Then your sample lies on a lower dimensional subspace. You can still do normal inference on the subspace. Dec 13 '15 at 13:46

Assume a $d$-variate normal $\mathbf N (\mathbf \mu, \Sigma)$ with Covariance matrix $\Sigma$ of rank $0<r < d$. Assume an i.i.d. sample of size $n$ and denote the whole sample by the $d \times n$ matrix $\mathbf Y$ (each column is an observation). Let $\mathbf P= I_n - \frac 1n\cdot \mathbf i \mathbf i'$, (where $\mathbf i$ is a column vector of ones). $\mathbf P$ centers a vector around its mean, is idempotent and symmetric. Let $\mathbf S = \mathbf Y\mathbf P \mathbf Y' = ( \mathbf Y\mathbf P)( \mathbf Y\mathbf P)'$. This is the outer product of the de-meaned sample. Denote $\mathbf H$ the $d \times r$ matrix containing the eigenvectors corresponding to the $r$ largest eigenvalues of $\mathbf S$. Let $\mathbf L$ be an $r \times r$ diagonal matrix containing in its diagonal those $r$ largest eigenvalues. Then :
1) The MLE for the mean-vector $\mathbf \mu$ is the sample mean vector, as usual
$$\hat \Sigma_{MLE} = \frac 1n \mathbf H \mathbf L \mathbf H'$$