When is it necessary to estimate the covariance matrix instead of calculating it directly?

Question

The covariance of two random variables $(X_i,X_j)$ is given by $$Cov(X_i,X_j) = E[X_iX_j] - E[X_i]E[X_j],$$ where $E[X_i]$ is the expected value of $X_i$, or its mean.

The covariance matrix $\Sigma$, in turn, is defined as the covariances between all $\textbf{X} = (X_1,...,X_n)$ random variables, such that $\Sigma_{ij} = Cov(X_i,X_j)$.

In my studies, however, I've come in contact with techniques dedicated to estimate the covariance matrix (such as using maximum likelihood) but I couldn't find any real world problems requiring the use of such techniques. Moreover, as stated above, it is possible to calculate every element of $\Sigma$ using the covariance formula stated above.

My question is when is it necessary to estimate a covariance matrix instead of calculating it by directly using the covariance formula?

Answer 1

You always estimate the covariance matrix, there is no "formula" for it. For instance, the "covariance formula" which you refer to involves estimation of several quantities: $E[X_iX_j],E[X_j],E[X_i]$. Notice, that you don't know what are these values, so you actually estimate them. In this regard the "formula" is really an estimator. It's not necessarily the best estimator in all cases.

Look at the discussion of bias in sample covariance matrix estimation here.