The covariance of two random variables $(X_i,X_j)$ is given by $$Cov(X_i,X_j) = E[X_iX_j] - E[X_i]E[X_j],$$ where $E[X_i]$ is the expected value of $X_i$, or its mean.

The covariance matrix $\Sigma$, in turn, is defined as the covariances between all $\textbf{X} = (X_1,...,X_n)$ random variables, such that $\Sigma_{ij} = Cov(X_i,X_j)$.

In my studies, however, I've come in contact with techniques dedicated to estimate the covariance matrix (such as using maximum likelihood) but I couldn't find any real world problems requiring the use of such techniques. Moreover, as stated above, it is possible to calculate every element of $\Sigma$ using the covariance formula stated above.

My question is when is it necessary to estimate a covariance matrix instead of calculating it by directly using the covariance formula?

  • $\begingroup$ Douglas: I changed the original title so it is less generic. Please free to amend it if you think it does not reflect your original theme but do be more specific that simply stating : Why do we need to estimate the covariance matrix? :) $\endgroup$ – usεr11852 May 4 '15 at 8:20
  • $\begingroup$ I don't think the Mahalanobis distance has anything to do with my question, so I re-edited it in order to express myself more clearly. $\endgroup$ – Douglas De Rizzo Meneghetti May 4 '15 at 18:55
  • 2
    $\begingroup$ If you happen to find an entire population of data in the wild that exactly follows a known joint distribution, feel free to calculate its exact covariance matrix. But you'll have your work cut out for you. Note that the maximum likelihood estimate just reduces to the sample covariance anyway. $\endgroup$ – shadowtalker May 4 '15 at 20:30

You always estimate the covariance matrix, there is no "formula" for it. For instance, the "covariance formula" which you refer to involves estimation of several quantities: $E[X_iX_j],E[X_j],E[X_i]$. Notice, that you don't know what are these values, so you actually estimate them. In this regard the "formula" is really an estimator. It's not necessarily the best estimator in all cases.

Look at the discussion of bias in sample covariance matrix estimation here.

| cite | improve this answer | |
  • 1
    $\begingroup$ I believe my problem was with the nomenclature then. Both the expected values, as well as the covariance matrix itself, will always be estimates simply because one cannot observe the entire population being studied, or the population itself does not perfectly follow a joint distribution. $\endgroup$ – Douglas De Rizzo Meneghetti May 4 '15 at 23:41
  • 1
    $\begingroup$ @DouglasDeRizzoMeneghetti You got it, it's always estimators on samples. $\endgroup$ – Aksakal May 4 '15 at 23:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.