1
$\begingroup$

The covariance of two random variables $(X_i,X_j)$ is given by $$Cov(X_i,X_j) = E[X_iX_j] - E[X_i]E[X_j],$$ where $E[X_i]$ is the expected value of $X_i$, or its mean.

The covariance matrix $\Sigma$, in turn, is defined as the covariances between all $\textbf{X} = (X_1,...,X_n)$ random variables, such that $\Sigma_{ij} = Cov(X_i,X_j)$.

In my studies, however, I've come in contact with techniques dedicated to estimate the covariance matrix (such as using maximum likelihood) but I couldn't find any real world problems requiring the use of such techniques. Moreover, as stated above, it is possible to calculate every element of $\Sigma$ using the covariance formula stated above.

My question is when is it necessary to estimate a covariance matrix instead of calculating it by directly using the covariance formula?

$\endgroup$
3
  • $\begingroup$ Douglas: I changed the original title so it is less generic. Please free to amend it if you think it does not reflect your original theme but do be more specific that simply stating : Why do we need to estimate the covariance matrix? :) $\endgroup$
    – usεr11852
    May 4, 2015 at 8:20
  • $\begingroup$ I don't think the Mahalanobis distance has anything to do with my question, so I re-edited it in order to express myself more clearly. $\endgroup$ May 4, 2015 at 18:55
  • 2
    $\begingroup$ If you happen to find an entire population of data in the wild that exactly follows a known joint distribution, feel free to calculate its exact covariance matrix. But you'll have your work cut out for you. Note that the maximum likelihood estimate just reduces to the sample covariance anyway. $\endgroup$ May 4, 2015 at 20:30

1 Answer 1

2
$\begingroup$

You always estimate the covariance matrix, there is no "formula" for it. For instance, the "covariance formula" which you refer to involves estimation of several quantities: $E[X_iX_j],E[X_j],E[X_i]$. Notice, that you don't know what are these values, so you actually estimate them. In this regard the "formula" is really an estimator. It's not necessarily the best estimator in all cases.

Look at the discussion of bias in sample covariance matrix estimation here.

$\endgroup$
2
  • 1
    $\begingroup$ I believe my problem was with the nomenclature then. Both the expected values, as well as the covariance matrix itself, will always be estimates simply because one cannot observe the entire population being studied, or the population itself does not perfectly follow a joint distribution. $\endgroup$ May 4, 2015 at 23:41
  • 1
    $\begingroup$ @DouglasDeRizzoMeneghetti You got it, it's always estimators on samples. $\endgroup$
    – Aksakal
    May 4, 2015 at 23:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.