I have a very basic question. I'm understanding Pearson Correlation right now and that is given by:

enter image description here

Given that, I'm trying to understand the Covariance. Wikipedia says:

In statistics, sometimes the covariance matrix of a multivariate random variable is not known but has to be estimated

And indeed, the formula I'm using for the correlation is as follows:

enter image description here

But I'm still a little confused. Why is it that we have to use an estimator for the covariance? When I looked this up online, I saw things explaning estimators of covariance, but not why we estimate covariance itself....

Furthermore, I'm confused on the logic of the above covariance estimator. For example, in wikipedia, I see the covariance estimator defined in two ways.


enter image description here

and this:

enter image description here

What exactly is the difference between the two, and how do they relate to the covariance estimator in the correlation?

Again, I understand this is a very basic question, but I haven't been able to understand from what I've seen online so far. Any help or links in the right direction would be greatly appreciated, thanks!

  • $\begingroup$ Could you clarify the intent of your question? Are you asking why it might be useful to estimate covariances at all, or are you asking why a covariance must be estimated rather than, say, looked up in a dictionary or produced by praying to the statistical gods? :-) $\endgroup$
    – whuber
    Mar 28, 2017 at 15:08
  • 2
    $\begingroup$ The difference between the two formulas: the first one is for random variables and gives you a scalar, the other one for random vectors (because of the transpose character) and gives your a matrix. $\endgroup$
    – mic
    Mar 28, 2017 at 15:12
  • 2
    $\begingroup$ About estimation: The correlation or covariance between two random variables exist, it is an underlying property of the two, but you have no access to it in practice. To do so, you need data AND an estimator. With both you can have a more or less precise view of the relationship (the true but unknown correlation or covariance) between the two variables. $\endgroup$
    – mic
    Mar 28, 2017 at 15:15
  • 1
    $\begingroup$ Observations for X: 0.2, -0.1, 0.1, 0.3, -0.2 Observations for Y: 0.1, -0.2, 0.2, 0.1, -0.1 What is the correlation/covariance between the two? Now, let cov(X,Y) = 12. What is the covariance of cov(3X + 2, 5Y - 1)? $\endgroup$
    – mic
    Mar 28, 2017 at 15:22
  • 3
    $\begingroup$ @mic is doing a fabulous job at approximating this. Let me propose an idea that could help understand it: in probability you can solve problems in the abstract because you take parameters as given $(\Pr(\text{fair die=4})=1/6)$, but this is not so in statistics - in statistics you work with samples, from which you estimate the parameters in the elusive population. Then you get into whether an estimator is biased or unbiased, etc. $\endgroup$ Mar 28, 2017 at 15:27

1 Answer 1


I think you misunderstand the process(and the goal ) of the statistical inference .

The probability space of a random experiment is given by the 3-tuple $\{S,\Upsilon,P\}$ where :

$S$ : sample space being the set of all outcomes of the experiment

$\Upsilon$: event space being the set of all events (typically a sigma-algebra on $S$)

$P$: probability set function having domain $\Upsilon$ used to assign probabilities to events .

The typical question in this case is " Given the probability space, what can we say about the characteristics and properties of outcomes of an experiment?"

Then The expected value of a random variable $X$ (bivariate random variable ) is denoted by $E(X)$ (which is also the first moment about the origin ) , now If we have two random variables $X$ and $Y$ (bivariate random variables) then the central joint moment $\mu _{1,1} = E(X −E(X))(Y −E(Y))$ is called the covariance between $X$ and $Y$ and is denoted by the symbol $\sigma_{XY}$ , or by $cov(X,Y) $.

Now what if we have $n$-variate random variable ?

The first thing we have to know that the expected value of a random vector (or matrix) is a vector (or matrix) whose elements are the expected values of the individual random variables that are the elements of the random vector .That means if $X$ is an $n$ variate random variable then $$E(X) = E[X_{1},...,X_{n}]^{T}= [E(X_{1}),...,E(X_{n})]^{T}$$ and the covariance matrix of the $n$_variate random variable $X$ is the $n × n$ symmetric matrix $Cov(X) = E(X − E(X)) (X − E(X))^{T}$ (Review basics in Linear Algebra).

But the term "statistical inference" refers to the inductive process of generating information about characteristics of a population or process, by analyzing a sample of objects or outcomes from the population or process that means the statistical inference turns the above question around:

Given the observed characteristics and properties of outcomes of an experiment, what can we say (infer) about the probability space?

Now let $((X_{1}, Y_{1}) , . . . , (X_{n}, Y_{n}))$ be a random sample from a population distribution, and let $S_{XY}$ be the sample covariance between $X$ and $Y$ . Then, $E(S_{XY}) = ((n-1)/n)\sigma_{XY}$ (biased estimator of the population covariance ) .

Take a look of the this figure it provides an overview of the process of statistical inference enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.