I am trying to understand how covariance matrix works. So let's suppose we have two variables: $X, Y$, where $\text{Cov}(X,Y) = \mathbb{E}[(x -\mathbb{E}[X])(y-\mathbb{E}[Y])]$ gives the relation between the variables, ie how much one depends on the other.

Now, three variable case it is less clear for me. An intuitive definition for covariance function would be $\text{Cov}(X,Y,Z) = \mathbb{E}[(x -\mathbb{E}[X])(y-\mathbb{E}[Y])(z-\mathbb{E}[Z])]$, but instead the literature suggests using covariance matrix that is defined as two variable covariance for each pair of variables.

So, does the covariance include full information about variable relations? If so, what is the relation to my definition of $\text{Cov}(X,Y,Z)$?

  • 1
    $\begingroup$ I think, I see that my definition simply does not work. But is the covariance matrix sufficient to quantify the relation between all the variables? $\endgroup$ – Karolis Feb 15 '16 at 7:15
  • 5
    $\begingroup$ The covariance matrix is sufficient to quantify the covariance between all the variables but not the "relations" as this is to general a concept (variables can be related or dependent in a lot of different non-linear ways which are not captured by covariance). An exception to this would be if you knew the variables where multi-variate normal. $\endgroup$ – Zachary Blumenfeld Feb 15 '16 at 7:58
  • $\begingroup$ Thanks @ZacharyBlumenfeld! Could you recommend a good textbook on this? $\endgroup$ – Karolis Feb 15 '16 at 10:23
  • 2
    $\begingroup$ What is the difference between $x$ and $X$ in the term $x-E[X]$? I know what you mean by $X$ -- it is a random variable -- and also by $E[X]$ -- it is the expected value of $X$, a real number -- but what is $x$? If $x$ is another real number, then $x-E[X]$ is a real number -- nothing random about it -- and so your definition reduces to $$\operatorname{cov}(X,Y,Z) = E[(x -E[X])(y-E[Y])(z-E[Z])] = (x -E[X])(y-E[Y])(z-E[Z])$$ because the expected value of a real number is the real number itself. $\endgroup$ – Dilip Sarwate Feb 15 '16 at 13:51
  • 2
    $\begingroup$ @ZacharyBlumenfeld, your comment nearly qualifies as an answer. Perhaps you should expand it a little bit (add that $\mathbb{E}[(x -\mathbb{E}[X])(y-\mathbb{E}[Y])(z-\mathbb{E}[Z])]$ is a third-order central cross moment, what else) and post as an answer? $\endgroup$ – Richard Hardy Feb 15 '16 at 20:47

To expand on Zachary's comment, the covariance matrix does not capture the "relation" between two random variables, as "relation" is too broad of a concept. For example, we'd probably want to include the dependence of two variables on each other to be include in any measure of their "relation". However, we know that $cov(X,Y)=0$ does not imply that they are independent, as for example is the case with two random variables X~U(-1,1) and Y=X^2 (for a short proof, see: https://en.wikipedia.org/wiki/Covariance#Uncorrelatedness_and_independence).

So if we'd think that the covariance includes full information about variable relations, as you ask, zero covariance would suggest no dependence. This is what Zachary means when he says that there can be non-linear dependences that the covariance does not capture.

However, let $X:=(X_{1},...,X_{n})'$ be multivariate normal, X~$N(\mu,\Sigma)$. Then $X_{1},...,X_{n}$ are independent iff $\Sigma$ is a diagonal matrix with all off-diagonal elements = 0 (if all covariances = 0).

To see that this condition is sufficient, observe that the joint density factors, \begin{equation} f(x_{1},...,x_{n}) = \dfrac{1}{ \sqrt{(2 \pi)^{n} | \Sigma |}} exp(- \dfrac{1}{2} (x - \mu)' \Sigma^{-1} (x - \mu))= \Pi^{n}_{i=1} \dfrac{1}{\sqrt{2 \pi \sigma_{ii}}} exp(- \dfrac{(x_{i}-\mu_{i})^{2}}{2 \sigma_{ii}})=f_{1}(x_{1})...f_{n}(x_{n})\end{equation}.

To see that the condition is necessary, recall the bivariate case. If $X_{1}$ and $X_{2}$ are independent, then $X_{1}$ and $X_{1}|X_{2} = x_{2}$ must have the same variance, so

\begin{equation} \sigma_{11}=\sigma_{11|2}=\sigma_{11}-\sigma^{2}_{12} \sigma^{-1}_{22} \end{equation}

which implies $\sigma_{12}=0$. By the same argument, all off-diagonal elements of $\Sigma$ must be zero.

(source: prof. Geert Dhaene's Advanced Econometrics slides)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.