0
$\begingroup$

I'm struggling to understand this presentation of covariance. It says:

The variance-covariance matrix (or simply the covariance matrix ) of a random vector $\overline{X}$ is given by:

$$Cov(\overline{X})=E[\overline{X}\overline{X}^T]-E\overline{X}(E\overline{X})^T$$

First of all , what does he/she mean with covariance MATRIX of A VECTOR ? Variance of a vector (covariance with itself) ? But isn't that supposed to be a scalar, not a matrix ?!

Second, if you multiply $\overline{X}$ with $\overline{X}^T$, you get a matrix. How do you get the expected value ($E[\overline{X}\overline{X}^T]$) of that matrix?? It doesn't have multiple layers!!

Then, if you take the expected value ($E\overline{X})$ of $\overline{X}$, it's a scalar! What is the point in multiplying it with its transpose? The transpose of a scalar is itself!

I know that the usual definition of covariance is

$$cov(x.y)=\frac{\sum_{i=1}^{n}(x_i-\overline{x})(y_i-\overline{y})}{N-1}$$

THIS makes perfect sense, but the earlier presentation with expected values and vectors is confusing!

$\endgroup$

1 Answer 1

1
$\begingroup$

The expected value of a matrix is defined as the matrix of expected values. Let $\mathbf{M}$ a $p\times q$ matrix, the expectation of $M$ is defined as follows $$ \mathbb{E}\left[\mathbf{M}\right] = \mathbb{E}\left[M_{i,j}\right]\qquad i\in [1,p], j\in[1,q] $$ so, for example, you can define the expectation of the sum of two matrices $$ \begin{align*} \mathbb{E}(\mathbf{X}+\mathbf{Y}) &= \mathbb{E}([X_{i,j}]+[Y_{i,j}])\\ &= \mathbb{E}[(X_{i,j}+Y_{i,j})]\\ &= [\mathbb{E}(X_{i,j})+\mathbb{E}(Y_{i,j})]\\ &= [\mathbb{E}(X_{i,j})]+[\mathbb{E}(Y_{i,j})]\\ &= [\mathbb{E}(\mathbf{X})]+[\mathbb{E}(\mathbf{Y})] \end{align*} $$

So, since the covariance is defined by the expectation, you can define a covariance also for a vector $\mathbf{X}=[X_1, X_2,\dots, X_n]$ of $n$ jointly distributed random variables $X_i$ (in particular for a vector with itself).

So I think that $Cov(\mathbf{X})$ is a shorthand to write the covariance of two random variable vectors, so $Cov(\mathbf{X})=Cov(\mathbf{X}, \mathbf{X})$.

You can find it by the definition. Let $\mathbf{C}_{\mathbf{X}}$ the covariance matrix $$ \begin{align*} \mathbf{C}_{\mathbf{X}} &= \mathbb{E}\left[(\mathbf{X}-\mathbb{E}\mathbf{X})(\mathbf{X}-\mathbb{E}\mathbf{X})^\top \right] \\ &=\mathbb{E}\left[(\mathbf{X}-\mathbb{E}\mathbf{X})(\mathbf{X}^\top-\mathbb{E}\mathbf{X}^\top) \right]\\ &=\mathbb{E}\left[\mathbf{X}\mathbf{X}^\top\right] - \mathbb{E}\mathbf{X}\mathbb{E}\mathbf{X}^\top- \mathbb{E}\mathbf{X}\mathbb{E}\mathbf{X}^\top+\mathbb{E}\mathbf{X}\mathbb{E}\mathbf{X}^\top\\ &=\mathbb{E}\left[\mathbf{X}\mathbf{X}^\top\right] - \mathbb{E}\mathbf{X}\mathbb{E}\mathbf{X}^\top \end{align*} $$

Let me know if it's clear. Hope it will help you.

Bibliography: Random Vectors Wikipwdia - Covariance

$\endgroup$
2
  • $\begingroup$ Thank you for your response. I think the problem lies in that I dont understand what a random vector is. Am I right that it is something else than a regular vector like [5 8 1]' $\endgroup$
    – Suvi
    Commented Jul 21, 2022 at 16:30
  • $\begingroup$ A random vector of size $n$ is a vector composed of $n$ random variables. So it's not just a regular vector, but a vector where each entry is a stochastic variable, so each entry depends on random events. You can see it also as an array of functions. I hope now the term sounds better. Check the first bibliography link $\endgroup$ Commented Jul 21, 2022 at 16:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.