# Covariance formula

I'm struggling to understand this presentation of covariance. It says:

The variance-covariance matrix (or simply the covariance matrix ) of a random vector $$\overline{X}$$ is given by:

$$Cov(\overline{X})=E[\overline{X}\overline{X}^T]-E\overline{X}(E\overline{X})^T$$

First of all , what does he/she mean with covariance MATRIX of A VECTOR ? Variance of a vector (covariance with itself) ? But isn't that supposed to be a scalar, not a matrix ?!

Second, if you multiply $$\overline{X}$$ with $$\overline{X}^T$$, you get a matrix. How do you get the expected value ($$E[\overline{X}\overline{X}^T]$$) of that matrix?? It doesn't have multiple layers!!

Then, if you take the expected value ($$E\overline{X})$$ of $$\overline{X}$$, it's a scalar! What is the point in multiplying it with its transpose? The transpose of a scalar is itself!

I know that the usual definition of covariance is

$$cov(x.y)=\frac{\sum_{i=1}^{n}(x_i-\overline{x})(y_i-\overline{y})}{N-1}$$

THIS makes perfect sense, but the earlier presentation with expected values and vectors is confusing!

The expected value of a matrix is defined as the matrix of expected values. Let $$\mathbf{M}$$ a $$p\times q$$ matrix, the expectation of $$M$$ is defined as follows $$\mathbb{E}\left[\mathbf{M}\right] = \mathbb{E}\left[M_{i,j}\right]\qquad i\in [1,p], j\in[1,q]$$ so, for example, you can define the expectation of the sum of two matrices \begin{align*} \mathbb{E}(\mathbf{X}+\mathbf{Y}) &= \mathbb{E}([X_{i,j}]+[Y_{i,j}])\\ &= \mathbb{E}[(X_{i,j}+Y_{i,j})]\\ &= [\mathbb{E}(X_{i,j})+\mathbb{E}(Y_{i,j})]\\ &= [\mathbb{E}(X_{i,j})]+[\mathbb{E}(Y_{i,j})]\\ &= [\mathbb{E}(\mathbf{X})]+[\mathbb{E}(\mathbf{Y})] \end{align*}

So, since the covariance is defined by the expectation, you can define a covariance also for a vector $$\mathbf{X}=[X_1, X_2,\dots, X_n]$$ of $$n$$ jointly distributed random variables $$X_i$$ (in particular for a vector with itself).

So I think that $$Cov(\mathbf{X})$$ is a shorthand to write the covariance of two random variable vectors, so $$Cov(\mathbf{X})=Cov(\mathbf{X}, \mathbf{X})$$.

You can find it by the definition. Let $$\mathbf{C}_{\mathbf{X}}$$ the covariance matrix \begin{align*} \mathbf{C}_{\mathbf{X}} &= \mathbb{E}\left[(\mathbf{X}-\mathbb{E}\mathbf{X})(\mathbf{X}-\mathbb{E}\mathbf{X})^\top \right] \\ &=\mathbb{E}\left[(\mathbf{X}-\mathbb{E}\mathbf{X})(\mathbf{X}^\top-\mathbb{E}\mathbf{X}^\top) \right]\\ &=\mathbb{E}\left[\mathbf{X}\mathbf{X}^\top\right] - \mathbb{E}\mathbf{X}\mathbb{E}\mathbf{X}^\top- \mathbb{E}\mathbf{X}\mathbb{E}\mathbf{X}^\top+\mathbb{E}\mathbf{X}\mathbb{E}\mathbf{X}^\top\\ &=\mathbb{E}\left[\mathbf{X}\mathbf{X}^\top\right] - \mathbb{E}\mathbf{X}\mathbb{E}\mathbf{X}^\top \end{align*}

Let me know if it's clear. Hope it will help you.

Bibliography: Random Vectors Wikipwdia - Covariance

• Thank you for your response. I think the problem lies in that I dont understand what a random vector is. Am I right that it is something else than a regular vector like [5 8 1]'
– Suvi
Commented Jul 21, 2022 at 16:30
• A random vector of size $n$ is a vector composed of $n$ random variables. So it's not just a regular vector, but a vector where each entry is a stochastic variable, so each entry depends on random events. You can see it also as an array of functions. I hope now the term sounds better. Check the first bibliography link Commented Jul 21, 2022 at 16:52