Covariance formula

Question

I'm struggling to understand this presentation of covariance. It says:

The variance-covariance matrix (or simply the covariance matrix ) of a random vector $\overline{X}$ is given by:

$$Cov(\overline{X})=E[\overline{X}\overline{X}^T]-E\overline{X}(E\overline{X})^T$$

First of all , what does he/she mean with covariance MATRIX of A VECTOR ? Variance of a vector (covariance with itself) ? But isn't that supposed to be a scalar, not a matrix ?!

Second, if you multiply $\overline{X}$ with $\overline{X}^T$, you get a matrix. How do you get the expected value ($E[\overline{X}\overline{X}^T]$) of that matrix?? It doesn't have multiple layers!!

Then, if you take the expected value ($E\overline{X})$ of $\overline{X}$, it's a scalar! What is the point in multiplying it with its transpose? The transpose of a scalar is itself!

I know that the usual definition of covariance is

$$cov(x.y)=\frac{\sum_{i=1}^{n}(x_i-\overline{x})(y_i-\overline{y})}{N-1}$$

THIS makes perfect sense, but the earlier presentation with expected values and vectors is confusing!

Answer 1

The expected value of a matrix is defined as the matrix of expected values. Let $\mathbf{M}$ a $p\times q$ matrix, the expectation of $M$ is defined as follows $$ \mathbb{E}\left[\mathbf{M}\right] = \mathbb{E}\left[M_{i,j}\right]\qquad i\in [1,p], j\in[1,q] $$ so, for example, you can define the expectation of the sum of two matrices $$ \begin{align*} \mathbb{E}(\mathbf{X}+\mathbf{Y}) &= \mathbb{E}([X_{i,j}]+[Y_{i,j}])\\ &= \mathbb{E}[(X_{i,j}+Y_{i,j})]\\ &= [\mathbb{E}(X_{i,j})+\mathbb{E}(Y_{i,j})]\\ &= [\mathbb{E}(X_{i,j})]+[\mathbb{E}(Y_{i,j})]\\ &= [\mathbb{E}(\mathbf{X})]+[\mathbb{E}(\mathbf{Y})] \end{align*} $$

So, since the covariance is defined by the expectation, you can define a covariance also for a vector $\mathbf{X}=[X_1, X_2,\dots, X_n]$ of $n$ jointly distributed random variables $X_i$ (in particular for a vector with itself).

So I think that $Cov(\mathbf{X})$ is a shorthand to write the covariance of two random variable vectors, so $Cov(\mathbf{X})=Cov(\mathbf{X}, \mathbf{X})$.

You can find it by the definition. Let $\mathbf{C}_{\mathbf{X}}$ the covariance matrix $$ \begin{align*} \mathbf{C}_{\mathbf{X}} &= \mathbb{E}\left[(\mathbf{X}-\mathbb{E}\mathbf{X})(\mathbf{X}-\mathbb{E}\mathbf{X})^\top \right] \\ &=\mathbb{E}\left[(\mathbf{X}-\mathbb{E}\mathbf{X})(\mathbf{X}^\top-\mathbb{E}\mathbf{X}^\top) \right]\\ &=\mathbb{E}\left[\mathbf{X}\mathbf{X}^\top\right] - \mathbb{E}\mathbf{X}\mathbb{E}\mathbf{X}^\top- \mathbb{E}\mathbf{X}\mathbb{E}\mathbf{X}^\top+\mathbb{E}\mathbf{X}\mathbb{E}\mathbf{X}^\top\\ &=\mathbb{E}\left[\mathbf{X}\mathbf{X}^\top\right] - \mathbb{E}\mathbf{X}\mathbb{E}\mathbf{X}^\top \end{align*} $$

Let me know if it's clear. Hope it will help you.

Bibliography: Random Vectors Wikipwdia - Covariance