# Covariance and Correlation Matrices

I have a somewhat dumb question. When determining the correlation or covariance (doesn't matter I suppose) amongst random vectors, is the covariance computed among features or among observations?

For instance, say I have some data matrix $$X = \begin{bmatrix} x_{11} & x_{12} & x_{13} & \dots & x_{1p} \\ x_{21} & x_{22} & x_{23} & \dots & x_{2p} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ x_{n1} & x_{d2} & x_{d3} & \dots & x_{dp} \end{bmatrix}$$

In which there are $$n$$ observations and $$p$$ features/predictors. Is the covariance matrix going to be an $$n \times n$$ symmetric matrix or a $$p \times p$$ symmetric matrix?

The reason for asking is that I have seen both $$(X - \mu)^T(X - \mu)$$ and $$(X - \mu)(X - \mu)^T$$ being used to define covariance matrices.

• Are your observations independent? If so, that implies their covariance matrix is diagonal and little calculation is needed. The reason you might have seen both formulas is that some people represent their data with the transpose of your $X.$
– whuber
Commented Feb 21, 2021 at 18:47
• @whuber so you calculate the correlation between features then? But what if the data is a video or images where each frame or snapshot is a column in my data matrix? Commented Feb 21, 2021 at 18:55