I know that PCA can be obtained by eigendecomposition of the covariance matrix, and the covariance matrix $S$ is obtained by the equation: $S = X^TX $, where $X$ is the centered data matrix.

But I am a bit confused about the dimensions of the covariance matrix.

In some resources, they define the data matrix as: $X_{n \times d}$ where n is the number of samples and d is the dimension. In other resources, it's the opposite: $X_{d \times n}$. And this definitely yields covariance matrices with different dimensions, and also eigenvectors of different dimensions.

I am not sure what am I getting wrong, but I think I am missing something important here.


When $X$ is $n\times d$, the scatter matrix (the scaled covariance) is $S=X^TX$. When it is $d\times n$, $S=XX^T$. And, in the latter case, the rows of $X$ are mean-centered as opposed to the former.

The logic is always to calculate $$\sum_{i=1}^n x_ix_i^T$$ where $x_i$ is one data sample of dimension $d\times 1$.

Covariance matrix is typically estimated as $S/n$ or $S/(n-1)$, since it's just a scalar, in PCA it doesn't matter.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.