Let's have spatio-temporal dataset ($Y \in \mathbb{R}^{L \times T}$). Where $L$ stands for spatial grid points and $T$ for time. Now let's say that the noise of the system follows a multivatiate gaussian distribution. $y^\ell_t = \epsilon^\ell_t$, where $\epsilon^\ell_t \sim \mathcal{N}(0, \Sigma)$. Standing $\Sigma$ for a the covariance matrix $L \times L$.

Now, we have three different blocs (close spatial points) where the spatial points of each bloc covary together, but independently of the others bloc. Then, our $\Sigma$ is a diagonal block matrix. For example

import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA

Sigma = np.zeros((8*30,30*8))
Sigma[:40, :40] = 0.3  # First bloc
Sigma[88:136, 88:136] = 0.3  # Second block
Sigma[200:, 200:] = 0.3

Correlation matrix between every grid point <span class=$\ell$">

Now we create a random data from that $\Sigma$ covariance matrix. And perform pca on top of it. I would expect the first three components to stand for each region where noise covary. For the shake of interpretability, we will assume that our spatio-temporal model lives in a $8 \times 30$ space, that is $L = 240$. In this spatio temporal model there are three blocs of covariant points corresponding to the contiguous grid-points 0:44, 88:136 and 200:240. According to $\Sigma$. Then,

Y = np.random.multivariate_normal(np.zeros(8*30), Sigma, size=50000)
pca = PCA(n_components=3)
for i in range(3):
    # Reshaping the components to a spatial representation allows us to plot
    # the components as the contribution of each grid point to each component. 
    # The spatial dimensions are 30 x 8.
    plt.imshow(pca.components_[i, :].reshape((30,8)))  

1st PC 2nd PC 3rd PC

The fact is, that I see that they are correlated. And I'm not really sure why, given that I would expect that the principal components of PCA to be orthogonal between them. I also plot the correlation matrix of the first three components.

components = pca.components_
corr_2 = np.corrcoef(components)

Covariance matrix of first 3 PCs

Shouldn't the elements outside of the diagonal of that matrix be 0?

I've also try Varimax rotation, where the three components are now independent.

1st PC rotated with Varimax 2nd PC rotated with Varimax 3rd PC rotated with Varimax

I guess I am missing some fundamental understanding of what means that the components are orthogonal. Could someone bring light on why the correlation matrix of the 3 comnponents has non-zeros outside of the diagonal?

  • $\begingroup$ your expectations of PCA component vectors are wrong. why are you even grouping 30x8? nothing in your code suggests there needs to be any kind of grouping by 8 or 30 $\endgroup$
    – Aksakal
    Feb 19, 2020 at 14:41
  • $\begingroup$ I edit it, hoping that now is more clear. At the very end, my question is: shouldn't be the components independent? $\endgroup$
    – Xbel
    Feb 19, 2020 at 15:41
  • $\begingroup$ the components are independent. it's your interpretation of independence that is puzzling. what is this about '.reshape((30,8))'? $\endgroup$
    – Aksakal
    Feb 19, 2020 at 15:58
  • $\begingroup$ "The components are independent". If that is the case, why are non-zero elements outside of the diagonal in the correlation matrix between the first three components? I mean, sure I am wrong, but I don't understand why the correlation matrix of the components is not showing non-correlation between the components. Because in my intuition of PCA the components must be orthogonal. $\endgroup$
    – Xbel
    Feb 20, 2020 at 8:38
  • $\begingroup$ Components are orthogonal $\endgroup$
    – Aksakal
    Feb 20, 2020 at 14:34


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.