Given such a matrix about the grades of 6 students in Maths, Computer Sciences and French:

$$A=\begin{bmatrix} 1 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 2 \\ 2 & 2 & 1 \\ 1 & 0 & 0 \\ 2 & 3 & 2 \end{bmatrix}$$

Work out the principal componant analysis.

I found that the eigen vectors were: $$ v_1=\begin{bmatrix} 1\\ -1\\ 1 \end{bmatrix}, v_2=\begin{bmatrix} 1\\ 0\\ -1 \end{bmatrix}, v_3=\begin{bmatrix} 1\\ 2\\ 1 \end{bmatrix}$$

Associated with $\operatorname{Sp}(A)=\{0,\frac{2}{3},2\}$

Thus, the two nontrivial factor axes are $v_2, v_3$ which explains $25\%,75\%$ of the inertia.

I'm then trying to work out the principal component analysis.

After calculating the matrix of centered data $Y=A-g$ with $g =\{1,1,1\}$,

Do I have to center and normalize eigenvectors ("explaining vectors") or solely to normalize them? Why?

That is to say do I get

$$\begin{bmatrix} \frac{1}{\sqrt{2}} & \frac{1}{\sqrt{6}}\\ 0 & \frac{2}{\sqrt{6}}\\ \frac{-1}{\sqrt{2}} & \frac{1}{\sqrt{6}} \end{bmatrix}$$

or something else and why?

  • 3
    $\begingroup$ By "explaining vectors" you mean eigenvectors? $\endgroup$ – amoeba says Reinstate Monica Dec 12 '15 at 14:28
  • $\begingroup$ Closely related threads with useful information about this (within answers or comments) include stats.stackexchange.com/questions/22329, stats.stackexchange.com/questions/162149, and stats.stackexchange.com/questions/35185 (which is very close to duplicating this question). For more, please search our site. $\endgroup$ – whuber Dec 12 '15 at 15:26
  • 1
    $\begingroup$ @whuber: I am not sure, but it looks like OP is not asking about centering/scaling the data matrix, but about centering/scaling PCA eigenvectors (probably in order to project the data onto them). $\endgroup$ – amoeba says Reinstate Monica Dec 12 '15 at 19:16
  • 1
    $\begingroup$ @amoeba That would be such a strange thing to do that I didn't catch it, but indeed that seems to be what's going on here. It may come down to what an "explaining vector" is intended to mean. $\endgroup$ – whuber Dec 12 '15 at 21:23
  • 1
    $\begingroup$ @amoeba Yes that's it! As far as they explain how the data is projected (feel free to correct me if I'm wrong). $\endgroup$ – IggyPass Dec 13 '15 at 20:33

Principal component analysis (PCA) in statistics commonly uses either the covariance matrix $\bf{C}$ or correlation matrix $\bf{R}$ for the variables (attributes). Therefore, for your problem, you would generate the $3 \times 3$ correlation matrix, and then perform eigendecomposition on $\bf{R}$. This is called solving the symmetric eigenvalue problem. The eigenvalues from $\bf{R}$ will essentially explain how many variables are correlated with one another, whereas, if the 3 variables are orthogonal (zero correlation) the eigenvalues should each be close to one. After eigendecomposition of $\bf{R}$, you can determine the determinant as $|\bf{R}|$$=\prod_j \lambda_j$, and the closer $|\bf{R}|$ is to unity the closer $\bf{R}$ is to the identity matrix $\bf{I}$ $\rightarrow$ zeroes in the off-diagonals of $\bf{R}$ and ones on the diagonal.

Without going into detail on the notation for the loadings and the PC scores, the loading matrix $\bf{L}$ will be a $3 \times 3$ matrix, which will the reveal correlation between each of the three variables with the 3 (orthogonal) principal components. I have always used 0.55 as an indicator for high loading. Sometimes variables can also load appreciably (rather than mostly) on more than one PC, and to decrease this tendency you can use a varimax orthogonal rotation -- which will make the variables load mostly on one component. (Computationally, there is also no distinction between a positive or negative sign of each loading, because the sign can change with the algorithm used. Therefore positive loading vs. negative should never be looked at as which sign is correct because you always have to compare signs using different packages).

Lastly, you will generate a $6 \times 3$ matrix $\bf{F}$ of PC scores, which represent you original data points in PC space. The 3 columns of PCs have zero correlation between them and by definition are standard normal distributed $\cal{N}(0,1)$ with mean zero and variance (s.d.) one.

Regarding normalization, the determination of $\bf{R}$ includes mean-zero standardization of your 3 variables, so skewness will be retained if present. Normalization will only rescale to range [0,1], whereas mean-zero standardization will result in a mean of zero for each variable, but not necessarily a variance of unity -- especially if you have skewness. If you want purely standard normal transforms, then calculate the van der Waerden scores for your data points within each variable.

  • 3
    $\begingroup$ I don't think the questions asks about centering/scaling the data matrix. It looks like it's asking about eigenvectors. $\endgroup$ – amoeba says Reinstate Monica Dec 12 '15 at 19:17

Eigen vectors matrix is commonly made orthonormal, i.e. eigenv vectors are scaled to make their norm equal to 1.

Remember that when you solve eigne values equation the eigen vector norms are not defined, i.e. you can multiply them by any scalar and they still remain eigenvectors.

You don't center eigenvectors. They're already centered if your data is centered

  • 2
    $\begingroup$ "They're already centered if your data is centered" - what would that mean? If the data matrix is centered, eigenvectors don't have to have zero mean. But of course I agree that it's nonsensical to "center" them. $\endgroup$ – amoeba says Reinstate Monica Dec 4 '17 at 21:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.