3
$\begingroup$

If $X$ is your data set with columns $x_i$ representing all your variables, then principal component analysis (PCA) of $X$ gives $$T=XW,$$ where $T$ represents scores and $W$ represents loadings (columns $w_i$ of $W$ are eigenvectors of the covariance matrix).

Can we say that $w_i$ vector represents the importance of $x_i$? Or is this information only conveyed by the scores? I am confused by the fact that $w_i$'s represent a totally different space than that of $x_i$'s, albeit they being the projection of the $x_i$.

$\endgroup$
  • $\begingroup$ What do you mean by "Or this information is only conveyed by scores."? $\endgroup$ – zhanxw Oct 2 '15 at 4:59
  • $\begingroup$ Hello @Hiren, welcome to CrossValidated. Your question is not entirely clear. As far as I understand, $W$ has eigenvectors of the covariance matrix in columns, right? That's what you mean by "loadings"? Next, what do you mean by $x_i$, is it one column of $X$, i.e. one of the variables? What is $w_i$, a column of $W$? $\endgroup$ – amoeba says Reinstate Monica Oct 2 '15 at 9:01
  • $\begingroup$ Hi amoeba7, x_i is the column of X and w_i is the column of W. My question was is x_i and w_i related, then how? $\endgroup$ – Hiren Oct 2 '15 at 16:26
  • 2
    $\begingroup$ 1) $W$ shouldn't be called loadings. It is eigenvectors. 2) $x_i$ and $w_i$ columns do not correspond to each other in one-to-one sense. Your question is unclear. $\endgroup$ – ttnphns Oct 3 '15 at 16:27
  • 1
    $\begingroup$ @ttnphns: If only I had one (insert monetary unit of choice) every time someone calls the eigenvectors with their favourite term. $\endgroup$ – usεr11852 says Reinstate Monic Oct 3 '15 at 20:18
3
$\begingroup$

Let's start by looking at your equation.

As an example, consider a dataset with $4$ variables and $100$ data points, so that $X$ is of size $100\times 4$ (and centered). PCA constructs $4\times 4$ covariance matrix and finds its eigenvectors. Suppose we selected $2$ eigenvectors to perform the dimensionality reduction. Then $W$ is of size $4 \times 2$. Multiplying $X$ by $W^\top$ (note the transpose!), we get a $100\times 2$ matrix of scores: $$T=XW^\top,$$ or spelled out:

$$\underbrace{\left(\begin{array}{cc} |&|\\|&|\\t_1&t_2\\|&|\\|&|\end{array}\right)}_T=\underbrace{\left(\begin{array}{cc} |&|&|&|\\|&|&|&|\\x_1&x_2&x_3&x_4\\|&|&|&|\\|&|&|&|\end{array}\right)}_X\cdot {\underbrace{\left(\begin{array}{cc} |&|\\w_1&w_2\\|&|\end{array}\right)}_W}^\top.$$

Can we say that $w_i$ vector represents the importance of $x_i$?

Absolutely not! In my example, there are four $x_i$ variables, but only two $w$ vectors. There is no correspondence between a particular $x_k$ and $w_k$ at all.

Or is this information only conveyed by the scores?

No! Scores $T$ don't tell you anything about the importance of the original variables.


In fact, nothing in PCA tells you about the "importance" of the original variables.

PCA is sometimes used for feature selection, see here: Using principal component analysis (PCA) for feature selection -- this is based on the assumption that the variables contributing most to PC1 are most "important", i.e. it is the elements of $w_1$ that reflect the "importance" of the original variables. However, there is no guarantee that this assumption should always be reasonable.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.