5 added 11 characters in body edited Feb 24 '13 at 6:17 ttnphns 40.8k1717 gold badges154154 silver badges340340 bronze badges Addition2. In Addition above I was speaking about "variable space" and "subject space" as if they are incompatible together like water and oil. I had to reconsider it and may say that - at least when we speak about PCA - both spaces are isomorphic in the end, and by that virtue we can correctly display all the PCA details - data points, variable axes, component axes, variables as points, - on a single undistorted biplot. Below are the scatterplot (variable space) and the loading plot (component space, which is the subject space, but by its "origin"genetic origin). Everything that could be shown on the one, could also be shown on the other. The pictures are identical, only rotated by 45 degrees (and reflected, in this particular case) relative each other. That was a PCA of variables v1 and v2 (standardized, thus it was r that was analyzed). Black lines on the pictures are the variables as axes; green/yellow lines are the components as axes; blue points are the data cloud (subjects); red points are the variables displayed as points (vectors). Addition2. In Addition above I was speaking about "variable space" and "subject space" as if they are incompatible together like water and oil. I had to reconsider it and may say that - at least when we speak about PCA - both spaces are isomorphic, and by that virtue we can correctly display all the PCA details - data points, variable axes, component axes, variables as points, - on a single undistorted biplot. Below are the scatterplot (variable space) and the loading plot (component space, which is the subject space, but its "origin"). Everything that could be shown on the one, could also be shown on the other. The pictures are identical, only rotated by 45 degrees (and reflected, in this particular case) relative each other. That was a PCA of variables v1 and v2 (standardized, thus it was r that was analyzed). Black lines on the pictures are the variables as axes; green/yellow lines are the components as axes; blue points are the data cloud (subjects); red points are the variables displayed as points (vectors). Addition2. In Addition above I was speaking about "variable space" and "subject space" as if they are incompatible together like water and oil. I had to reconsider it and may say that - at least when we speak about PCA - both spaces are isomorphic in the end, and by that virtue we can correctly display all the PCA details - data points, variable axes, component axes, variables as points, - on a single undistorted biplot. Below are the scatterplot (variable space) and the loading plot (component space, which is subject space by its genetic origin). Everything that could be shown on the one, could also be shown on the other. The pictures are identical, only rotated by 45 degrees (and reflected, in this particular case) relative each other. That was a PCA of variables v1 and v2 (standardized, thus it was r that was analyzed). Black lines on the pictures are the variables as axes; green/yellow lines are the components as axes; blue points are the data cloud (subjects); red points are the variables displayed as points (vectors). 4 added 1144 characters in body edited Feb 23 '13 at 11:15 ttnphns 40.8k1717 gold badges154154 silver badges340340 bronze badges Addition2. In Addition above I was speaking about "variable space" and "subject space" as if they are incompatible together like water and oil. I had to reconsider it and may say that - at least when we speak about PCA - both spaces are isomorphic, and by that virtue we can correctly display all the PCA details - data points, variable axes, component axes, variables as points, - on a single undistorted biplot. Below are the scatterplot (variable space) and the loading plot (component space, which is the subject space, but its "origin"). Everything that could be shown on the one, could also be shown on the other. The pictures are identical, only rotated by 45 degrees (and reflected, in this particular case) relative each other. That was a PCA of variables v1 and v2 (standardized, thus it was r that was analyzed). Black lines on the pictures are the variables as axes; green/yellow lines are the components as axes; blue points are the data cloud (subjects); red points are the variables displayed as points (vectors). Addition2. In Addition above I was speaking about "variable space" and "subject space" as if they are incompatible together like water and oil. I had to reconsider it and may say that - at least when we speak about PCA - both spaces are isomorphic, and by that virtue we can correctly display all the PCA details - data points, variable axes, component axes, variables as points, - on a single undistorted biplot. Below are the scatterplot (variable space) and the loading plot (component space, which is the subject space, but its "origin"). Everything that could be shown on the one, could also be shown on the other. The pictures are identical, only rotated by 45 degrees (and reflected, in this particular case) relative each other. That was a PCA of variables v1 and v2 (standardized, thus it was r that was analyzed). Black lines on the pictures are the variables as axes; green/yellow lines are the components as axes; blue points are the data cloud (subjects); red points are the variables displayed as points (vectors). 3 added 162 characters in body edited Feb 22 '13 at 19:51 ttnphns 40.8k1717 gold badges154154 silver badges340340 bronze badges I will further illustrate the same process and idea as @whuber did, but with the loading plots, - because loadings are the essense of PCA results. Here is three 3 analyses. In the first, we have two variables, $$X_1$$ and $$X_2$$ (in this example, they do not correlate). In the second, we added $$X_3$$ which is almost a copy of $$X_2$$ and therefore correlates with it strongly. In the third, we still similarly added 2 more "copies" of it: $$X_4$$ and $$X_5$$. The plots of loadings of the first 2 principal components then go. Red spikes on the plots tell of correlations between the variables, so that the bunch of several spikes is where a cluster of tightly correlated variables is found. The components are the grey lines; the relative "strength" of a component (its relative eigenvalue magnitude) is given by weight of the line. Two effects of adding the "copies" can be observed: Component 1 becomes stronger and stronger, and Component 2 weaker and weaker. Orientation of the components change: at first, Component 1 went in the middle between $$X_1$$ and $$X_2$$; as we added $$X_3$$ to $$X_2$$ Component 1 immediately re-oriented itself to follow the emergent bunch of variables; and you may be sure that after we further added two more variables to the bunch the attachment of Component 1 to that bunch of closely correlated variables became more undisputable. I will not resume the moral because @whuber already did it. Addition. Below are some pictures in response to @whuber's comments. It is about a distinction between "variable space" and "subject space" and how components orient themselves here and there. Three bivariate PCAs are presented: first row analyzes $$r=0$$, second row analyzes $$r=0.62$$, and third row $$r=0.77$$. The left column is scatterplots (of standardized data) and the right column is loading plots. On a scatterplot, the correlation between $$X_1$$ and $$X_2$$ is rendered as oblongness of the cloud. The angle (its cosine) between a component line and a variable line is the corresponding eigenvector element. Eigenvectors are identical in all three analyses (so the angles on all 3 graphs are the same). But[But, it is true, that with $$r=0$$ exactly, eigenvectors (and hence the angles) are theoretically arbitrary; because the cloud is perfectly "round" any pair of orthogonal lines coming through the origin could serve as the two components, - even $$X_1$$ and $$X_2$$ lines themselves could be chosen as the components.] The coordinates of data points (200 subjects) onto a component are component scores, and their sum of squares devided by 200-1 is the component's eigenvalue. On a loading plot, the points (vectors) are variables; they spread the space which is 2-dimensional (because we have 2 points + origin) but is actually a reduced 200-dimensional (number of subjects) "subject space". Here the angle (cosine) between the red vectors is $$r$$. The vectors are of equal, unit length, because the data had been standardized. The first component is such a dimension axis in this space which rushes towards the overal accumulation of the points; in case of just 2 variables it is always the bisector between $$X_1$$ and $$X_2$$ (but adding a 3rd variable can deflect it anyhow). The angle (cosine) between a variable vector and a component line is the correlation between them, and because the vectors are unit lenght and the components are orthogonal, this is nothing else than the coordinates, the loading. Sum of squared loadings onto the component is its eigenvalue (the component just orients itself in this subject space so as to maximize it) I will further illustrate the same process and idea as @whuber did, but with the loading plots, - because loadings are the essense of PCA results. Here is three 3 analyses. In the first, we have two variables, $$X_1$$ and $$X_2$$ (in this example, they do not correlate). In the second, we added $$X_3$$ which is almost a copy of $$X_2$$ and therefore correlates with it strongly. In the third, we still similarly added 2 more "copies" of it: $$X_4$$ and $$X_5$$. The plots of loadings of the first 2 principal components then go. Red spikes on the plots tell of correlations between the variables, so that the bunch of several spikes is where a cluster of tightly correlated variables is found. The components are the grey lines; the relative "strength" of a component (its relative eigenvalue magnitude) is given by weight of the line. Two effects of adding the "copies" can be observed: Component 1 becomes stronger and stronger, and Component 2 weaker and weaker. Orientation of the components change: at first, Component 1 went in the middle between $$X_1$$ and $$X_2$$; as we added $$X_3$$ to $$X_2$$ Component 1 immediately re-oriented itself to follow the emergent bunch of variables; and you may be sure that after we further added two more variables to the bunch the attachment of Component 1 to that bunch of closely correlated variables became more undisputable. I will not resume the moral because @whuber already did it. Addition. Below are some pictures in response to @whuber's comments. It is about a distinction between "variable space" and "subject space" and how components orient themselves here and there. Three bivariate PCAs are presented: first row analyzes $$r=0$$, second row analyzes $$r=0.62$$, and third row $$r=0.77$$. The left column is scatterplots (of standardized data) and the right column is loading plots. On a scatterplot, the correlation between $$X_1$$ and $$X_2$$ is rendered as oblongness of the cloud. The angle (its cosine) between a component line and a variable line is the corresponding eigenvector element. Eigenvectors are identical in all three analyses (so the angles on all 3 graphs are the same). But, it is true, that with $$r=0$$ exactly, eigenvectors (and hence the angles) are theoretically arbitrary; because the cloud is perfectly "round" any pair of orthogonal lines coming through the origin could serve as the two components, - even $$X_1$$ and $$X_2$$ lines themselves could be chosen as the components. On a loading plot, the points (vectors) are variables; they spread the space which is 2-dimensional (because we have 2 points + origin) but is actually a reduced 200-dimensional (number of subjects) "subject space". Here the angle (cosine) between the red vectors is $$r$$. The vectors are of equal, unit length, because the data had been standardized. The first component is such a dimension axis in this space which rushes towards the overal accumulation of the points; in case of just 2 variables it is always the bisector between $$X_1$$ and $$X_2$$ (but adding a 3rd variable can deflect it anyhow). The angle (cosine) between a variable vector and a component line is the correlation between them, and because the vectors are unit lenght and the components are orthogonal, this is nothing else than the coordinates, the loading. Sum of squared loadings onto the component is its eigenvalue (the component just orients itself in this subject space so as to maximize it) I will further illustrate the same process and idea as @whuber did, but with the loading plots, - because loadings are the essense of PCA results. Here is three 3 analyses. In the first, we have two variables, $$X_1$$ and $$X_2$$ (in this example, they do not correlate). In the second, we added $$X_3$$ which is almost a copy of $$X_2$$ and therefore correlates with it strongly. In the third, we still similarly added 2 more "copies" of it: $$X_4$$ and $$X_5$$. The plots of loadings of the first 2 principal components then go. Red spikes on the plots tell of correlations between the variables, so that the bunch of several spikes is where a cluster of tightly correlated variables is found. The components are the grey lines; the relative "strength" of a component (its relative eigenvalue magnitude) is given by weight of the line. Two effects of adding the "copies" can be observed: Component 1 becomes stronger and stronger, and Component 2 weaker and weaker. Orientation of the components change: at first, Component 1 went in the middle between $$X_1$$ and $$X_2$$; as we added $$X_3$$ to $$X_2$$ Component 1 immediately re-oriented itself to follow the emergent bunch of variables; and you may be sure that after we further added two more variables to the bunch the attachment of Component 1 to that bunch of closely correlated variables became more undisputable. I will not resume the moral because @whuber already did it. Addition. Below are some pictures in response to @whuber's comments. It is about a distinction between "variable space" and "subject space" and how components orient themselves here and there. Three bivariate PCAs are presented: first row analyzes $$r=0$$, second row analyzes $$r=0.62$$, and third row $$r=0.77$$. The left column is scatterplots (of standardized data) and the right column is loading plots. On a scatterplot, the correlation between $$X_1$$ and $$X_2$$ is rendered as oblongness of the cloud. The angle (its cosine) between a component line and a variable line is the corresponding eigenvector element. Eigenvectors are identical in all three analyses (so the angles on all 3 graphs are the same). [But, it is true, that with $$r=0$$ exactly, eigenvectors (and hence the angles) are theoretically arbitrary; because the cloud is perfectly "round" any pair of orthogonal lines coming through the origin could serve as the two components, - even $$X_1$$ and $$X_2$$ lines themselves could be chosen as the components.] The coordinates of data points (200 subjects) onto a component are component scores, and their sum of squares devided by 200-1 is the component's eigenvalue. On a loading plot, the points (vectors) are variables; they spread the space which is 2-dimensional (because we have 2 points + origin) but is actually a reduced 200-dimensional (number of subjects) "subject space". Here the angle (cosine) between the red vectors is $$r$$. The vectors are of equal, unit length, because the data had been standardized. The first component is such a dimension axis in this space which rushes towards the overal accumulation of the points; in case of just 2 variables it is always the bisector between $$X_1$$ and $$X_2$$ (but adding a 3rd variable can deflect it anyhow). The angle (cosine) between a variable vector and a component line is the correlation between them, and because the vectors are unit lenght and the components are orthogonal, this is nothing else than the coordinates, the loading. Sum of squared loadings onto the component is its eigenvalue (the component just orients itself in this subject space so as to maximize it) 2 added 2102 characters in body edited Feb 22 '13 at 19:15 ttnphns 40.8k1717 gold badges154154 silver badges340340 bronze badges 1 answered Feb 22 '13 at 8:58 ttnphns 40.8k1717 gold badges154154 silver badges340340 bronze badges