This expounds upon the insightful hint provided in a comment by @ttnphns.
Adjoining nearly correlated variables increases the contribution of their common underlying factor to the PCA. We can see this geometrically. Consider these data in the XY plane, shown as a point cloud:
There is little correlation, approximately equal covariance, and the data are centered: PCA (no matter how conducted) would report two approximately equal components.
Let us now throw in a third variable $Z$ equal to $Y$ plus a tiny amount of random error. The correlation matrix of $(X,Y,Z)$ shows this with the small off-diagonal coefficients except between the second and third rows and columns ($Y$ and $Z$):
1. & -0.0344018 & -0.046076 \\
-0.0344018 & 1. & 0.941829 \\
-0.046076 & 0.941829 & 1.
Geometrically, we have displaced all the original points nearly vertically, lifting the previous picture right out of the plane of the page. This pseudo 3D point cloud attempts to illustrate the lifting with a side perspective view (based on a different dataset, albeit generated in the same way as before):
The points originally lie in the blue plane and are lifted to the red dots. The original $Y$ axis points to the right. The resulting tilting also stretches the points out along the YZ directions, thereby doubling their contribution to the variance. Consequently, a PCA of these new data would still identify two major principal components, but now one of them will have twice the variance of the other.
This geometric expectation is borne out with some simulations in
R. For this I repeated the "lifting" procedure by creating near-collinear copies of the second variable a second, third, fourth, and fifth time, naming them $X_2$ through $X_5$. Here is a scatterplot matrix showing how those last four variables are well correlated:
The PCA is done using correlations (although it doesn't really matter for these data), using the first two variables, then three, ..., and finally five. I show the results using plots of the contributions of the principal components to the total variance.
Initially, with two almost uncorrelated variables, the contributions are almost equal (upper left corner). After adding one variable correlated with the second--exactly as in the geometric illustration--there are still just two major components, one now twice the size of the other. (A third component reflects the lack of perfect correlation; it measures the "thickness" of the pancake-like cloud in the 3D scatterplot.) After adding another correlated variable ($X_4$), the first component is now about three-fourths of the total; after a fifth is added, the first component is nearly four-fifths of the total. In all four cases components after the second would likely be considered inconsequential by most PCA diagnostic procedures; in the last case it's possible some procedures would conclude there is only one principal component worth considering.
We can see now that there may be merit in discarding variables thought to be measuring the same underlying (but "latent") aspect of a collection of variables, because including the nearly-redundant variables can cause the PCA to overemphasize their contribution. There is nothing mathematically right (or wrong) about such a procedure; it's a judgment call based on the analytical objectives and knowledge of the data. But it should be abundantly clear that setting aside variables known to be strongly correlated with others can have a substantial effect on the PCA results.
Here is the
n.cases <- 240 # Number of points.
n.vars <- 4 # Number of mutually correlated variables.
set.seed(26) # Make these results reproducible.
eps <- rnorm(n.vars, 0, 1/4) # Make "1/4" smaller to *increase* the correlations.
x <- matrix(rnorm(n.cases * (n.vars+2)), nrow=n.cases)
beta <- rbind(c(1,rep(0, n.vars)), c(0,rep(1, n.vars)), cbind(rep(0,n.vars), diag(eps)))
y <- x%*%beta # The variables.
cor(y) # Verify their correlations are as intended.
plot(data.frame(y)) # Show the scatterplot matrix.
# Perform PCA on the first 2, 3, 4, ..., n.vars+1 variables.
p <- lapply(2:dim(beta), function(k) prcomp(y[, 1:k], scale=TRUE))
# Print summaries and display plots.
tmp <- lapply(p, summary)
tmp <- lapply(p, plot)