Let's say I have X1, X2, X3, and X4. All four variables are strictly positive (no values below zero). The variables are on different scales.

I do a PCA on the four variables' correlation matrix and get the first principal component, with loadings on X1, X2, X3, and X4 of 0.8, 0.5, 0.4, and 0.7 respectively.

How come first principal component contains negative values? And is it possible to identify the value of the first principal component at which X1, X2, X3, and X4 = 0?

  • $\begingroup$ Do you refer to the component scores? So is your question how the component scores can be negative? Regarding your second question: What is the ultimate goal? $\endgroup$ – StoryTeller0815 Aug 3 at 6:11
  • $\begingroup$ Regarding PC scores - PCA by default centers data first which is implied in computation of a covariance matrix. The result is that the origin of the PCA transform is moved onto the cloud's centroid. Hence, there will be positive and negative scores. It is possible to do PCA without centering of data, then all scores of the 1st PC will have one sign (if all input valus had one sign) stats.stackexchange.com/a/22331/3277 $\endgroup$ – ttnphns Aug 3 at 6:34

Regarding your first question: PCA scores are latent variables. From a more general perspective, scale and mean of latent variables are essentially unidentified. Therefore, constraints/conventions are necessary. PCA scores are typically centered at zero such that zero means "an observation that has an average score on the PCA". Negative values just mean "lower than average" component scores. This makes sense because when you conduct PCA on a covariance or correlation matrix instead of the raw data, you essentially remove any information about means from your data (because data are inherently centered in the computation of the correlation matrix). Here's a minimal example from R for that:

library(mvtnorm); library(psych)
sigma = matrix(0.7,ncol = 10, nrow = 10)
diag(sigma) = 1
X = rmvnorm(n = 200, mean = rep(100,10), sigma = sigma)

fit = princomp(X,scores = TRUE)



Regarding your second question: If you are interested in the value of the component at certain values of the observed variables, you just need to insert these values into the scoring function of the PCA. BUT pay attention that your data is inherently centered when conducting the PCA. The covariance matrix is X'X/(n-1) where X is your centered data matrix. That is, from the perspective of PCA, your data was actually centered. Therefore, I think more discussion is necessary about this point and I refrain from making more specific recommendations without knowing more in detail why you need this.

Edit: Just to be precise. Sometimes PCA is conducted on the uncentered matrix X'X. Then, deviations from the origin also occur as principal components. However, the default in most software packages is to center before calculating X'X and I assumed that you did use some standard software.


Principal components are the eigenvectors of the scatter matrix, and if $u$ is a PC, $-u$ could also be chosen as PC. So, even if your all PCs have all positive values, we can invert them to have negative ones. The critical equation is, (let $X_c$ be the centered data): $$X_c^TX_c=\sum\lambda_iu_iu_i^T$$ This is satisfied by both $u_i$ and $-u_i$.

Moreover, PCs show the directions of variation in the data. The first PC is the direction of the most variation. By enforcing all positive values, you're confining the data to the first orthant, but it may not be. Consider a 2D data, which is scattered around $y=-x$ line with some small noise. The first PC will be $[1/\sqrt{2},-1/\sqrt{2}]$, which has both positive and negative values.

  • $\begingroup$ Sorry, but I think you are missing the core of the question. At first, I also thought that some basic indeterminacy was overlooked here but I think the question is more about the fact why negative values occur and not about sign indeterminacies. $\endgroup$ – StoryTeller0815 Aug 3 at 6:51

Figured this out myself. Simply add the lowest value of the first principal component to bound the variable at zero, which corresponds to the "lowest" values of the input variables.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.