Do I need to rescale dummy variables for PCA?

Question

I am wondering if for a PCA for which I rescale my numeric variables, I need to rescale my dummy variables as well ? I have read on the internet that I should not but I do not see why. I guess the question is very similar to this one : whether to rescale indicator / binary / dummy predictors for LASSO but I was not sure about the answer. If it is no, please explain why. Thanks

Answer 1

This depends on whether your PCA is based on the eigendecomposition of the correlation matrix ($\mathbf{R}$) or the covariance matrix ($\mathbf{\Sigma}$). In the former case PCA is insensitive to linear transformations (e.g. $a\mathbf{X} + b$, for values of constants $0 < a$, and $-\infty < b < \infty$) of your data, $\mathbf{X}$. PCA on $\mathbf{R}$ carries an assumption that each variable (regardless of scaling or centering) contributes precisely one unit to total variance (out of $p$ total units of variance, where $p$ is the number of variables).

By contrast, PCA on $\mathbf{\Sigma}$ is sensitive to linear transformations, in that the eigenvalues are different, although the eigenvectors (e.g. loadings) are not. This form of PCA assumes that each variable contributes its actual variance to total variabce, so that total variance equals $\text{trace}(\mathbf{\Sigma})$. Standardizing transformations of variables (or transforming variables by dividing by their standard deviations) will make $\mathbf{\Sigma} = \mathbf{R}$.