-1
$\begingroup$

This question already has an answer here:

I have read a lot of threads concerning this question but I still seem not to understand the reason we normalize data for PCA. If we normalize the data, every feature is on the same scale. To make use of this normalized data for, say, making similarity prediction, we have to normalize new data and project the new data to the obtained PC's. However, if we don't normalize the training data and simply perform PCA, although the PC will change direction to favor features with large variations, we don't need to normalize input data to make similarity prediction anymore. And the new data's features will be on different scales which nicely fit into the PCA model without normalization. Wouldn't it achieve the same thing as normalizing everything and making predictions on normalized data?

$\endgroup$

marked as duplicate by amoeba, Peter Flom Jul 26 '17 at 12:02

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    $\begingroup$ What will happen in PCA without normalization if you switch a feature from being measured in meters to being measured in millimeters? $\endgroup$ – Matthew Gunn Jul 26 '17 at 1:05
  • $\begingroup$ Yes the corresponding feature will become very large but if we consistently use millimeters in all data, wouldn't it achieve the same thing as normalizing everything? $\endgroup$ – aldnoah Jul 26 '17 at 1:41
0
$\begingroup$

From slide 2 here:

Why use PCA?

  • There are several uses (and abuses) for PCA. The most important use of PCA is probably in multiple regression.
  • Suppose a response variable Y is to be regressed against a large number of covariates.
  • Variable selection techniques are not often very effective, and there may be scientific interest in including information from most or all of the covariates.
  • Retaining all covariates will likely lead to severe multicollinearity or non-identifiability of regression coefficients.
  • Without remedy, standard errors will be unacceptably large, and predictions may be very inaccurate.

PCA doesn't just center and rescale the individual variables. It constructs a set of orthogonal (non-collinear, uncorrelated, independent) variables. (See the visualization here.) For many model fitting algorithms, these variables are much easier to fit than "natural" (somewhat collinear, somewhat correlated, not-independent) variables.

Edit

To see why, let's construct some simulated data. I'm using R with the tidyverse package.

library(tidyverse)

set.seed(42)
dataf = tibble(x = rnorm(50), 
               y = 1 + .75*x + .5*rnorm(50))

X and Y are highly correlated, as shown in both the scatterplot and their correlation coefficient.

ggplot(dataf, aes(x, y)) + geom_point()

X and Y are highly correlated

with(dataf, cor(x, y))
[1] 0.8588732   

We can normalize X and Y separately by converting them to z-scores. X and Y will then both have mean 0 and standard deviation 1.

dataf = dataf %>%
    mutate(x.norm = (x - mean(x) / sd(x)),
           y.norm = (y - mean(y) / sd(y)))

But they're exactly as correlated as before. Centering and rescaling doesn't change correlation. As an exercise, you might show that, for any variables X and Y and constants a and b, $$Cor(X,Y) = Cor(aX + c, Y).$$

ggplot(dataf, aes(x.norm, y.norm)) + geom_point()

After centering and rescaling, X and Y are still correlated

with(dataf, cor(x.norm, y.norm))
[1] 0.8588732

Now let's extract the principal components. The plot is a scree plot. It shows the variance associated with each individual principal component. The first principal component has much more variance than the second. The calculation shows the fraction of total variance associated with each principal component; it confirms that about 93% of the total variance is captured by the first component.

pc = prcomp(dataf[c('x','y')], scale. = TRUE)
screeplot(pc)

Screeplot shows much more variance in the first principal component

pc$sdev^2 %>% {./sum(.)}
[1] 0.92943659 0.07056341

Let's add these to the dataframe for plotting.

dataf = pc$x %>% 
    as_tibble() %>%
    bind_cols(dataf, .)

ggplot(dataf, aes(PC1, PC2)) + geom_point()

Plotting shows that the principal components are uncorrelated

with(dataf, cor(PC1, PC2))
[1] -4.379748e-16

The plot and correlation coefficient indicate that PC1 and PC2 are effectively uncorrelated.

$\endgroup$
  • $\begingroup$ Sorry I am not quite familiar with all the terminologies... Could you elaborate on why normalization's purpose is not only centering and rescaling? What other purposes specifically make it so desirable before implementing PCA? $\endgroup$ – aldnoah Jul 26 '17 at 1:46
  • $\begingroup$ I've added more detail $\endgroup$ – Dan Hicks Jul 26 '17 at 12:53

Not the answer you're looking for? Browse other questions tagged or ask your own question.