Normalizing vs Scaling before PCA

Question

I know there's a lot of content about PCA pre-processing, but I am still somewhat confused.

I have a dataset that contains some clear patterns: 1 variable is whether a person has financial resources (Yes/No), another is the volume of financial_resources (=0 for people that have no resources), then others are variables that tend to be correlated with the volume of financial resources. The dataset also contains several dummified variables, and some discrete variables ranging between 0 and 5.

I would assume I would find two clusters: one for people with no resources, another for people with resources.

When applying PCA with two components, I had two approaches: - Scale, then apply PCA - Normalize, then apply PCA

This leads to completely different results. I know that scaling is a standard pre-processing step to PCA. But does normalizing make any sense? I figure I have done something close to a Sparse PCA, where I captured the most significant patterns first then fitted the PCA. Can you give me some insights on this?

Note that the goal of my PCA is to visualize the results of a clustering algorithm.

Fig 1 - Top 2 components with scaling

Fig 2 - Top 2 components with normalizing

Answer 1

Scaling (what I would call centering and scaling) is very important for PCA because of the way that the principal components are calculated. PCA is solved via the Singular Value Decomposition, which finds linear subspaces which best represent your data in the squared sense. The two parts I've italicized are the reason that we center and scale (respectively).

Linear Subspaces are an important topic of study in Linear Algebra and the most important consequence of a linear subspace for PCA is that it has to go through the origin, the point [0, 0, ..., 0]. So if, say, you're measuring something like the GDP and population of a country, your data are likely to live very far from the origin, and be poorly approximated by any linear subspace. By centering our data, we guarantee that they exist near the origin, and it may be possible to approximate them with a low dimension linear subspace. In your case your data seem to all be positive, so they are most certainly not centered around 0 prior to preprocessing.

Here is an example of 2D a dataset far from the origin, which gets a useless first component until it is centered:

Scaling is important because SVD approximates in the sum of squares sense, so if one variable is on a different scale than another, it will dominate the PCA procedure, and the low D plot will really just be visualizing that dimension.

I will illustrate with an example in python.

Let's first set up an environment:

import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import scale, normalize
import matplotlib.pyplot as plt
plt.ion()

# For reproducibility 
np.random.seed(123)

We're going to generate data that are standard normal/uncorrelated in 4 dimensions, but with one additional variable that takes value either 0 or 5 randomly, giving a 5 dimensional dataset that we wish to visualize:

N = 200
P = 5
rho = 0.5

X = np.random.normal(size=[N,P])
X = np.append(X, 3*np.random.choice(2, size = [N,1]), axis = 1)

We will first do PCA without any preprocessing:

# No preprocessing:
pca = PCA(2)
low_d = pca.fit_transform(X)
plt.scatter(low_d[:,0], low_d[:,1])

Which produces this plot:

We clearly see two clusters, but the data were generated completely at random with no structure at all!

Normalizing changes the plot, but we still see 2 clusters:

# normalize
Xn = normalize(X)
pca = PCA(2)
low_d = pca.fit_transform(Xn)
plt.scatter(low_d[:,0], low_d[:,1])

The fact that the binary variable was on a different scale from the others has created a clustering effect where one might not necessarily exist. This is because the SVD considers it more than other variables as it contributes more to squared error. This may be solved by scaling the dataset:

# Scale
Xs = scale(X)
low_d = pca.fit_transform(Xs)
plt.scatter(low_d[:,0], low_d[:,1])

We finally see (correctly) that the data are completely random noise.

I conjecture that in your case your 0-5 variable may be dominating the 0-1 dummy variables, leading to clustering where there shouldn't be any (does it happen that the 0-5 variable accumulates on the edges of the scale?).

Edit: I recently came across the concept of the Spatial Sign Covariance Matrix, which would appear to conduct the normalization we discussed in order to generate a robust covariance matrix. Eigenanalysis thereupon would yield a normalized PCA algorithm, which is discussed in this article.

Answer 2

I'd like to point out sklearn normalize and scale use different default axes. normalize defaults to axis=1 whereas scale defaults to axis=0. Following @John Madden great example, but if we do normalization along axis=0, the two clusters will also disappear.

import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import scale, normalize
import matplotlib.pyplot as plt

# For reproducibility 
np.random.seed(123)

N = 200
P = 5
rho = 0.5

X = np.random.normal(size=[N,P])
X = np.append(X, 3*np.random.choice(2, size = [N,1]), axis = 1)

X = normalize(X, axis=0)

# No preprocessing:
pca = PCA(2)
low_d = pca.fit_transform(X)
plt.scatter(low_d[:,0], low_d[:,1])
plt.show()