I know there's a lot of content about PCA pre-processing, but I am still somewhat confused.

I have a dataset that contains some clear patterns: 1 variable is whether a person has financial resources (Yes/No), another is the volume of financial_resources (=0 for people that have no resources), then others are variables that tend to be correlated with the volume of financial resources. The dataset also contains several dummified variables, and some discrete variables ranging between 0 and 5.

I would assume I would find two clusters: one for people with no resources, another for people with resources.

When applying PCA with two components, I had two approaches: - Scale, then apply PCA - Normalize, then apply PCA

This leads to completely different results. I know that scaling is a standard pre-processing step to PCA. But does normalizing make any sense? I figure I have done something close to a Sparse PCA, where I captured the most significant patterns first then fitted the PCA. Can you give me some insights on this?

Note that the goal of my PCA is to visualize the results of a clustering algorithm.

Fig 1 - Top 2 components with scaling


Fig 2 - Top 2 components with normalizing



Scaling (what I would call centering and scaling) is very important for PCA because of the way that the principle components are calculated. PCA is solved via the Singular Value Decomposition, which finds linear subspaces which best represent your data in the squared sense. The two parts I've italicized are the reason that we center and scale (respectively).

Linear Subspaces are an important topic of study in Linear Algebra (and generally functional analysis), but the most important consequence of a linear subspace for PCA is that it has to go through the origin, the point [0, 0, ..., 0]. So if, say, you're measuring something like the GDP and population of a country, your data are likely to live very far from the origin, and be poorly approximated by any linear subspace. By centering our data, we guarantee that they exist near the origin, and it may be possible to approximate them with a low dimension linear subspace. In your case your data seem to all be positive, so they are most certainly not centered around 0 to prior to preprocessing.

Here is an example of 2D a dataset far from the origin, which gets a useless first component until it is centered:

centerPCA Scaling is important because SVD approximates in the sum of squares sense, so if one variable is on a different scale than another, it will dominate the PCA procedure, and the low D plot will really just be visualizing that dimension.

I will illustrate with an example in python.

Let's first set up an environment:

import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import scale, normalize
import matplotlib.pyplot as plt

# For reproducibility 

We're going to generate data that are standard normal/uncorrelated in 4 dimensions, but with one additional variable that takes value either 0 or 5 randomly, giving a 5 dimensional dataset that we wish to visualize:

N = 200
P = 5
rho = 0.5

X = np.random.normal(size=[N,P])
X = np.append(X, 3*np.random.choice(2, size = [N,1]), axis = 1)

We will first do PCA without any preprocessing:

# No preprocessing:
pca = PCA(2)
low_d = pca.fit_transform(X)
plt.scatter(low_d[:,0], low_d[:,1])

Which produces this plot: unprocessed

We clearly see two clusters, but the data were generated completely at random with no structure at all!

Normalizing changes the plot, but we still see 2 clusters:

# normalize
Xn = normalize(X)
pca = PCA(2)
low_d = pca.fit_transform(Xn)
plt.scatter(low_d[:,0], low_d[:,1])


The fact that the binary variable was on a different scale from the others has created a clustering effect where one might not necessarily exist. This is because the SVD considers it more than other variables as it contributes more to squared error. This may be solved by scaling the dataset:

# Scale
Xs = scale(X)
low_d = pca.fit_transform(Xs)
plt.scatter(low_d[:,0], low_d[:,1])


We finally see (correctly) that the data are completely random noise.

It is uncommon to see normalization in PCA. I can't tell you why this is, and, in fact, I have found through some personal (unpublished) preliminary research of mine that it can be useful to normalize for the purpose of rendering the procedure robust to outliers, but I can't suggest this as I haven't seen it in the published literature (no longer true, see below).

I conjecture that in your case your 0-5 variable may be dominating the 0-1 dummy variables, leading to clustering where there shouldn't be any (does it happen that the 0-5 variable accumulates on the edges of the scale?).

Edit: I recently came across the concept of the Spatial Sign Covariance Matrix, which would appear to conduct the normalization we discussed in order to generate a robust covariance matrix. Eigenanalysis thereupon would yield a normalized PCA algorithm, which is discussed in this article.

  • $\begingroup$ This is a really explicit example, thanks a lot! In my case, there is one dominant variable, I believe: the volume of resources, which can reacfh much higher values. There is a large amount of 0s (and another variable that isequal to 1 if the resources are equal to zero), so it must clearly be these two variables which build up the "clusters" when scaling is not applied. It thus appears normalizing is generally a mistake. $\endgroup$ – Konrad Jan 6 '19 at 16:22
  • $\begingroup$ Gochta, you can check for sure by coloring the plot according to whether each point is 0 or 1 in each variable $\endgroup$ – John Madden Jan 6 '19 at 16:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.