38
$\begingroup$

Background: I want to classify the residential areas of a city into groups based on their social-economic characteristics, including housing unit density, population density, green space area, housing price, number of schools / health centers / day care centers, etc. I want to understand how many different groups the residential areas can be divided into, and what are their unique characteristics. This information could facilitate city planning.

Based on some examples (cf., this blog post: PCA and K-means Clustering of Delta Aircraft), I figured out the way to do the analysis is:

  1. First do PCA analysis.

  2. Determine the number of unique groups (clusters) based on PCA results (e.g., using the "elbow" method, or alternatively, the number of components that explains 80 to 90% of total variance).

  3. After determining the number of clusters, apply k-means clustering to do the classification.

My questions: it seemed the number of PCA components is related to clusters analysis. So is that's true, if, say, we found 5 PCA components explained more than 90% of variation of all features, then we would apply k-means clustering and get 5 clusters. So would the 5 groups exactly corresponding to the 5 components in PCA analysis?

In another words, I guess my question is: What is the connection between the PCA analysis and k-means clustering?

Updates: Thanks to Emre, xeon, and Kirill's inputs. So the current answers:

  1. Doing PCA before clustering analysis is also useful for dimensionality reduction as a feature extractor and visualize / reveal clusters.

  2. Doing PCA after clustering can validate the clustering algorithm (reference: Kernel principal component analysis).

  3. PCA is sometimes applied to reduce the dimensionality of the dataset prior to clustering. However, Yeung & Ruzzo (2000) showed that clustering with the PC's instead of the original variables does not necessarily improve cluster quality. In particular, the first few PC's (which contain most of the variation in the data) do not necessarily capture most of the cluster structure.

    • Yeung, Ka Yee, and Walter L. Ruzzo. An empirical study on principal component analysis for clustering gene expression data. Technical report, Department of Computer Science and Engineering, University of Washington, 2000. (pdf)
  4. It seemed PCA is necessary before a two-step clustering analysis. Based on Ibes (2015), in which cluster analysis was run using the factors identified in the PCA.

$\endgroup$
4
  • 1
    $\begingroup$ You can use PCA for dimensionality reduction as a feature extractor, and to visualize the clusters. $\endgroup$
    – Emre
    Jun 18, 2015 at 20:38
  • 3
    $\begingroup$ Start simple: run a classifier directly on the data you have and note the performance. If you are not satisfied with the performance, try PCA (select the number of components at the "knee" of sorted eigenvalue plot) and the run k-means. If you see nice clusters, there is a good chance that PCA+classifier will do a good job. $\endgroup$ Jun 18, 2015 at 21:03
  • 1
    $\begingroup$ You can do PCA after clustering too, to validate your clustering algorithm; color code each point by its cluster label. I also recommend looking into kernel PCA. $\endgroup$
    – Emre
    Jun 18, 2015 at 22:35
  • $\begingroup$ There are methods that simultaneously perform dimensionality reduction and clustering. These methods seek an optimally chosen low-dimensional representation so as to facilitate the identification of clusters. For example, see clustrd package in R and the associated references. $\endgroup$
    – Nat
    Oct 3, 2018 at 19:45

2 Answers 2

17
$\begingroup$

PCA is not a clustering method. But sometimes it helps to reveal clusters.

Let's assume you have 10-dimensional Normal distributions with mean $0_{10}$ (vector of zeros) and some covariance matrix with 3 directions having bigger variance than others. Applying principal component analysis with 3 components will give you these directions in decreasing order and 'elbow' approach will say to you that this amount of chosen components is right. However, it will be still a cloud of points (1 cluster).

Let's assume you have 10 10-dimensional Normal distributions with means $1_{10}$, $2_{10}$, ... $10_{10}$ (means are staying almost on the line) and similar covariance matrices. Applying PCA with only 1 component (after standardization) will give you the direction where you will observe all 10 clusters. Analyzing explained variance ('elbow' approach), you will see that 1 component is enough to describe this data.

In the link you show PCA is used only to build some hypotheses regarding the data. The amount of clusters is determined by 'elbow' approach according to the value of within groups sum of squares (not by explained variance). Basically, you repeat K-means algorithm for different amount of clusters and calculate this sum of squares. If the number of clusters equal to the number of data points, then sum of squares equal $0$.

$\endgroup$
2
  • $\begingroup$ Thank you for your inputs. Could you explain what is 10-dimensional Normal distributions with mean 0? Do you mean ten input feature variables and each of them follows a normal distribution? $\endgroup$
    – enaJ
    Jun 18, 2015 at 19:57
  • $\begingroup$ Sorry, I am speaking about a random variable that follows multivariate normal distribution with mean that will be 10-dimensional vector and covariance matrix that is 10x10 symmetric matrix. $\endgroup$
    – Kirill
    Jun 18, 2015 at 20:55
-2
$\begingroup$

to my opinion:

PCA extracts main non-correlated features (x) that explains most of the variety in y=f(x).

Cluster analysis mostly extracts groups of samples & their labels (or y results in y=f(x) )

$\endgroup$
4
  • 1
    $\begingroup$ These statements are not statistically correct. Neither PCA nor cluster analysis require a dependent variable. $\endgroup$
    – R Carnell
    May 30 at 15:20
  • $\begingroup$ Yes 'dependency' can be ommited (though with 'y' I ment mostly 'probabilities'), But the sence of Cluster analysis as grouping more-or-less similar samples && PCA as extraction of main non-correlated features that constitute the resulting variety -- I still think that is the matter of difference of 2 approaches. And can be applyed in any order, as mentioned above, - either first PCA, then Cluster analysis, or vc.vs $\endgroup$
    – JeeyCi
    May 31 at 6:36
  • $\begingroup$ in general wiki gives good explanation of PCA purposes -- saying "PCA is used in Exploratory Data Analysis and for making predictive models"... EDA is the main aim of using unsupervised PCA (& it really means dependency)... Cluster An. being also unsupervised does'n explain variabilty of samples $\endgroup$
    – JeeyCi
    May 31 at 6:46
  • $\begingroup$ BTW, here is the article, answering the topic's title... and here suggested the contrary order -- everything depends on the overall size of data - many use PCA for dim.reduction.. In any case, PCA can also be helpfull in outliers identification (but can fail in non-gaussian distr.) ... AND I believe ONLY Cluster an. helps to distinguish outlier vs. novelty $\endgroup$
    – JeeyCi
    Jun 9 at 16:40

Not the answer you're looking for? Browse other questions tagged or ask your own question.