Dig deeper on "Determine the Number of Clusters and Validate It"

Question

Updates to this thread: Based on Anony-Mousse's comments on my current results, there is only one big cluster in my data set. However, I think it might still be possible to reveal the clusters if I dig deepr (e.g., get rid of attributes that might be useless or distracting). Any suggestions on how to better organize the data to help the clusters arise?

I want to classify the residential areas of a city into groups based on their social-economic characteristics, including housing unit density, population density, green space area, housing price, number of schools/health centers/day care centers, etc. So that we can understand how many different groups the residential areas can be divided into? And what are their unique characteristics? My data looks likes:

I have 15 variables and their correlation is not strong, most around 0.1 to 0.2. I plan to do the k-mean clustering for this grouping task. To determine the number of clusters, I first did PCA analysis, and this is the cumulative proportion of components:

It seemed 9 components would account for >80% of cumulative propotrion. But I think 9 clusters is too many for my project. So I used the NbClust package to determine the number of clusters. It reported 3 clusters is the best based on 26 criteria.

My questions: 1) It seemed that any arbitrary data sets can be classified into groups, as long as we apply k-mean clustering. Is that possible some data sets naturally don't have any structure and cannot be divided into groups (e.g., uniform distributed in all feature space and don't have any pattern)? I am concerned about whether my data set really has any structure or not, since the cumulative proportion of components is so low.

2) Is the 3 clusters recommendation given by NbClust reasonable? How can I validate this?

By the way, using plot(comp, col=k$clust, pch=16) command, the 3 components looks like:

Answer 1

1) It seemed that any arbitrary data sets can be classified into groups, as long as we apply k-mean clustering. Is that possible some data sets naturally don't have any structure and cannot be divided into groups (e.g., uniform distributed in all feature space and don't have any pattern)? I am concerned about whether my data set really has any structure or not, since the cumulative proportion of components is so low.

Borrowing an analogy from regression modeling, you may estimate non-zero correlations, but you need to use inference to determine if these differences are significant. Or, if you were interested in prediction, you might use split-sample validation to confirm that the regression model indeed has predictive value. Clustering is exactly like that. Estimating the number of clusters as well as predicting to which clusters individuals belong is a bit difficult to calibrate. Bayesian models make it a bit easier. If you're not comfortable with that, take it as a prediction validation problem. You should consider split sample, leave-one-out, or k-fold and compare their merits and flaws.

2) Is the 3 clusters recommendation given by NbClust reasonable? How can I validate this?

It depends entirely upon your desired application. The most practical way to verify classification from PCA is to look at distributions and at least "verify" that they make sense. For instance, I recall the Netflix prize statistical modeler first applied PCA and found 2 principal components in a scree plot accounted for a substantial amount of variation. When he applied rotation and plotted the movies, he found the axes distinctly represented fantasy vs. reality and action vs. intimacy.

Answer 2

There might be no clusters. And unfortunately, many methods will not notice, and divide your data nevertheless.

Judging from your last visualization, I'd say all the data is in one big cluster; sorry. Judging from the WCSS plot, 12 might be your best bet. But carefully visualize and analyse the result.

But if you dig deeper into your data (you have attributes with mixed scales - such data needs to be handled very carefully. Some attributes might be useless or distracting) clusters might still arise.