# What is maximum number of variables that we can use to run a cluster analysis?

To do one clustering analysis, the model that I developed contains some 30 variables. I need to run this clustering for some 2-3 million data points. I need to know whether number of variables that I intend to use (30) is just too many? Would my result be impacted 'coz of large number of variables?

Also there would be issues with processing limitations of tool (spotfire) that I am using for this exercise. But that is secondary concern at this moment.

-

What makes you think there is a limit? Most clustering algorithms just use distance functions, and these usually are linear in the number of variables.

However, not every result will be meaningful. There are issues associated with high-dimensional data (albeit most people would consider 30 to be just "medium" dimensionality).

The most obvious one is that the notion of distance itself is often rather meaningless. Say, your first variable is "shoe size", the second is "age". How can you expect an euclidean distance function to perform a meaningful distance computation?

However, even when you are in the situation that the euclidean distance is meaningful, simple methods such as k-means and "hierarchical clustering" may not work very well due to the low contrast in distances. Others may be hard to parameterize, as you need to give a distance threshold. So you will need more advanced methods (and spotfire probably only offers the most basic ones, unfortunately).

You might however want to check out:

• Subspace clustering (Wikipedia), in particular a survey article such as:
Kriegel, Hans-Peter; Kröger, Peer; Zimek, Arthur (2009), "Clustering high-dimensional data: A survey on subspace clustering, pattern-based clustering, and correlation clustering", ACM Transactions on Knowledge Discovery from Data
• Correlation clustering (same wikipedia article and survey), when you have correlations in your attributes
• Problems of mining high-dimensional data in general, in particular the "curse of dimensionality" (link is to Wikipedia again, but a different article). Many people think that you cannot mine high-dimensional data at all, but there is research that shows that high-dimensional data can work, or not (again from Wikipedia. time series are a particular type of high-dimensional data - often in the thousand dimensions - that works very well, in the first article the authors touch the issue of distance functions):
Houle, Michael E.; Kriegel, Hans-Peter; Kröger, Peer; Schubert, Erich; Zimek, Arthur (2010). Can Shared-Neighbor Distances Defeat the Curse of Dimensionality? Scientific and Statistical Database Management.
Bernecker, Thomas; Houle, Michael E.; Kriegel, Hans-Peter; Kröger, Peer; Renz, Matthias; Schubert, Erich; Zimek, Arthur (2011). Quality of Similarity Rankings in Time Series. Advances in Spatial and Temporal Databases
-
 Your notion "k-means and "hierarchical clustering" may not work very well due to the low contrast in distances" is interesting. Could you elaborate a bit? – ttnphns Feb 20 '12 at 11:07 Have a look at "When is “nearest neighbor” meaningful?" by Beyer, Goldstein, Ramakrishnan. And after all, k-means is based on "nearest cluster center", and hierarchical clusering on "merge nearest two clusters". – Anony-Mousse Feb 20 '12 at 12:27 @Anony-Mousse, I understand what you are suggesting on distanced being meaningless if your variables are at different scales. All of my 30 variables will fall in two categories. For one set (15 variables) values will be btw 0-50 and for other set values (dollar amounts) would be btw 0-100K. Is there way I can normalize variables so that both set of variables are assessed on same scale? For analysis, all variables are equally important, so analysis should treat distances at same scale. – Raghvendra Feb 21 '12 at 1:17 Partially. If the values are on different scales, even a 2D distance can be meaningless. Say somone has houshold size 3 people, and donated $100, and the other household is 2 people, and they donated$100000. The dollar amount completely dominates the difference in household size. Once you get to more variables, read up on the curse of dimensionality. Often (but not necessarily, see Houle et al.) all distances appear to be the same, and somewhat random. – Anony-Mousse Feb 21 '12 at 7:45

If only for crossvalidation purposes, it would seem to make sense to choose a random subset of your millions of cases. But 30 variables shouldn't be a problem even with, say, several thousand, let alone with hundreds of thousands or millions. You have more than enough information to map out the different relationships you want to explore.

-
 Thanks Ronaldo. Your comment gives me confidence in model that I prepared to run cluster analysis on. – Raghvendra Feb 21 '12 at 1:18 Seeing as there are so many rows of data, the suggestion to set aside a subset for training and then use the rest for crossvalidation seems very sensible. – Peter Ellis Mar 24 '12 at 2:35