What is maximum number of variables that we can use to run a cluster analysis?

Question

To do one clustering analysis, the model that I developed contains some 30 variables. I need to run this clustering for some 2-3 million data points. I need to know whether number of variables that I intend to use (30) is just too many? Would my result be impacted 'coz of large number of variables?

Also there would be issues with processing limitations of tool (spotfire) that I am using for this exercise. But that is secondary concern at this moment.

Answer 1

What makes you think there is a limit? Most clustering algorithms just use distance functions, and these usually are linear in the number of variables.

However, not every result will be meaningful. There are issues associated with high-dimensional data (albeit most people would consider 30 to be just "medium" dimensionality).

The most obvious one is that the notion of distance itself is often rather meaningless. Say, your first variable is "shoe size", the second is "age". How can you expect an euclidean distance function to perform a meaningful distance computation?

However, even when you are in the situation that the euclidean distance is meaningful, simple methods such as k-means and "hierarchical clustering" may not work very well due to the low contrast in distances. Others may be hard to parameterize, as you need to give a distance threshold. So you will need more advanced methods (and spotfire probably only offers the most basic ones, unfortunately).

You might however want to check out:

• Subspace clustering (Wikipedia), in particular a survey article such as:
  Kriegel, Hans-Peter; Kröger, Peer; Zimek, Arthur (2009), "Clustering high-dimensional data: A survey on subspace clustering, pattern-based clustering, and correlation clustering", ACM Transactions on Knowledge Discovery from Data
• Correlation clustering (same wikipedia article and survey), when you have correlations in your attributes
• Problems of mining high-dimensional data in general, in particular the "curse of dimensionality" (link is to Wikipedia again, but a different article). Many people think that you cannot mine high-dimensional data at all, but there is research that shows that high-dimensional data can work, or not (again from Wikipedia. time series are a particular type of high-dimensional data - often in the thousand dimensions - that works very well, in the first article the authors touch the issue of distance functions):
  Houle, Michael E.; Kriegel, Hans-Peter; Kröger, Peer; Schubert, Erich; Zimek, Arthur (2010). Can Shared-Neighbor Distances Defeat the Curse of Dimensionality? Scientific and Statistical Database Management.
  Bernecker, Thomas; Houle, Michael E.; Kriegel, Hans-Peter; Kröger, Peer; Renz, Matthias; Schubert, Erich; Zimek, Arthur (2011). Quality of Similarity Rankings in Time Series. Advances in Spatial and Temporal Databases

Answer 2

If only for crossvalidation purposes, it would seem to make sense to choose a random subset of your millions of cases. But 30 variables shouldn't be a problem even with, say, several thousand, let alone with hundreds of thousands or millions. You have more than enough information to map out the different relationships you want to explore.