# Cluster many thousands observations (mixed variable types). Cluster subsample and then classify the rest observations?

I'm trying to run a cluster analysis on a large dataset (70k+ observations to cluster) with mixed variables (numeric, ordinal, binary and nominal). I don't think I can create the distance matrix using SAS over the entire dataset. So, I have tried to run a hierarchical clustering using Gower's distance over a subsample of my data. I've got some questions.

1. If the above method (hierarchical clustering of a subsample) is appropriate, how can I then score the rest of the observations and assign (classify) them to the clusters obtained?

2. If the above method isn't good, what are other recommended methods to cluster a large dataset with mixed variables? (Available in SAS if possible.)

3. How can I check for correlations/multicolinearity among mixed variables? I don't know if running something like PCA or factor analysis makes sense with categorical data.

• Please check my editing of your question. – ttnphns Sep 25 '13 at 20:57

Hierarchical clustering in general does not scale well to large data sets. There are some special cases such as SLINK that need only $O(n)$ memory and $O(n^2)$ runtime (naive implementations need $O(n^2)$ memory and $O(n^3)$ runtime). So may need to look into alternative methods such as DBSCAN. DBSCAN will work with arbitrary distance measures; but you will probably not have index acceleration, so it will be $O(n^2)$ runtime, too. But it should still scale to 70k observations; I have ran DBSCAN on 100k years ago. The key is to not compute a complete distance matrix, because that needs $O(n^2)$ memory then.