# Alternatives to Cluster Analysis with Very Large Data

My colleague and I have a very large dataset on agricultural data. There are over 20,000 rows and this was collected as a weighted survey. Most of our variables are binary (yes/no) on characteristics (heat tolerance, cold tolerance, susceptibility to some diseases, etc.). There are over 60 variables in the full, few are categorical, most binary. We are wishing to complete work in STATA but it is very unclear what technique is right to look at most common co-occurring characteristics of the independent variables. That is, we very much expect cluster of 3+ characteristics to be found (heat tolerance, but susceptible to disease x and disease y... for example). So we would like to be seeing the distinctively groupings in total data and in post-estimate assess strength of those.

For this we considered cluster analysis with twelve of the binary variables:

cluster wardslinkage ASH_CON V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12, measure(L2)
cluster tree


But it always gives error:

too many leaves; consider using the cutvalue() or cutnumber() options

When cutvalue or cutnumber is tried it is rejected because:

cannot cut exactly x groups because of ties in the dendrogram

We are told this is because STATA cluster dislikes very large datasets.

So we look for alternative data exploration technique. Since this is survey weighted data and each binary characteristic is a separate variable is there a technique in STATA or perhaps R that permit seeing the variables most commonly are together like cluster analysis but good for very large data?

• Hierarchical CA is the best approach when there are binary features or a mix of features types. But 20000x20000 proximity matrix is too big for it. So you simply do the clustering on random subsamples of it (of size, say, 1000 objects). If there are clear clusters in your data, they must show in each subsample. Feb 22, 2021 at 7:45
• @ttnphns Thank you, apparently 1000 is still too much, but around 100 did work. Multiple small samples were used and slight difference in clusters seen. What would be the best technique to validate best cluster arrangement in the whole data? I am concerned the very small sample (~100) will pop out some relationship spurious and doubtful. Would principal component analysis be better outcome for the whole dataset? Feb 28, 2021 at 22:37