My colleague and I have a very large dataset on agricultural data. There are over 20,000 rows and this was collected as a weighted survey. Most of our variables are binary (yes/no) on characteristics (heat tolerance, cold tolerance, susceptibility to some diseases, etc.). There are over 60 variables in the full, few are categorical, most binary. We are wishing to complete work in STATA but it is very unclear what technique is right to look at most common co-occurring characteristics of the independent variables. That is, we very much expect cluster of 3+ characteristics to be found (heat tolerance, but susceptible to disease x and disease y... for example). So we would like to be seeing the distinctively groupings in total data and in post-estimate assess strength of those.
For this we considered cluster analysis with twelve of the binary variables:
cluster wardslinkage ASH_CON V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12, measure(L2)
cluster tree
But it always gives error:
too many leaves; consider using the cutvalue() or cutnumber() options
When cutvalue or cutnumber is tried it is rejected because:
cannot cut exactly x groups because of ties in the dendrogram
We are told this is because STATA cluster dislikes very large datasets.
So we look for alternative data exploration technique. Since this is survey weighted data and each binary characteristic is a separate variable is there a technique in STATA or perhaps R that permit seeing the variables most commonly are together like cluster analysis but good for very large data?