1
$\begingroup$

My colleague and I have a very large dataset on agricultural data. There are over 20,000 rows and this was collected as a weighted survey. Most of our variables are binary (yes/no) on characteristics (heat tolerance, cold tolerance, susceptibility to some diseases, etc.). There are over 60 variables in the full, few are categorical, most binary. We are wishing to complete work in STATA but it is very unclear what technique is right to look at most common co-occurring characteristics of the independent variables. That is, we very much expect cluster of 3+ characteristics to be found (heat tolerance, but susceptible to disease x and disease y... for example). So we would like to be seeing the distinctively groupings in total data and in post-estimate assess strength of those.

For this we considered cluster analysis with twelve of the binary variables:

cluster wardslinkage ASH_CON V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12, measure(L2)
cluster tree

But it always gives error:

too many leaves; consider using the cutvalue() or cutnumber() options

When cutvalue or cutnumber is tried it is rejected because:

cannot cut exactly x groups because of ties in the dendrogram

We are told this is because STATA cluster dislikes very large datasets.

So we look for alternative data exploration technique. Since this is survey weighted data and each binary characteristic is a separate variable is there a technique in STATA or perhaps R that permit seeing the variables most commonly are together like cluster analysis but good for very large data?

$\endgroup$
2
  • 1
    $\begingroup$ Hierarchical CA is the best approach when there are binary features or a mix of features types. But 20000x20000 proximity matrix is too big for it. So you simply do the clustering on random subsamples of it (of size, say, 1000 objects). If there are clear clusters in your data, they must show in each subsample. $\endgroup$
    – ttnphns
    Feb 22, 2021 at 7:45
  • $\begingroup$ @ttnphns Thank you, apparently 1000 is still too much, but around 100 did work. Multiple small samples were used and slight difference in clusters seen. What would be the best technique to validate best cluster arrangement in the whole data? I am concerned the very small sample (~100) will pop out some relationship spurious and doubtful. Would principal component analysis be better outcome for the whole dataset? $\endgroup$
    – iPlexipen
    Feb 28, 2021 at 22:37

1 Answer 1

2
$\begingroup$

If the problem is really the size of the data set, reduce it and try again, so you can make sure if that's the reason or not.

If that's the reason, maybe you can compute first a correlation matrix, showing the correlations between all pairs of variables. You might find that some of them are extremely correlated, in which case you don't need all, so you can remove one and reduce the data set (e.g. if characteristic A and B have a correlation of 0.95, they are essentially giving you the same info and you could remove one of them without affecting you predictive or classification power). You can also reduce the size by applying principal component analysis to find the most meaningful combination of characteristics.

Another alternative is to apply a clustering algorithm which does not have that problem. I don't know the details of what STATA is using for clustering, but I think a naive k-means algorithm should be able to handle it (although you might need to do it in R or Python).

$\endgroup$
1
  • $\begingroup$ Thank you for this reply. The size does seem to be the issue. Unfortunately this lab responsibility is to look at the majority of the binary variables so we still are left with ~22 binary characteristics of potential agricultural resilience even after cleaning. There is much variety. I asked the other commentator too - would something like principal component analysis perhaps be better in Stata? We are exploring your suggestion for R use too. $\endgroup$
    – iPlexipen
    Feb 28, 2021 at 22:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.