0
$\begingroup$

I have three data sets that are connected through one column( all of them have a similar column called "ID"). However the data sets are large and when I want to merge them together in R, it return a memory error. therefore, I was wondering whether I can do cluster analysis separately on each data sets and then compare the result of the cluster analysis. I'll try to make it a bit more clear. I have these three data sets from the International Corpus of Learner English; TAACO_ICLE,TAALES_ICLE, L2SCA_ICLE. each measures lexical, discoursal, and syntactical complexity respectively. By doing cluster analysis, I want to see if they constitute separate subconstructs of linguistic complexity. I have realized that cluster analysis is always done on one data set, so I was wondering if such a comparison between different data sets is even possible.

$\endgroup$
0
$\begingroup$

Since clusters give us a notion of similarity between data points in the same dataset, it would be difficult to map the meaning of clusters from dataset A onto clusters from dataset B. (please keep in mind that I'm keeping this response very generic for any domain)

Given the memory issues you're having, however, I'd consider taking smaller samples from each dataset - while verifying using some representative feature that dataset distribution is more or less intact in the samples - and combining these smaller samples into one representative dataset.

I'd then proceed to run clustering on this "representative" dataset. To be certain, I'd also repeat the above with different random (but stratified) samples a few times to test the integrity of the number of clusters generated each time. If I obtain more or less the same number of clusters across a few different such representative datasets, I'd be more confident of the groupings generated as being representative across the original datasets.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.