Split clustered data into calibration and validation sample (Cross validation) I have a dataset with >800 cases ($n$) from >30 ($k$) different organisations (clustered data). The number of cases within each organisation differ (unbalanced data; e.g.: organisation 1 = 30 cases, organisation 2 = 13 cases ...).
I want to randomly split the dataset into a calibration (training) and a validation (test) sample in order to cross-validate a structural equation model.
However, I am unsure how I should actually do the split. In my opinion, there are two valid options:


*

*Randomly splitting the dataset neglecting the clustering into different organisations (randomly choosing participants from different organisations).

*Randomly splitting the dataset based on the clustering (i.e., randomly choosing $k_a = k/2$ organisations for the calibration sample and $k_b = k - k_a$ organisations for the validation sample).


Option 1 has the advantage that I get two samples that have identical sample sizes ($n_a = n_b$). Option 2, on the other hand, has the advantage to take the clustered data structure into account but produces samples with different sample sizes ($n_a \neq n_b$).
Is there a preferred way to split datasets in cases of clustered data structures?
Ps.: I calculated intraclass correlation coefficients (ICC1, in R with multilevel::mult.icc) for all dependent variables. The ICC is below .1 for all variables. It can therefore be assumed that only small amounts of variance are explained due to organisational membership.
PPs.: I added machine-learning as tag since cross validation is often done in this field.
Edit:
I reconsidered the whole problem and came up with another option:


*Randomly choosing ~50% of individuals out of each of the $k$ different organisations. This approach would allow to keep the original cluster structure in both subsamples and $n_a = n_b$.


However, I am still quite unsure how to tackle the subsetting since I do not have a rational that guides me. I didn't not find literature yet that considers such issues.
 A: Your sample size of 800 is too low by an order of magnitude for data splitting to be a reliable validation method.  You will get much different results each time you split.  I suggest using the optimism bootstrap, repeating all possible modeling steps each of say 400 times.
In the R rms package validate and calibrate series of functions there are options for clustered/grouped bootstrapping.
A: If I understand correctly, it appears you are asking the general about cross-validation when you have unbalanced data.  In such cases, it is generally best to do stratified cross-validation.  This is essentially what your option 2 is whereby the proportion of samples in each group (e.g. organization) is equal between the testing and training datasets.  This is important in very unbalanced data because if you simply do random sampling, your training dataset may not contain all the different groups resulting in a much worse model.
Second, regarding your concern about equal sample sizes between the training and testing dataset.  That is completely normal, it is more typical to split 80:20 or even 90:10 in some settings.  You should be able to split 50:50 if you really want where $n_a=n_b$.
