2
$\begingroup$

I have a dataset with >800 cases ($n$) from >30 ($k$) different organisations (clustered data). The number of cases within each organisation differ (unbalanced data; e.g.: organisation 1 = 30 cases, organisation 2 = 13 cases ...).

I want to randomly split the dataset into a calibration (training) and a validation (test) sample in order to cross-validate a structural equation model.

However, I am unsure how I should actually do the split. In my opinion, there are two valid options:

  1. Randomly splitting the dataset neglecting the clustering into different organisations (randomly choosing participants from different organisations).
  2. Randomly splitting the dataset based on the clustering (i.e., randomly choosing $k_a = k/2$ organisations for the calibration sample and $k_b = k - k_a$ organisations for the validation sample).

Option 1 has the advantage that I get two samples that have identical sample sizes ($n_a = n_b$). Option 2, on the other hand, has the advantage to take the clustered data structure into account but produces samples with different sample sizes ($n_a \neq n_b$).

Is there a preferred way to split datasets in cases of clustered data structures?

Ps.: I calculated intraclass correlation coefficients (ICC1, in R with multilevel::mult.icc) for all dependent variables. The ICC is below .1 for all variables. It can therefore be assumed that only small amounts of variance are explained due to organisational membership.

PPs.: I added machine-learning as tag since cross validation is often done in this field.

Edit:

I reconsidered the whole problem and came up with another option:

  1. Randomly choosing ~50% of individuals out of each of the $k$ different organisations. This approach would allow to keep the original cluster structure in both subsamples and $n_a = n_b$.

However, I am still quite unsure how to tackle the subsetting since I do not have a rational that guides me. I didn't not find literature yet that considers such issues.

Improve this question
$\endgroup$
1
  • $\begingroup$ Option 3 can be easily realised with sampling::strata, e.g. ind <- strata(dat, stratanames="organisation", size=floor(table(dat$organisation)*.5), method="srswor")$ID_unit. Ind can then be used to subset the data. $\endgroup$
    – phx
    Nov 18, 2014 at 6:52

2 Answers 2

Reset to default
3
$\begingroup$

Your sample size of 800 is too low by an order of magnitude for data splitting to be a reliable validation method. You will get much different results each time you split. I suggest using the optimism bootstrap, repeating all possible modeling steps each of say 400 times.

In the R rms package validate and calibrate series of functions there are options for clustered/grouped bootstrapping.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I am not quite sure whether your approach will work with my kind of analysis. I am trying to estimate a theoretically derived path model. The sample size is quite big comparing to other studies in my field of research. Furthermore, it has been proposed to just split the data into half and see whether the model still holds. Do you have a study example that employed optimism bootstrap that is somehow comparable to my situation? The package description of rms is rather long and I am not sure how the package can help me. $\endgroup$
    – phx
    Nov 14, 2014 at 10:08
  • $\begingroup$ Data splitting is too inefficient for your situation. You need all the sample size you can get for the estimation phase, and a bigger sample for validation. The bootstrap has been used in a huge variety of situations. Efron published a paper on getting a confidence envelope for a phylogenetic tree, for example. Short of doing formal analysis, start with generating 10 bootstrap samples and running your whole procedure on each sample and study how the results vary. $\endgroup$
    – Frank Harrell
    Nov 14, 2014 at 13:05
1
$\begingroup$

If I understand correctly, it appears you are asking the general about cross-validation when you have unbalanced data. In such cases, it is generally best to do stratified cross-validation. This is essentially what your option 2 is whereby the proportion of samples in each group (e.g. organization) is equal between the testing and training datasets. This is important in very unbalanced data because if you simply do random sampling, your training dataset may not contain all the different groups resulting in a much worse model.

Second, regarding your concern about equal sample sizes between the training and testing dataset. That is completely normal, it is more typical to split 80:20 or even 90:10 in some settings. You should be able to split 50:50 if you really want where $n_a=n_b$.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Do you have any rational why option 2 is preferred to 1? Maybe you could even point out literature? Although 80:20 splits are quite common this is not possible with my data. The SEM approach requires large data sets. So splitting 50:50 is my only chance. Is there a package that allows to do stratified samples with R that lead to $n_a = n_b$? It seems that sampling might help but I am not quite sure. $\endgroup$
    – phx
    Nov 14, 2014 at 10:12
  • $\begingroup$ @phx, I am less familiar with SEM so speaking just from a predictive modeling background. If sampling is a possibility, you could look into cost-sensitive learning methods such as 'over-sampling' or more advanced methods like SMOTE and SMOTEBoost. Boostrapping, as mentioned by Frank, should also work for your purposes given that SEM requires larger datasets. $\endgroup$
    – cdeterman
    Nov 14, 2014 at 13:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.