Let's say I ran a model (partially stochastic) 100 times, with 10 repetitions (with 10 different seeds) on 10 data random subsamplings. I measured an error metric. I am thus able to draw 10 boxplots to visualise the difference between the 10 subsets.
To see whether one subset dominates an other one, I did a Kruskal-Wallis test. If the test is significant, I could do pairwise Mann–Whitney U tests (correct me if I'm wrong).
I want to identify whether the variability in my error metric comes from the repetitions on the same subset (i.e. from the random part of my model) or comes from the difference between data subsets (i.e. from the random subsampling). Thanks to Kruskal-Wallis, I can tell if there is a significant difference between subsamplings, but how may I go further in the analysis ?
Thanks in advance !
EDIT: following Björn answer
As explained in comments, I runned a regression model with a random effect on the sampled dataset and a random effect of the seed nested within dataset (+ a constant).
With lme4 package in R:
lmer(metric ~ (1 | subset) + (1 | rep:subset))
As expected, I got the following error:
Error: number of levels of each grouping factor must be < number of observations (problems: rep:subset)
The number of observations (10*10) is equal to the number of random effects (100). My model is overspecified, since I have only one measure per repetition (as model output is deterministic with a fixed seed). But I'm able to force lmer, with the following code:
lmer(metric ~ (1 | subset) + (1 | rep:subset), control=lmerControl(check.nobs.vs.nlev = "ignore", check.nobs.vs.rankZ = "ignore", check.nobs.vs.nRE="ignore"))
In this case, I get the following warning:
boundary (singular) fit: see ?isSingular
Again, this is indicating that the model is overfitted. The random effects may be too complex to be supported by the data ? One of the random effect is very small:
Random effects: Groups Name Std.Dev. rep:subset (Intercept) 0.6914 subset (Intercept) 0.0000 Residual 0.7224
Are these results still interpretable ? Can one argue that since the variance of subset random effect is zero, that means there is no effect of random subsampling on the error metric variability ?
Thanks again !