1
$\begingroup$

Background

Let's say I ran a model (partially stochastic) 100 times, with 10 repetitions (with 10 different seeds) on 10 data random subsamplings. I measured an error metric. I am thus able to draw 10 boxplots to visualise the difference between the 10 subsets.

To see whether one subset dominates an other one, I did a Kruskal-Wallis test. If the test is significant, I could do pairwise Mann–Whitney U tests (correct me if I'm wrong).

Question

I want to identify whether the variability in my error metric comes from the repetitions on the same subset (i.e. from the random part of my model) or comes from the difference between data subsets (i.e. from the random subsampling). Thanks to Kruskal-Wallis, I can tell if there is a significant difference between subsamplings, but how may I go further in the analysis ?

Thanks in advance !

EDIT: following Björn answer

As explained in comments, I runned a regression model with a random effect on the sampled dataset and a random effect of the seed nested within dataset (+ a constant).

With lme4 package in R:

lmer(metric ~ (1 | subset) + (1 | rep:subset))

As expected, I got the following error:

Error: number of levels of each grouping factor must be < number of observations (problems: rep:subset)

The number of observations (10*10) is equal to the number of random effects (100). My model is overspecified, since I have only one measure per repetition (as model output is deterministic with a fixed seed). But I'm able to force lmer, with the following code:

lmer(metric ~ (1 | subset) + (1 | rep:subset), control=lmerControl(check.nobs.vs.nlev = "ignore", check.nobs.vs.rankZ = "ignore", check.nobs.vs.nRE="ignore"))

In this case, I get the following warning:

boundary (singular) fit: see ?isSingular

Again, this is indicating that the model is overfitted. The random effects may be too complex to be supported by the data ? One of the random effect is very small:

Random effects:
 Groups     Name        Std.Dev.
 rep:subset (Intercept) 0.6914  
 subset     (Intercept) 0.0000  
 Residual               0.7224

Are these results still interpretable ? Can one argue that since the variance of subset random effect is zero, that means there is no effect of random subsampling on the error metric variability ?

Thanks again !

Improve this question
$\endgroup$
10
  • $\begingroup$ Have you considered running a model that explicitly models this and estimate each source of variability? Something like a regression model like explaining a suitably scaled error metric through a constant (assuming all datasets about the same size), a random effect of the sampled dataset (I guess you could also assume that those that overlap more would be more correlated, but if you have a lot of data that might not vary much, so could be negligible) and a random effect of the seed nested within dataset? $\endgroup$
    – Björn
    Jul 19, 2022 at 7:53
  • $\begingroup$ Thank you for your answer. That seems to be a very good idea ! My metric ranges from 0 to 1, is the scaling still important ? I forgot to precise each ith repetition on a subset was done with the same seed i, but I guess it does not change your approach. $\endgroup$
    – carambarbe
    Jul 19, 2022 at 8:06
  • $\begingroup$ @Björn, in fact if it's the same seed i, i might use crossed random effect rather than nested one ? $\endgroup$
    – carambarbe
    Jul 19, 2022 at 8:26
  • $\begingroup$ I doubt the behaviour of the model for the same see used on different subsets shares any similarities. At least, it would rather surprise me for something like neural networks, RF or GBDT. Thus, why I suggested something like nested. $\endgroup$
    – Björn
    Jul 19, 2022 at 8:43
  • $\begingroup$ Ok, thank you very much. I have only one measure per repetition (since it's deterministic with a fixed seed), so adding (1 | subset : rep) (as one can specified in lme4 in R) is likely to cause an error, right ? $\endgroup$
    – carambarbe
    Jul 19, 2022 at 8:54

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.