My aim is to compare the performance of non-nested but closely related mixed-effects regression models fit to two datasets of different sample size. The data was obtained from behavioral experiments using different tasks but analyzed for the role of the same predictors. At a first (qualitative) glance, the models seem to perform very well on one dataset and quite poorly on the second dataset. But because of the differences between the two datasets, I cannot tell for sure if the performance is really much different. I need to know the exact answer in order not to hand-waive my interpretation of the supposed differences.
I have researched the issue and have come up with a specific procedure for mixed-effect regression, since measures for assessing the performance of this kind of regression (or the estimation of such measures) differ somewhat from those typically used for fixed effects regression. I would like to get feedback from you whether this procedure is correct. I also have unresolved questions. I will cite all consulted books, articles, and Cross-validated posts used to come up with this procedure.
- Before assessing performance I check whether the two sets are statistically independent (the use of different tasks in experiments in which the data was elicited provides one important reason for assuming independence, yet the fact that they are used to examine the same phenomenon might suggest they are not entirely independent). This is not a trivial question, since if the two sets are not independent, one needs to factor in the correlations at a later point when using tests to test for significant differences in performance, as argued in this post.
To test for independence, I use Chi-square because it does not assume equal variances between the two groups or homoskedasticity in the data (see McHugh 2013). Moreover, Chi-square does not require equal sample sizes, as argued in this post.
Now, let's assume that this test shows the two sets to be significantly different (my first question is how to assess performance of models if the test fails to show independence, see Question 1 below for details). This brings be to step 2.
In order to assess the performance of two datasets A and B relative to each other, we first need to ensure we are comparing datasets of the same size. This can be done by subsampling with replacement from the larger dataset the number of datapoints that corresponds to the size of the smaller dataset. The subsampling can be run, say, a 1000 times. This step is taken in its entirely from this post. Once I have obtained sets of equal sizes, I can compare the performance of mixed-effect models on these sets.
I sample with replacement from the two sets for a 1000 times and to each of the 1000 bootstrapped samples from the two datasets I fit a mixed-effects regression model (once again, note that there are slight differences in the fixed effects between the model used for the two respective datasets). From each model I record the AIC and then calculate the bootstrapped mean AIC and confidence intervals for the two datasets (Question 2). The reason I'm using AIC here is that it factors in differences in model complexity (e.g. different number of predictors), although I could also use Area under ROC curve instead. I then check if the mean AICs are different by means of another test (Question 3).
Is this procedure correct? Then I have three specific questions.
If the Chi-square test fails to show independence at step 1, then this post proposes that I should only select the points that are independent. Assuming this is the way to proceed, how do I determine which specific points are not independent?
In the post I adapted step 3 from, what is extracted from each model is the difference between the exact same model fit to a pairs of samples from two datasets. What I'm doing here is different, in that I am calculating two bootstrapped means and then test for the difference between them. Since my models are not exactly the same, I cannot extract differences directly. Is testing for the difference between mean AICs appropriate here?
This question depends on the distribution of AIC values (or AUC values if one chooses that measure instead). What test is appropriate here? Can I use a parametric t-test here and assume that the two AIC sets have the same variances and are normally distributed?
I'd be grateful for any and all of your help. Even answering one of the questions will help me piece together the whole answer eventually.
Later edit, after more research and thought:
I think that the Chi-square for independence is not useful for me here, because it is used to establish whether there is a relation between two categorical variables applied to the same sample. I think I found a more appropriate test.
But before I get into the test, I need to provide more insight into the data. X1 amd X2 consist of behavioral responses to stimuli varying along parameter P (i.e. critical predictor) using different experimental task. Despite task differences both studies used their data to advance the same claim: humans are sensitive to P. Before I can answer the question whether X1 was a better fit to M1 than X2 was to M2 (given that both M1 and M2 test for sensitivity to P but employ different controls), I need to understand if the two tasks are not different enough to ascribe potential differences in performance to prior task differences. In other words, there are two questions here than need to be answered: 1) task difference (i.e. do both sets of data come from the same source, in a generative sense?), 2) performance differences on a given task (i.e did M1 show more sensitivity to P than M2 did?). The first question corresponds to my step 1, the second question covers steps 2 and 3.
It seems to me that, in statistical terms, testing whether X1 and X2 come from the same source means testing whether X1 and X2 have the same distribution (i.e. significantly different means and variances) . That can be done using a paired Student's T-test. Since the Student t-test assumes equal variances and my samples are of unequal size I need to use Welch-Satterthwaite version of the t-test, which does not assume equal variances (see this post). So I think this is the test I am looking for. Would you agree? (This edit effectively substitutes Step 1 and Question 1).