My aim is to compare the performance of non-nested but closely related mixed-effects regression models fit to two datasets of different sample size. The data was obtained from behavioral experiments using different tasks but analyzed for the role of the same predictors. At a first (qualitative) glance, the models seem to perform very well on one dataset and quite poorly on the second dataset. But because of the differences between the two datasets, I cannot tell for sure if the performance is really much different. I need to know the exact answer in order not to hand-waive my interpretation of the supposed differences.

I have researched the issue and have come up with a specific procedure for mixed-effect regression, since measures for assessing the performance of this kind of regression (or the estimation of such measures) differ somewhat from those typically used for fixed effects regression. I would like to get feedback from you whether this procedure is correct. I also have unresolved questions. I will cite all consulted books, articles, and Cross-validated posts used to come up with this procedure.

  1. Before assessing performance I check whether the two sets are statistically independent (the use of different tasks in experiments in which the data was elicited provides one important reason for assuming independence, yet the fact that they are used to examine the same phenomenon might suggest they are not entirely independent). This is not a trivial question, since if the two sets are not independent, one needs to factor in the correlations at a later point when using tests to test for significant differences in performance, as argued in this post.

To test for independence, I use Chi-square because it does not assume equal variances between the two groups or homoskedasticity in the data (see McHugh 2013). Moreover, Chi-square does not require equal sample sizes, as argued in this post.

Now, let's assume that this test shows the two sets to be significantly different (my first question is how to assess performance of models if the test fails to show independence, see Question 1 below for details). This brings be to step 2.

  1. In order to assess the performance of two datasets A and B relative to each other, we first need to ensure we are comparing datasets of the same size. This can be done by subsampling with replacement from the larger dataset the number of datapoints that corresponds to the size of the smaller dataset. The subsampling can be run, say, a 1000 times. This step is taken in its entirely from this post. Once I have obtained sets of equal sizes, I can compare the performance of mixed-effect models on these sets.

  2. I sample with replacement from the two sets for a 1000 times and to each of the 1000 bootstrapped samples from the two datasets I fit a mixed-effects regression model (once again, note that there are slight differences in the fixed effects between the model used for the two respective datasets). From each model I record the AIC and then calculate the bootstrapped mean AIC and confidence intervals for the two datasets (Question 2). The reason I'm using AIC here is that it factors in differences in model complexity (e.g. different number of predictors), although I could also use Area under ROC curve instead. I then check if the mean AICs are different by means of another test (Question 3).

Is this procedure correct? Then I have three specific questions.

Question 1

If the Chi-square test fails to show independence at step 1, then this post proposes that I should only select the points that are independent. Assuming this is the way to proceed, how do I determine which specific points are not independent?

Question 2

In the post I adapted step 3 from, what is extracted from each model is the difference between the exact same model fit to a pairs of samples from two datasets. What I'm doing here is different, in that I am calculating two bootstrapped means and then test for the difference between them. Since my models are not exactly the same, I cannot extract differences directly. Is testing for the difference between mean AICs appropriate here?

Question 3

This question depends on the distribution of AIC values (or AUC values if one chooses that measure instead). What test is appropriate here? Can I use a parametric t-test here and assume that the two AIC sets have the same variances and are normally distributed?

I'd be grateful for any and all of your help. Even answering one of the questions will help me piece together the whole answer eventually.

Later edit, after more research and thought:

I think that the Chi-square for independence is not useful for me here, because it is used to establish whether there is a relation between two categorical variables applied to the same sample. I think I found a more appropriate test.

But before I get into the test, I need to provide more insight into the data. X1 amd X2 consist of behavioral responses to stimuli varying along parameter P (i.e. critical predictor) using different experimental task. Despite task differences both studies used their data to advance the same claim: humans are sensitive to P. Before I can answer the question whether X1 was a better fit to M1 than X2 was to M2 (given that both M1 and M2 test for sensitivity to P but employ different controls), I need to understand if the two tasks are not different enough to ascribe potential differences in performance to prior task differences. In other words, there are two questions here than need to be answered: 1) task difference (i.e. do both sets of data come from the same source, in a generative sense?), 2) performance differences on a given task (i.e did M1 show more sensitivity to P than M2 did?). The first question corresponds to my step 1, the second question covers steps 2 and 3.

It seems to me that, in statistical terms, testing whether X1 and X2 come from the same source means testing whether X1 and X2 have the same distribution (i.e. significantly different means and variances) . That can be done using a paired Student's T-test. Since the Student t-test assumes equal variances and my samples are of unequal size I need to use Welch-Satterthwaite version of the t-test, which does not assume equal variances (see this post). So I think this is the test I am looking for. Would you agree? (This edit effectively substitutes Step 1 and Question 1).

  • $\begingroup$ I'm not sure I understand what your overall question is. You have datasets X1 and X2; and corresponding models M1 and M2. And you want to know whether M1 fits X1 better than M2 fits X2, as measured by AIC? $\endgroup$
    – Dan Hicks
    Jul 18, 2017 at 11:27
  • $\begingroup$ That's correct. I proposed a procedure for finding this out that I am not sure about. $\endgroup$
    – Des Grieux
    Jul 18, 2017 at 11:28
  • $\begingroup$ I don't understand AIC well enough to tell you whether it's legitimate to use it for an inter-dataset comparison in this way (step 2). Whichever goodness-of-fit statistic you use in step 3, you might bootstrap on the differences and use a Z-test, rather than bootstrap the statistics and apply a test. That is: in each bootstrap round, draw samples for both X1 and X2, use these to fit M1 and M2, calculate AIC1 and AIC2 (or whatever statistic you use), and record the difference AIC1 - AIC2. This lets you estimate the sampling distribution for the difference directly. $\endgroup$
    – Dan Hicks
    Jul 18, 2017 at 11:41
  • $\begingroup$ That makes sense, thanks! That's also what Frank Harrell mentioned in the post I cited, but I wasn't sure exactly how he proposed to calculate the differences. That's clear to me now. $\endgroup$
    – Des Grieux
    Jul 18, 2017 at 11:54


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.