Comparing GLME fits for different data

Question

I have a data set consisting of two behavioral response variables where each response variable is associated with a relevant physiological predictor, with many trials for each subject.

The hypothesis I wish to test, which I believe is difficult if not impossible to test in this framework, is: Is ${P_1}$ a better predictor for ${B_1}$ than ${P_2}$ is for ${B_2}$

Independently, it makes sense to model each of these as a logistic regression something like

${B_1}$ ~ 1 + ${P_1}$ + (1|Subject)

${B_2}$ ~ 1 + ${P_2}$ + (1|Subject)

but no a priori reason to expect ${P_1}$ to be correlated with ${B_2}$ and vice-versa.

The response/predictors are related enough that it would be completely appropriate to model the difference between them with an additional fixed effect for "1" versus "2" and test the interaction.

However, I don't really hypothesize that the coefficients for ${P_1}$ and ${P_2}$ are necessarily different. Rather, I expect that ${P_1}$ is a better predictor for ${B_1}$ than ${P_2}$ is for ${B_2}$: I expect larger error in the second model than there is in the first.

This would be a trivial comparison if the response was the same, but it isn't. Is there a principled way to make a comparison like this in this context?

Answer 1

TL;DR:

A proper bivariate logistic regression with both $P_1$ and $P_2$ as predictors, taking into account correlations between response types and within subjects, might give you a structure within which you could do something close to what you want. Otherwise you have several types of large problems.

Details:

If you are trying to compare two different models on two different data sets, as a quick look at your question might suggest, then the following comment by @whuber on this question about comparing models applies: "If model A fits dataset 1 and model B fits dataset 2, there's nothing at all to compare: the models and the data are totally different." There are ways to use measures like residual variance, fraction of log-likelihood explained, or AIC to compare models (see for example answers in and links from the question linked above). You will find that some question whether some such comparisons are valid for non-nested models, but even those who claim such validity note that the models need to be on the same data.

Of course, you don't really have two independent data sets. You have multiple blocks of trials on the same subjects, with values of 2 continuous covariates for each subject in each block and 2 types of binary responses measured in different blocks. Both the covariates and the response types are evidently correlated. It would seem best to approach this problem with a single coherent model.

Even with a single response variable, determining which of two predictors is "better" can be fraught with difficulty. On this site, the tag feature-selection covers such issues. For example, When you leave out predictors that might be correlated both with included predictors and with outcome in a linear regression, the signs of regression coefficients for the included predictors can differ from what they would be in a full model. Follow the simpsons-paradox tag on this site for examples in several contexts.

Using logistic regressions adds another layer of complexity to comparing predictors: absence of a predictor can affect the regression coefficients even of predictors that are uncorrelated to it. The variance in logistic regression, in a true model, only includes variance from random effects plus an assumed fixed variance from the logistic distribution of the error term (or equivalently from the combination of binomial family with logit link). When you remove a true fixed predictor from a logistic model, the added variance due to removing the predictor cannot be picked up in the fixed-variance error term and will change the coefficients of all the remaining predictors. This 2010 paper by Carina Mood illustrates the problem. This freely available working paper by Maarten Buis presents a less pessimistic view. But even he concludes: "It is indeed problematic to compare coefficients across models with different sets of explanatory variables, since effects on chances are supposed to change when variables are added to the model even if they are uncorrelated with the other explanatory variables."

So for any comparisons you need to include both $P_1$ and $P_2$ in modeling both $B_1$ and $B_2$.

Then you have the problem of two binomial response variables. In the extreme, what if the probability of success for one response is 1% while that for the other response is 50%? There's not much unexplained variance for the first response even without predictors, but there's a lot for the other.

The best I can figure out to provide something close to what you want would be to develop a true bivariate model (i.e., with both response types and their correlations taken into account) with both your fixed effects as predictors and with appropriate handling of random effects. Then you have a single data set and rigorous model within which you might be able to evaluate whether $P_1$ or $P_2$ is a "more important" predictor for the model as a whole. If your a priori sense that $P_1$ is unrelated to $B_2$ (and vice-versa for the other predictor-response pair) is correct, then the result of that comparison within the combined model would seem to provide a test of your desired hypothesis. You will also be able to test your a priori sense about the cross-effects of the predictors on the responses.