0
$\begingroup$

I have a data set consisting of two behavioral response variables where each response variable is associated with a relevant physiological predictor, with many trials for each subject.

The hypothesis I wish to test, which I believe is difficult if not impossible to test in this framework, is: Is ${P_1}$ a better predictor for ${B_1}$ than ${P_2}$ is for ${B_2}$

Independently, it makes sense to model each of these as a logistic regression something like

${B_1}$ ~ 1 + ${P_1}$ + (1|Subject)

${B_2}$ ~ 1 + ${P_2}$ + (1|Subject)

but no a priori reason to expect ${P_1}$ to be correlated with ${B_2}$ and vice-versa.

The response/predictors are related enough that it would be completely appropriate to model the difference between them with an additional fixed effect for "1" versus "2" and test the interaction.

However, I don't really hypothesize that the coefficients for ${P_1}$ and ${P_2}$ are necessarily different. Rather, I expect that ${P_1}$ is a better predictor for ${B_1}$ than ${P_2}$ is for ${B_2}$: I expect larger error in the second model than there is in the first.

This would be a trivial comparison if the response was the same, but it isn't. Is there a principled way to make a comparison like this in this context?

$\endgroup$
  • $\begingroup$ What you have is a "multivariate" model in the generally accepted sense, that is with multiple outcome/response variables each potentially related to a set of predictors. The possibility of correlations between $B_1$ and $B_2$ makes combined multivariate analysis preferable. This page provides one place to start; search for "multivariate generalized linear model," probably with "mixed" as part of the search. Note that if you have time-to-response data, a logistic regression might be throwing away useful information. $\endgroup$ – EdM Dec 27 '17 at 18:23
  • $\begingroup$ @EdM Thanks, I am aware that one can fit a multivariate model due to correlations between the response variables, but I don't see any mechanism for comparing the quality of fits of different predictors in that context. If anything, a multivariate model seems to get further from that goal. To your second point, response latency data are not relevant for this dataset. $\endgroup$ – Bryan Krause Dec 27 '17 at 18:32
  • $\begingroup$ Is your main hypothesis "that $P_1$ is a better predictor for $B_1$ than $P_2$ is for $B_2$"? That's not easy to test in regressions in general, and with logistic regression you don't really have "residual variance" to test. You might want to revise your question to elaborate a bit on the specific hypotheses you wish to test. $\endgroup$ – EdM Dec 27 '17 at 19:47
  • $\begingroup$ @EdM Yes, exactly - I know it's not easy which is partly why I'm asking here :) - rereading my question I see I did not make that very clear. I will revise. $\endgroup$ – Bryan Krause Dec 27 '17 at 19:53
  • $\begingroup$ In your experimental design: are both responses $B_1$ and $B_2$ measured on all subjects? Are both responses measured for all trials? Are $P_1$ and $P_2$ experimental manipulations or observed covariates? If the latter, do their values change from trial to trial? Are the predictors continuous or factors? $\endgroup$ – EdM Dec 28 '17 at 17:40
1
$\begingroup$

TL;DR:

A proper bivariate logistic regression with both $P_1$ and $P_2$ as predictors, taking into account correlations between response types and within subjects, might give you a structure within which you could do something close to what you want. Otherwise you have several types of large problems.

Details:

If you are trying to compare two different models on two different data sets, as a quick look at your question might suggest, then the following comment by @whuber on this question about comparing models applies: "If model A fits dataset 1 and model B fits dataset 2, there's nothing at all to compare: the models and the data are totally different." There are ways to use measures like residual variance, fraction of log-likelihood explained, or AIC to compare models (see for example answers in and links from the question linked above). You will find that some question whether some such comparisons are valid for non-nested models, but even those who claim such validity note that the models need to be on the same data.

Of course, you don't really have two independent data sets. You have multiple blocks of trials on the same subjects, with values of 2 continuous covariates for each subject in each block and 2 types of binary responses measured in different blocks. Both the covariates and the response types are evidently correlated. It would seem best to approach this problem with a single coherent model.

Even with a single response variable, determining which of two predictors is "better" can be fraught with difficulty. On this site, the tag feature-selection covers such issues. For example, When you leave out predictors that might be correlated both with included predictors and with outcome in a linear regression, the signs of regression coefficients for the included predictors can differ from what they would be in a full model. Follow the simpsons-paradox tag on this site for examples in several contexts.

Using logistic regressions adds another layer of complexity to comparing predictors: absence of a predictor can affect the regression coefficients even of predictors that are uncorrelated to it. The variance in logistic regression, in a true model, only includes variance from random effects plus an assumed fixed variance from the logistic distribution of the error term (or equivalently from the combination of binomial family with logit link). When you remove a true fixed predictor from a logistic model, the added variance due to removing the predictor cannot be picked up in the fixed-variance error term and will change the coefficients of all the remaining predictors. This 2010 paper by Carina Mood illustrates the problem. This freely available working paper by Maarten Buis presents a less pessimistic view. But even he concludes: "It is indeed problematic to compare coefficients across models with different sets of explanatory variables, since effects on chances are supposed to change when variables are added to the model even if they are uncorrelated with the other explanatory variables."

So for any comparisons you need to include both $P_1$ and $P_2$ in modeling both $B_1$ and $B_2$.

Then you have the problem of two binomial response variables. In the extreme, what if the probability of success for one response is 1% while that for the other response is 50%? There's not much unexplained variance for the first response even without predictors, but there's a lot for the other.

The best I can figure out to provide something close to what you want would be to develop a true bivariate model (i.e., with both response types and their correlations taken into account) with both your fixed effects as predictors and with appropriate handling of random effects. Then you have a single data set and rigorous model within which you might be able to evaluate whether $P_1$ or $P_2$ is a "more important" predictor for the model as a whole. If your a priori sense that $P_1$ is unrelated to $B_2$ (and vice-versa for the other predictor-response pair) is correct, then the result of that comparison within the combined model would seem to provide a test of your desired hypothesis. You will also be able to test your a priori sense about the cross-effects of the predictors on the responses.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.