0
$\begingroup$

I am analyzing data for a psychological study using multilevel models. I have a dependent variable (DV) and two predictors (A & B). DV, A, and B are all linear. I want to assess whether A or B is the better predictor of DV (30,000 observations & 1,000 participants). A and B are themselves strongly correlated (r = .70).

When I fit a multilevel regression including both A and B as predictors, A is significant (p < .001) but B is not (p > .5). Suppose that the coefficient for A is also significantly larger than the coefficient for B.

DV ~ 1 + A + B + (1 + A + B | participant)

However, when I fit two models, one using only A as a predictor and one using only B, the latter model achieves a much better fit, even when both models are fit on the exact same data (log-likelihood B >> A; I am not mixing up positive and negative log-likelihoods, to be clear):

DV ~ 1 + A + (1 + A | participant)
DV ~ 1 + B + (1 + B | participant)

Per a comment below, when I run this analysis with an interaction term (after mean centering A and B around zero), I find that A is significant, the A*B interaction is significant, but B is insignificant. The model with an interaction term achieves the best fit and best AIC. I know what an interaction is, but I'm not sure what this says about my question about A vs. B.

DV ~ 1 + A + B + A*B + (1 + A + B + A*B | participant)

So, which is the "better predictor"? I suspect that I may be confused about what "better predictor" means, but in general, how should these types of results be interpreted?

If we want this to be more concrete, suppose this is a reaction time experiment where participants are instructed to press a button upon cue. DV corresponds to their reaction time on a given trial. The cue is given both via sound and via visual indicator. Each cue varies in intensity, where A represents the intensity of the sound cue and B represents the intensity of the visual cue. Each participant did many trials of this study with varying A and B. I am interested in assessing whether sound intensity or visual intensity has a "larger bearing" on reaction time. Which is more "meaningfully" related to reaction time?

I have encountered this several times and it has always stumped me. I would appreciate any insight into how such results should be interpreted. I don't think any conflict like this would be possible in a classic (non-multilevel) regression?

$\endgroup$
2
  • $\begingroup$ Does "group" represent the individuals here? I think that's what you imply, but it's not completely clear. Were A and B treated as strictly linear predictors in the model? Was there any pattern to the changes in A versus B levels as they were varied? Did you evaluate interactions of A with B? How many observations, individuals, and groups were involved? Please provide that information by editing the question, as comments are easy to overlook and can be deleted. The more details you can provide, the more likely that you will get a helpful answer. $\endgroup$
    – EdM
    Jul 10, 2022 at 18:28
  • $\begingroup$ Thanks, I updated my original post per your suggestions. Yes, "group" meant individuals/participants. A and B are linear predictors of DV, which is also linear. A and B are strongly correlated (r = .7). When I include an interaction term in the multi-predictor regression, it is significant (B remains insignificant). There were 30,000 observations and 1,000 participants. $\endgroup$ Jul 10, 2022 at 18:42

1 Answer 1

0
$\begingroup$

With highly correlated values of A and B it will be difficult to determine which is the "better predictor." The model with the interaction is the "best predictor" by all of your criteria. The significant interaction means that the association of A with outcome depends on the level of B, and vice versa. Thus B is still an important predictor of outcome that should not be discarded.

The "insignificant" coefficient for B in that model only means that the slope of the outcome/B association isn't different from zero when A is at its original mean value (as you centered both A and B at their mean values). The significant interaction term means that the slope of that association will probably differ from 0 when A is at values away from its mean.

Also potentially at play here is whether the association between outcome and A or B is strictly linear. If a true association is curvilinear then your strictly linear handling of A and B (and their interaction) might be leading you astray. You have more than enough data to contemplate more flexible fitting of A and B with regression splines or a generalized additive model. Plots of outcome versus A and B at different levels of the other could be very informative.

Otherwise, I would worry that some quirk in the way that A and B were varied together in the study might be accounting for your apparent conflict in results.

$\endgroup$
2
  • $\begingroup$ Thanks, this is helpful. I will poke around the data, based on your suggestions, and see what comes out. $\endgroup$ Jul 10, 2022 at 21:53
  • $\begingroup$ @user9154908 plots are often very informative. I'd include plots of B versus A in your evaluation, not just plots of outcome versus each. $\endgroup$
    – EdM
    Jul 10, 2022 at 22:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.