Assessing which variable is more predictive in multilevel modeling. Question about seemingly conflicting results

Question

I am analyzing data for a psychological study using multilevel models. I have a dependent variable (DV) and two predictors (A & B). DV, A, and B are all linear. I want to assess whether A or B is the better predictor of DV (30,000 observations & 1,000 participants). A and B are themselves strongly correlated (r = .70).

When I fit a multilevel regression including both A and B as predictors, A is significant (p < .001) but B is not (p > .5). Suppose that the coefficient for A is also significantly larger than the coefficient for B.

DV ~ 1 + A + B + (1 + A + B | participant)

However, when I fit two models, one using only A as a predictor and one using only B, the latter model achieves a much better fit, even when both models are fit on the exact same data (log-likelihood B >> A; I am not mixing up positive and negative log-likelihoods, to be clear):

DV ~ 1 + A + (1 + A | participant)
DV ~ 1 + B + (1 + B | participant)

Per a comment below, when I run this analysis with an interaction term (after mean centering A and B around zero), I find that A is significant, the A*B interaction is significant, but B is insignificant. The model with an interaction term achieves the best fit and best AIC. I know what an interaction is, but I'm not sure what this says about my question about A vs. B.

DV ~ 1 + A + B + A*B + (1 + A + B + A*B | participant)

So, which is the "better predictor"? I suspect that I may be confused about what "better predictor" means, but in general, how should these types of results be interpreted?

If we want this to be more concrete, suppose this is a reaction time experiment where participants are instructed to press a button upon cue. DV corresponds to their reaction time on a given trial. The cue is given both via sound and via visual indicator. Each cue varies in intensity, where A represents the intensity of the sound cue and B represents the intensity of the visual cue. Each participant did many trials of this study with varying A and B. I am interested in assessing whether sound intensity or visual intensity has a "larger bearing" on reaction time. Which is more "meaningfully" related to reaction time?

I have encountered this several times and it has always stumped me. I would appreciate any insight into how such results should be interpreted. I don't think any conflict like this would be possible in a classic (non-multilevel) regression?

Answer 1

With highly correlated values of A and B it will be difficult to determine which is the "better predictor." The model with the interaction is the "best predictor" by all of your criteria. The significant interaction means that the association of A with outcome depends on the level of B, and vice versa. Thus B is still an important predictor of outcome that should not be discarded.

The "insignificant" coefficient for B in that model only means that the slope of the outcome/B association isn't different from zero when A is at its original mean value (as you centered both A and B at their mean values). The significant interaction term means that the slope of that association will probably differ from 0 when A is at values away from its mean.

Also potentially at play here is whether the association between outcome and A or B is strictly linear. If a true association is curvilinear then your strictly linear handling of A and B (and their interaction) might be leading you astray. You have more than enough data to contemplate more flexible fitting of A and B with regression splines or a generalized additive model. Plots of outcome versus A and B at different levels of the other could be very informative.

Otherwise, I would worry that some quirk in the way that A and B were varied together in the study might be accounting for your apparent conflict in results.