I am analyzing data for a psychological study using multilevel models. I have a dependent variable (DV) and two predictors (A & B). DV, A, and B are all linear. I want to assess whether A or B is the better predictor of DV (30,000 observations & 1,000 participants). A and B are themselves strongly correlated (r = .70).
When I fit a multilevel regression including both A and B as predictors, A is significant (p < .001) but B is not (p > .5). Suppose that the coefficient for A is also significantly larger than the coefficient for B.
DV ~ 1 + A + B + (1 + A + B | participant)
However, when I fit two models, one using only A as a predictor and one using only B, the latter model achieves a much better fit, even when both models are fit on the exact same data (log-likelihood B >> A; I am not mixing up positive and negative log-likelihoods, to be clear):
DV ~ 1 + A + (1 + A | participant) DV ~ 1 + B + (1 + B | participant)
Per a comment below, when I run this analysis with an interaction term (after mean centering A and B around zero), I find that A is significant, the A*B interaction is significant, but B is insignificant. The model with an interaction term achieves the best fit and best AIC. I know what an interaction is, but I'm not sure what this says about my question about A vs. B.
DV ~ 1 + A + B + A*B + (1 + A + B + A*B | participant)
So, which is the "better predictor"? I suspect that I may be confused about what "better predictor" means, but in general, how should these types of results be interpreted?
If we want this to be more concrete, suppose this is a reaction time experiment where participants are instructed to press a button upon cue. DV corresponds to their reaction time on a given trial. The cue is given both via sound and via visual indicator. Each cue varies in intensity, where A represents the intensity of the sound cue and B represents the intensity of the visual cue. Each participant did many trials of this study with varying A and B. I am interested in assessing whether sound intensity or visual intensity has a "larger bearing" on reaction time. Which is more "meaningfully" related to reaction time?
I have encountered this several times and it has always stumped me. I would appreciate any insight into how such results should be interpreted. I don't think any conflict like this would be possible in a classic (non-multilevel) regression?