6
$\begingroup$

This question is related to one on Stack Overflow where the user wanted to obtain predictions and their standard errors using the "conditional" averaged coefficients from the R package MuMIn. I answered by demonstrating that if one uses vcov(model, full = FALSE) for an averaging object, the result may not even be positive definite.

I further commented briefly that I don't think the conditional averages even make sense. That's really a topic outside of the SO guidelines, so I thought I'd pursue that more on CV.

Briefly, in model averaging, the "full" averages are obtained by assigning values of zero as the regression coefficients for variables that are not included in a given model, and the "conditional" averages are averages only over the models where each variable is included.

My thinking is that there is no conceptual difference between (1) excluding a variable from a model and (2) constraining its regression coefficient to be zero. Thus, the "full" average is exactly the average of the regression coefficients of all the models considered. Accordingly, the "conditional" averages are a distortion.

Suppose, for example, we have a dozen are so models to be averaged, and only one of them, model #5, includes the predictor $x_{17}$; suppose that the coefficient of $x_{17}$ in model #5 is 2.50, and that model #5 has weight 0.01 in the model averaging. Then the conditional average for the coefficient of $x_{17}$ is 2.50, and the full average is 0.0250. Would one find it reasonable to make a prediction from the averaged model where $x_{17}$'s coefficient is 2.50? I don't think that makes sense, especially considering that $x_{17}$ doesn't even appear in 99% of the models. In general, following the logic of this example, the variables that appear the least are the ones that get the most distorted influence with the conditionally averaged coefficients.

So, my question is this: are the conditional averages of coefficients ever useful? And if so, why and in what context?

Note: A closely related question is this, which unfortunately has received no answers.

Addendum

The user who posted the related SO question requests that I add this question:

What happens if you have two models with similar weights (thus about 0.5 each), and one of the variables that appears in one of them has a coefficient of 90? Using the full average and by that reducing its coefficient to 45 might as problematic as using the conditional average in your example.

My comment would be to consider an analogy: if I have a certain amount of money to invest, and I am told that I'll earn \$90 if I invest it in a certain fund, I should expect to earn only \$45 if I invest only half of my money in that fund. Presumably, I'd earn some other amount by investing the other half in some other fund, so that my total earnings will probably be more than \$45.

$\endgroup$

2 Answers 2

3
$\begingroup$

Suppose you have a treatment $X$ with unknown effect $\beta$ that could quite likely be (approximately) zero. For example, someone's currently doing a small trial of tannic acid to reduce viral load in Covid.

I think you could reasonably ask, given the results of a trial:

  • how likely is it that there's a (non-negligible) effect? ​
  • if there is a (non-negligible) effect, how big is it likely to be?

The second question is the average conditional coefficient. It's unlikely to be the optimal summary -- you might really want to know the probability that the effect is greater than some 'clinically meaningful' threshold given that it's non-negligible -- but I don't think it's meaningless. If, for comparison, you had a full Bayesian model and a bimodal posterior distribution with one mode at zero, you couldn't really say it was universally meaningless to ask where the other mode was located.

That said, I'm not arguing it should be a default summary of the output. In particular, if there are multiple variables in the model, it probably won't be a meaningful summary for all of them even if it is for one of them.

$\endgroup$
3
$\begingroup$

One thing to be really careful about is when the Xs are categorical predictors - then the reference group (intercept) is different from model to model. Interaction terms would make it even more complicated.

So it might make more sense to average coefficients from two models with a shared reference group than to average over two models with different reference groups.

$\endgroup$
3
  • $\begingroup$ Good and interesting point. Interestingly, from the standpoint of prediction, which is my main concern (i.e. the emmeans package in R), the linearity of everything implies that predictions from the full averaged model are the same as the average of the separate models' predictions, even though the averaged coefficients are not interpretable. $\endgroup$
    – Russ Lenth
    Commented Jan 14, 2022 at 18:02
  • $\begingroup$ Also, if you center your variables, maybe it's less of a concern? $\endgroup$
    – Neal Fultz
    Commented Jan 19, 2022 at 19:04
  • $\begingroup$ Well, an additive model, centering has no effect at all on the coefficients, except for the intercept. With interactions, things are more complicated. But all in all, I am not convinced that centering would provide a clear advantage (or disadvantage) (except of course for numerical stability). $\endgroup$
    – Russ Lenth
    Commented Jan 19, 2022 at 19:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.