12
$\begingroup$

To compute the model-averaged predictions on the response scale of a GLM, which is "correct" and why?

  1. Compute the model averaged prediction on the link scale and then back-transform to the response scale, or
  2. Back transform the predictions to the response scale and then compute the model average

The predictions are close but not equal if the model is a GLM. The different R packages give options for both (with different defaults). Several colleagues have argued vociferously that #1 is wrong because "everyone does #2". My intuition says that #1 is "correct" as it keeps all linear math linear (#2 averages things that are not on a linear scale). A simple simulation finds that #2 has a very (very!) slightly smaller MSE than #1. If #2 is correct, what is the reason? And, if #2 is correct, why is my reason (keep linear math linear) poor reasoning?

Edit 1: Computing marginal means over the levels of another factor in a GLM is a similar problem to the question that I am asking above. Russell Lenth computes marginal means of GLM models using the "timing" (his words) of #1 (in the emmeans package) and his argument is similar to my intuition.

Edit 2: I am using model-averaging to refer to the alternative to model selection where a prediction (or a coefficient) is estimated as the weighted average over all or a subset of "best" nested models (see references and R packages below).

Given $M$ nested models, where $\eta_i^m$ is the linear prediction (in the link space) for individual $i$ for model $m$, and $w_m$ is the weight for model $m$, the model-averaged prediction using #1 above (average on the link scale and then backtransform to the response scale) is:

$$\hat{Y}_i = g^{-1}\Big(\sum_{m=1}^M{w_m \eta_i^m}\Big)$$

and the model-averaged prediction using #2 above (back transform all $M$ predictions and then average on the response scale) is:

$$\hat{Y}_i = \sum_{m=1}^M{w_m g^{-1}(\eta_i^m})$$

Some Bayesian and Frequentist methods of model averaging are:

  • Hoeting, J.A., Madigan, D., Raftery, A.E. and Volinsky, C.T., 1999. Bayesian model averaging: a tutorial. Statistical science, pp.382-401.

  • Burnham, K.P. and Anderson, D.R., 2003. Model selection and multimodel inference: a practical information-theoretic approach. Springer Science & Business Media.

  • Hansen, B.E., 2007. Least squares model averaging. Econometrica, 75(4), pp.1175-1189.

  • Claeskens, G. and Hjort, N.L., 2008. Model selection and model averaging. Cambridge Books.

R packages include BMA, MuMIn, BAS, and AICcmodavg. (Note: this is not a question about the wisdom of model-averaging more generally.)

$\endgroup$
  • 1
    $\begingroup$ I suspect the reason your question is receiving no answers is that other readers, like me, don't understand your question. What you do you mean exactly by "model-averaging"? Please describe a context in detail so we understand what problem it is that you are trying to solve. As far as I can see, the emmeans package does not average predictions from different models. $\endgroup$ – Gordon Smyth Aug 19 '18 at 6:32
  • 1
    $\begingroup$ Thanks for asking this and I can see that adding the Russell Lenth note confuses my question. I tried to clarify this above. The emmeans package will compute marginal means and SE over the levels of another factor and these statistics are computed on the link scale and then back-transformed. See the section "The model is our best guide". $\endgroup$ – JWalker Aug 20 '18 at 12:03
  • $\begingroup$ I'd really be interested in any answers to this question. Meanwhile, a comment. That MSE result is computed on the back-transformed scale. I would bet that with the same simulation results, the MSE, when computed on the link scale, would be smaller with #1 than with #2. The reason is that the sample mean is the least-squares estimator of the population mean, even on the wrong scale. $\endgroup$ – Russ Lenth Aug 21 '18 at 13:53
6
$\begingroup$

The optimal way of combining estimators or predictors depends on the loss function that you are trying to minimize (or the utility function you are trying to maximize).

Generally speaking, if the loss function measures prediction errors on the response scale, then averaging predictors on the response scale correct. If, for example, you are seeking to minimize the expected squared error of prediction on the response scale, then the posterior mean predictor will be optimal and, depending on your model assumptions, that may be equivalent to averaging predictions on the response scale.

Note that averaging on the linear predictor scale can perform very poorly for discrete models. Suppose that you are using a logistic regression to predict the probability of a binary response variable. If any of the models give a estimated probability of zero, then the linear predictor for that model will be minus infinity. Taking the average of infinity with any number of finite values will still be infinite.

Have you consulted the references that you list? I am sure that Hoeting et al (1999) for example discuss loss functions, although perhaps not in much detail.

$\endgroup$
  • 1
    $\begingroup$ Excellent. Thanks for this response (I welcome others!). I assume that "then averaging predictors is likely to be optimal or close to it" is averaging predictors on the response scale. The logistic note is especially helpful. $\endgroup$ – JWalker Aug 22 '18 at 11:38
  • 1
    $\begingroup$ @rvl Regarding linearity of the loss function, I was thinking in terms of the loss's influence function. I agree that's a bit cryptic, so I have edited my comments. I have to disagree with your other remarks. GLMs are estimated by ML, not by squared error loss. Despite the name, the IRLS algorithm that is popular for GLMs does not minimize a sum of squares and the IRLS working variable involves standardized residuals on the response scale, not the link scale. In any case, estimation and prediction are not the same and not need to have the same loss functions. $\endgroup$ – Gordon Smyth Aug 31 '18 at 10:15
  • $\begingroup$ @rvl Exact zero fitted values occur frequently in logistic regression and have been discussed on this forum several times. $\endgroup$ – Gordon Smyth Aug 31 '18 at 10:16
  • $\begingroup$ @rvl The loss is not evaluated on the link scale. This discussion is not the right place for me to offer you a tutorial on GLMs -- I refer you instead to my book on GLMs which Springer will publish in about a month. Nor is this discussion the right place for you to offer an alternative answer to the original question. Write a proper answer if you want to do that. $\endgroup$ – Gordon Smyth Sep 1 '18 at 8:41
  • $\begingroup$ Here is the link to our book on GLMs: doi.org/10.1007/978-1-4419-0118-7 $\endgroup$ – Gordon Smyth Dec 10 '18 at 3:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.