6
$\begingroup$

Qualitatively, I want to answer the question:

Given that the effect of a covariate is statistically significant (as assessed by the p-value / confidence interval of its coefficient), how important is it in explaining the response? A little bit? A lot?"

We can compare the magnitude of coefficients for continuous variables but doing the same for categorical variables with $n > 2$ values (the motivation for my question) is not so straightforward since there are $n-1 \geq 2$ parameters.

$R^2$, in the case of an identity link, and pseudo-$R^2$ in the case of a logit link, quantifies how much variation is explained by a model. So one way to answer the question would be to compute the differences in $R^2$ between models that include vs. exclude a particular covariate. These differences could then, at least in the case of an identity link, be interpreted as the differences in "how much" (of the variability) of the response is explained by a covariate. How to interpret the result is not as straightforward in the case of pseudo-$R^2$.

My questions are:

  1. Is the method of comparing $R^2$ between models that include / exclude covariates one that is used in practice?
  2. Are there any caveats to consider with this method of comparing $R^2$?
  3. Are there other well-known methods that quantify the "contribution" of a categorical covariate that can take on more than 2 values?
$\endgroup$

1 Answer 1

3
$\begingroup$

People do compare $R^2$ values, but for some GLMs there are approximations to $R^2$, but they are of limited value. A little more common is to compare deviance values (these are at least well defined). But comparing $R^2$ or deviance is more about statistical significance than the real importance of a variable.

I prefer to look at predictions, set all the variables that you are not currently examining to their mean, median, most common category, or other meaningful value, then set the predictor variable of interest to some meaningful values and make predictions from these values, then compare the predictions.

Another option for looking at importance is a nomogram. This stack overflow post (https://stackoverflow.com/questions/38276973/r-how-to-read-nomograms-to-predict-the-desired-variable) shows and example of a nomogram (and the code used to create it) and the answer shows how to interpret it. But this lets us see the relative importance of the different variables at a glance. Look at the line for Blood Pressure, you can see that a big change in blood pressure will only change the total points by a very small amount (and therefore the final prediction will change very little), but a modest change in Age will have a much larger effect. If there were categorical variables in that nomogram (other than the one involved in an interaction) then there would be a line with a tick mark for each category and you could see how far apart the ticks are for relative importance.

$\endgroup$
3
  • 2
    $\begingroup$ Frank Harrell's textbook illustrates such partial-effect plots and a nomogram for a logistic multiple regression in Chapter 11. He also illustrates a variable-importance estimate based on Wald-test statistics, with Wald chi-square values corrected for the associated degrees of freedom. As Wald tests can be done on collections of individual regression coefficients, that nicely provides a combined estimate for multi-level categorical predictors, as requested by the OP. (+1) $\endgroup$
    – EdM
    Dec 22, 2021 at 20:37
  • $\begingroup$ Thank you both for the suggestions! I'll have to spend more time to understand nomograms but, if I understand correctly from my brief review, they are more well-suited for understanding the effects of unit changes of covariates on categorical outcomes / risk. @Greg Snow you mention that "But comparing $R^2$...is more about statistical significance", but I am having trouble reconciling this with the fact this statistic literally provides the percentage of variance explained by a set of covariates, which (at least in my opinion) coincides nicely with the notion of contribution. $\endgroup$
    – cdlm
    Dec 23, 2021 at 14:57
  • $\begingroup$ @cdlm, It looks like I may have interpreted "contribution" differently than you meant. This can get really complicated if your covariates are not independent of each other. While $R^2$ has a nice definition, it does not always translate to meaningful interpretation. Try this: simulate 100 x,y pairs of data from a bivariate normal such that $R^2$ is between 0.7 and 0.9, run the regression, plots, etc. Now remove the points where the x value is between the first and third quartiles (middle half) and rerun the regression, what happened to $R^2$? is this really an improvment? $\endgroup$
    – Greg Snow
    Dec 23, 2021 at 21:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.