1
$\begingroup$

I want to calculate the relative importance of predictors of a hurdle model, my first choice is dominance analysis. For that I would need a suitable metric of model quality. My first thought is to use a pseudo-R2 like McFadden, but I am not an expert.

Could anyone provide advice on a suitable metric of model quality, to assess relative importance of predictors, in a hurdle model?

Note: I am an R user, and would fit the hurdle model with pscl::hurdle(). For dominance analysis, I would use domir::domir().

Thank you!

EDIT (12 september 2023)

If I understand correctly, performance::r2_kullback() does not calculate the R2 endorsed by Cameron&Windmeijer (1996, eq. 1.15), based on a comparison with a toy Poisson model.

library(performance)
#> Warning: package 'performance' was built under R version 4.2.3
library(tidyverse)
#> Warning: package 'ggplot2' was built under R version 4.2.2
#> Warning: package 'tibble' was built under R version 4.2.3
#> Warning: package 'tidyr' was built under R version 4.2.3
#> Warning: package 'purrr' was built under R version 4.2.3
#> Warning: package 'dplyr' was built under R version 4.2.3
#> Warning: package 'stringr' was built under R version 4.2.3

# Toy model with forced 0s in the response
df <- mtcars
df$carb <- ifelse(df$carb == 4, 0, df$carb)
model <- glm(carb ~ wt + mpg, data = df, family = "poisson")
summary(model)
#> 
#> Call:
#> glm(formula = carb ~ wt + mpg, family = "poisson", data = df)
#> 
#> Deviance Residuals: 
#>     Min       1Q   Median       3Q      Max  
#> -2.0524  -1.3685  -0.1410   0.4457   3.3902  
#> 
#> Coefficients:
#>             Estimate Std. Error z value Pr(>|z|)  
#> (Intercept)  3.57695    2.02825   1.764   0.0778 .
#> wt          -0.54005    0.33681  -1.603   0.1088  
#> mpg         -0.07089    0.05188  -1.367   0.1718  
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> (Dispersion parameter for poisson family taken to be 1)
#> 
#>     Null deviance: 57.644  on 31  degrees of freedom
#> Residual deviance: 54.810  on 29  degrees of freedom
#> AIC: 117.52
#> 
#> Number of Fisher Scoring iterations: 5

# Custom function to calculate R2 based on deviance residuals, from Cameron&Vermeijer (1996) ep.1.15, p.211
r2.kull <- function(mod){
  num <- sum(mod$y*log(if_else(mod$y == 0, true = 1, false = mod$y)/fitted(mod)) - (mod$y - fitted(mod)))
  den <- sum(mod$y*log(if_else(mod$y == 0, true = 1, false = mod$y) / mean(mod$y)))
  res <- 1 - (num/den)
  return(res)
}

# Both do not yield the same result.
performance::r2_kullback(model, adjust = TRUE)
#> Kullback-Leibler R2 
#>         -0.01640953


r2.kull(model)*(model$df.null/model$df.residual)
#> [1] 0.05255599

Created on 2023-09-12 with reprex v2.0.2

Indeed, performance::r2_kullback() won't work with models fitted by pscl::hurdle(). r2_kullback() relies on deviance, but models fit with pscl::hurdle() don't provide a deviance by design. This could be overcome by calculating the deviance manually (I think), but, in the face of approaching deadlines, I find it less time-consuming to use performance::r2_zeroinflated() for now. Incidentally, there was some discussion on which R2 measure to implement in performance.

$\endgroup$
1

1 Answer 1

1
$\begingroup$

It is worth noting that the pscl package does include the pr2() function that computes several pseudo-R2 metrics that could be applied to pscl::hurdle().

The r2_zeroinflated() function in the performance can be dominance analyzed as well though guidance from authors such as Cameron and Windmeijer (1996) would suggest the use of the r2_kullback() metric in performance (though I do not believe there is a method that applies to hurdle for that function).

As is noted by Cameron and Windmeijer, the McFadden $R^2$ is a linear transformation of the Kullback metric which does suggest it might be the most interpretable metric (as a form of the percent of model information explained by the predictors).

As a related aside, it is also worthwhile to consider whether the hurdle model estimated should be evaluated in terms of independent variable relative importance or parameter estimate relative importance (see Luchman, Lei, & Kaplan, 2020) which can be implemented by the formula_list method of domir() as of version 1.1).

References

Cameron, A. C., & Windmeijer, F. A. (1996). R-squared measures for count data regression models with applications to health-care utilization. Journal of Business & Economic Statistics, 14(2), 209-220.

Luchman, J. N., Lei, X., & Kaplan, S. A. (2020). Relative importance analysis with multivariate models: Shifting the focus from independent variables to parameter estimates. Journal of Applied Structural Equation Modeling, 4(2), 1-20.

$\endgroup$
1
  • $\begingroup$ Thank you for the thoughtful reply! Long story short, I think I'll use the performance::r2_zeroinflated(), mainly due to time constraints. I've edited my question to show a comparison between the R2 endorsed by Cameron&Windmeijier (eq.1.15, based on deviance residuals) and r2_kullback(): they appear to differ. As for the parameter estimate relative importance, I'll look into it, wasn't aware of it till now! $\endgroup$
    – M. Riera
    Sep 12, 2023 at 17:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.