I am building xgboost models for prediction of insurance risk, the risk being assumed to follow a tweedie distribution with tweedie variance power between 1 and 2 (https://en.wikipedia.org/wiki/Compound_Poisson_distribution). The data I am using is heavily imbalanced with almost 98% 0s. I have approached this in two ways:

  1. Just fitting a tweedie model directly to the data
  2. Using a hurdle model, i.e., first modelling whether the response is zero or non-zero and then fitting a tweedie model to the the non-zero response

I am using XGboost in R for the tweedie models and logistic regression for the binary classification (zero or non-zero). However, I am unsure how I best compare the models and evaluate which one is best. I have used deviance as a metric for the direct model, but the problem is that the hurdle model consists of two models and I do not know how to combine the performance of the two models into one metric that can be compared with the direct model. Note, that I am aware that I can use metrics such as MSE or MAE, however as my data is so imbalanced I do not believe these metrics are that useful.

So, to my question: What metric can I use so that I can compare the hurdle model and the direct model?

Thanks in advance!



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.