I am building xgboost models for prediction of insurance risk, the risk being assumed to follow a tweedie distribution with tweedie variance power between 1 and 2 (https://en.wikipedia.org/wiki/Compound_Poisson_distribution). The data I am using is heavily imbalanced with almost 98% 0s. I have approached this in two ways:
- Just fitting a tweedie model directly to the data
- Using a hurdle model, i.e., first modelling whether the response is zero or non-zero and then fitting a tweedie model to the the non-zero response
I am using XGboost in R for the tweedie models and logistic regression for the binary classification (zero or non-zero). However, I am unsure how I best compare the models and evaluate which one is best. I have used deviance as a metric for the direct model, but the problem is that the hurdle model consists of two models and I do not know how to combine the performance of the two models into one metric that can be compared with the direct model. Note, that I am aware that I can use metrics such as MSE or MAE, however as my data is so imbalanced I do not believe these metrics are that useful.
So, to my question: What metric can I use so that I can compare the hurdle model and the direct model?
Thanks in advance!