2
$\begingroup$

I have an insurance claims data with claim amount as a target variable. The claim amount has a lot of 0s . The output is to predict the high-risk customers. i.e., to preserve the order of claim amounts. I have planned to convert the claims amount variable to an ordinal value ( highest amount =1, second highest amount = 2.......) and perform ordinal logistic regression. Please let me know if my approach is correct. Also please help with any algorithms that deal with thid kind of data. The evaluation of the model would be done on gini index. How do i handle the high amount of 0s in data.

$\endgroup$
1
  • $\begingroup$ What's the rationale behind converting to ordinal variables? Why can't you do a ordinary linear regression, and then rank the predicted outcomes? I have a feeling that will be much more efficient. If you notice a poor fit because of the zeroes you can consider migrating to a zero-inflated/hurdle model, in which you can also use your predictors to estimate the chance on structural zeroes. $\endgroup$
    – Knarpie
    Sep 8, 2017 at 10:34

2 Answers 2

1
$\begingroup$

I would not recommend making the variable ordinal.

Liu, Shih, Strawderman et al (2009), Analysis of Zero-Inflated Nonnegative Continuous Data: A Review. Statistical Science, 34 (2) 253-279 is an excellent source. They recommend a two part model, first modeling whether Y is 0 or not, then modeling the positive values:

$P(Y ≤ y|X) = 1−p(X) + I(y >0)p(X)F_e(log(y) - X^T\beta)$

where $F_e(u) = P(e ≤ u) \text{and } p(X) = expit(X^T α)$

They consider many variations and complications (which don't seem needed in your case) and have a github with SAS code to implement the models they recommend.

$\endgroup$
1
$\begingroup$

Take a look at zero-inflated lognormal (ziln) regression models.

See https://github.com/google/lifetime_value for an application and implementation in neural networks, that estimate the distribution of the Y=0 and positive part jointly, rather than through 2 separate models.

I have used this successfully on insurance claims data with excellent results. In particular it allows you to estimate / predict tail risk and expected shortfall of insurance claims accurately.

For reference see Wang, Xiaojing, Liu, Tianqi, and Miao, Jingang. (2019). A Deep Probabilistic Model for Customer Lifetime Value Prediction. arXiv:1912.07753.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.