Using Duan Smear factor on a two-part model

Question

I'm running a two-part model on a health insurance claims dataset where I predict the probability of nonzero health care costs using a logistic regression (1st part), then predict the magnitude of the log health care costs using an OLS regression (2nd part).

Question: When calculating the expected health care costs = (prob. nonzero claim)x(expected claim amount)x(Duan Smear factor) what is the value for the expected claim amount for the zero $ claims?

For example in the following table:

    member  observed cost  prob. of nonzero cost  expected cost  predicted cost                         
    1       0              .3                     ?              (.3) x (?) x (ds factor)      
    2       0              .2                     ?              (.2) x (?) x (ds factor)      
    3       1,000          .6                     900            (.6) x (900) x (ds factor)  
    4       5,000          .7                     7,000          (.7) x (7,000) x (ds factor)   

what is entered into the expected cost column for members 1 and 2 if the OLS regression is only applied to the nonzero cost data? Do we score the zero cost claims using the OLS regression parameters to calculate expected cost?

Answer 1

The OLS model is the model of expected cost given that there is a non-zero cost. Therefore by conditionality principal you simply don't use the data that has zero observed cost when fitting that part of the model.

Reading between the lines I'm guessing your real issue is how to calculate the Duan factors. The duan factors are usually calculated by using the residuals and so without getting the residuals how do you calculate the duan factors?

The key here is to look carefully at the model and think about what the Duan factors are there for.

You want to know the expected loss $E[L]$. This is given by the expected loss given non-zero loss times the probability of loss, $E[L] = E[L|L>0]P(L>0)$. The Duan factors come in because you are not fitting your model to $E[L|L>0]$, but rather to $E[\log(L)|L>0]$. The Duan Smearing attempts to correct for this bias using the residuals of the fit. It should now be clear that the correct thing to do here is to only use the residuals from the OLS fit when calculating the DS factor.

Effectively treat this as two separate bits of analysis. First discard all the zero losses, then with the data that remains fit your OLS model and calculate the DS factor. The put your dataset back together and fit the logistic model.

Answer 2

I have not seen the Duan smearing correction used in this way, but at first blush, it seems sensible. Here's the intuition.

The general re-transformation problem we have is that we want to get \begin{equation}E[y_i \vert x_i]=\exp (x_i'\beta) \cdot E[\exp (u_i)].\end{equation} For the first term, you can use the exponentiated prediction from the logged model. The second is more tricky. If we assume normality and independence, we can approximate the second term with $\exp (\frac{\hat \sigma^2}{2}),$ where we use the RMSE from the logged regression for the unobserved $\sigma$. Or we can use a weaker assumption of $iid$ on $u_i$, and use the sample average of the exponentiated residuals from the logged model for the second term.* That's the Duan "smearing" approach.

If you have access to Cameron and Trivedi's Microeconometrics Using Stata, take a look at chapter 16, in general, and table 16.2, in particular. There they give expressions for the conditional and unconditional means of expenditure when the outcome is in logs.

However, they are using a probit for the first stage, and making assumptions about joint normality and homoskedasticity of the errors in the two stages, so it's a slightly different model than the weaker $iid $ assumptions of the Duan approach with the logit. But the intuition carries over.

Combining the intuition from the first paragraph and their table, we have that \begin{equation}E[y_2 \mid x,y_2>0 ] = \exp (x_2' \beta) \cdot \exp( \frac{ \sigma^2_2}{2} )\end{equation} and \begin{equation}E[y_2 \mid x ] = \exp (x_2' \beta) \cdot \exp(\frac{ \sigma^2_2}{2} ) \cdot \Phi(x_1' \beta).\end{equation} The $\sigma_2$ is the variance in the logged expenditure equation, which you can approximate as above. The subscripts indicate the stage (since you can have different predictors). The $\Phi$ is just the probability of going to see the doc from the probit first stage, which is like your logit.

Thus, it seems that as long as you maintain the assumption that the two stages are independent, all you need to do is multiply by $\frac{1}{N}\sum_i^N \exp (\hat u_i)$ if you want the expected positive expenditure. You will also need to create out of sample predictions for the expenditure for those who did not get to see the doc.

Addendum:

Stata now has the a user-written tpmcommand that deals really nicely with this problem.

---

* If you're worried about heteroskedasticity, you might try smearing by group. For example, you might try an age-group specific average residual.