1
$\begingroup$

I am running a logistic model for catastrophic health expenditure (CHE) in Argentina. The sample size is 22500. I followed Xu et al. methodology to define CHE and adjusted for 8 socioeconomic variables. The are all significant. Then I assessed the goodness of fit by looking at the H-L statistic (p-value>0.05) and the test indicates that the model fits the data well. Then, to assess the discrimination of the fitted model I estimated the area under the ROC curve, which is 78.1%. So far the model looks good. Unfortunately when looking at predicted probabilities and residuals I realized that the predicted probabilities are very low. They range between .0001645 and .3187172. Therefore I have very large residuals and leverage. Of course this also translates into a large number of influential observations.

Any suggestions why this could be happening? I have some ideas but I am not sure they are the right explanation: Is it possible that the proportion of households with CHE==1 is very low (3%) in the sample and this could be affecting the results?

Can I still use this model as a descriptive model of CHE in Argentina?

Thanks you! Mercedes

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

An unbalanced data set (like you suggest when you say the proportion of households with CHE = 1 make up 3% of your data) will influence the results of a logistic regression. See the answers to this question for instance.

In short, the class imbalance in your data set affects only the intercept term. If you are using your model to say something about you4 8 socioeconomic variables, you should be fine. If it is for prediction, then you could consider lowering your probability threshold below 0.5.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you, this was very helpful! I have some further questions. (1) Could you explain me please how lowering the probability threshold works? how do I do this in STATA? (2) The aim of my analysis was to explore what socioeconomic determinants were associated with CHE. If I am not wrong, according to your post and Paul Allison (statisticalhorizons.com/logistic-regression-for-rare-events) I should be ok. Is this right? (3) are you familiar with the Firth method (penalized MLE)? $\endgroup$
    – Mercedes
    Jun 29, 2015 at 14:45
1
$\begingroup$

My sense is that your model has significant variables mostly due to the large sample size. However the fit statistics are also quite good. When you say that the predicted probabilities are low, are they on the order of the 3% incidence of CHE?

It would seem that the model could be used, but perhaps other methods could be used as well and could allow more complete use of the available data. Have you considered using a Poisson or negative binomial distribution to fit the model? The former would allow you to analyze CHE rates/household if your data support events/unit time.

Given the low event rate, you would need to employ some caution in interpreting the results, as what is significant may not be important. Ultimately, the utility of your model will depend on your question.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Than you!! After some reading I found that one solution to this problem would be to use the Firth method (penalized MLE), see statisticalhorizons.com/logistic-regression-for-rare-events. I am doing my analysis in stata. I am not very familiar with the procedure, but the results changed. The estimated incidence is 2.47%. The objective of my model was to explore the determinants of CHE, not to predict. Do think that if I draw conclusions with caution I will be ok? Any further recommendations? $\endgroup$
    – Mercedes
    Jun 29, 2015 at 14:37
  • $\begingroup$ I think the best thing to do is to explore your data in detail and show yourself that your model makes sense. Group your socioeconomic variables into sensible ranges and make tables against CHE, plot the variables versus CHE, look for correlations between the socioeconomic variables. Augment your exploration of the data with the diagnostics you describe (H-L statistics, ROC, penalized MLE vs MLE, etc.) $\endgroup$
    – bill_e
    Jun 29, 2015 at 20:52
  • $\begingroup$ @Mercedes If the answer was useful to you, please upvote it! $\endgroup$
    – Matthew Drury
    Aug 27, 2016 at 20:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.