# Estimating classification probability, with low event rates -- options other than logistic regression?

I am trying to predict the probability of occurrence of a low event rate outcome (~2% readmission risk after hospital discharge in the population of interest).

With the available limited predictors, I am not expecting accurate classification -- I would be satisfied with a prediction of the probability of readmission (e.g., if the predicted risk was 6%, simply knowing that there was a 3-fold higher risk of readmission would be helpful). There is no specific "threshold" risk that would be meaningful, so approaches of weighting / using ROC curves for sensitivity/specificity tradeoffs would not be helpful.

Logistic regression was my obvious choice, but I am wondering if there are other classifiers that are well-suited to reporting percent probabilities of classification outcome.

There seem to be multiple related questions re: class imbalance, but most seem to focus on feature selection, and here, weighting, performance metrics, and still focus on pure "classification" as the output, rather than a continuous rather than discrete output of relative / percent risk.

My preference would be solutions that are implemented in R.

Apologies for such a basic question -- I am a clinician with no formal training in data science.

======

Edited to add: maybe some more searching would have been wise. For Random Forests, is this giving me what I'm asking for?

predict(model_rf, test_cases, type = "prob")


Do other classification models typically offer this option for output?

• Is there some reason why you do not want to use logistic regression? Do you care about time to readmission, or just, say, readmission within 90 days of discharge? If the latter, do you have complete follow-up for the time period of interest?
– EdM
Oct 23 '15 at 13:11
• Logistic regression may be fine for now, but there will soon be a huge increase in predictor availability (due to electronic health record changes), at which point overfitting may be a major concern. So, I'm looking for other options that might not be so sensitive to overfitting, or would allow regularization. For now, looking at readmission within 30 days (reportable metric), and no, there is not "complete" follow-up, but the fraction of unrecognized cases should not be changing. Oct 23 '15 at 13:30
• your reasons for rejecting logistic regression are incorrect. you can use regularisation with logistic regression.. see eg glmnet R package. I would also go along with a survival approach to maximise the use of your data ( ie effectively combining 10day/20 day/30 day/60 day etc metrics) Oct 24 '15 at 15:28

Logistic regression seems suited to your needs. It gives the continuous measure of predicted probability of readmission that you want, and it is compatible with methods to deal with overfitting when that becomes an issue (as it already may be with a low-probability event like this).

The lrm function in the R ols package might be a wise choice to start, as that package has associated tools for validation and calibration. The author of the package, Frank Harrell, maintains a useful wiki on regression modeling, is author of the recently released second edition of "Regression Modeling Strategies" (links available from that wiki site), and is a frequent contributor to Cross Validated.

The main limitation, as you seem to understand, is the low percentage of events (readmissions). For every predictor variable you want to evaluate you need on the order of 10-20 events. That limitation, however, holds for any regression or classification scheme.

There are several ways to deal with these issues; this page includes many solutions available in R, and the relogit function in the R zelig package was designed to handle this situation. The penalized-likelihood estimation already provided by the lrm package might be adequate, although this is getting a bit beyond my practical experience. This recent thesis compares several glm approaches (including asymmetric link functions), SVM, and Random Forest on 4 test data sets. Some competent local statistical support thus would be important for your project; you might find it useful (or at least amusing) to "please read this comment".