Estimating classification probability, with low event rates -- options other than logistic regression?

Question

I am trying to predict the probability of occurrence of a low event rate outcome (~2% readmission risk after hospital discharge in the population of interest).

With the available limited predictors, I am not expecting accurate classification -- I would be satisfied with a prediction of the probability of readmission (e.g., if the predicted risk was 6%, simply knowing that there was a 3-fold higher risk of readmission would be helpful). There is no specific "threshold" risk that would be meaningful, so approaches of weighting / using ROC curves for sensitivity/specificity tradeoffs would not be helpful.

Logistic regression was my obvious choice, but I am wondering if there are other classifiers that are well-suited to reporting percent probabilities of classification outcome.

There seem to be multiple related questions re: class imbalance, but most seem to focus on feature selection, and here, weighting, performance metrics, and still focus on pure "classification" as the output, rather than a continuous rather than discrete output of relative / percent risk.

My preference would be solutions that are implemented in R.

Apologies for such a basic question -- I am a clinician with no formal training in data science.

======

Edited to add: maybe some more searching would have been wise. For Random Forests, is this giving me what I'm asking for?

predict(model_rf, test_cases, type = "prob")

Do other classification models typically offer this option for output?

Answer 1

Logistic regression seems suited to your needs. It gives the continuous measure of predicted probability of readmission that you want, and it is compatible with methods to deal with overfitting when that becomes an issue (as it already may be with a low-probability event like this).

The lrm function in the R ols package might be a wise choice to start, as that package has associated tools for validation and calibration. The author of the package, Frank Harrell, maintains a useful wiki on regression modeling, is author of the recently released second edition of "Regression Modeling Strategies" (links available from that wiki site), and is a frequent contributor to Cross Validated.

The main limitation, as you seem to understand, is the low percentage of events (readmissions). For every predictor variable you want to evaluate you need on the order of 10-20 events. That limitation, however, holds for any regression or classification scheme.

Added in response to comment, another answer, and further thought:

Some care does need to be taken in how you perform the logistic regression, if your 30-day hospital readmission rates are as low as 2% (versus typical rates closer to 20%, as summarized in this review of readmission models).

As @DJohnson points out in another answer, logistic regression can pose problems when there are few events, with the maximum-likelihood solution having a bias that underestimates the risk of the low-probability event. This paper discusses the issue. Problems can arise even if there appear to be many events if some particular combination of predictors is always associated with an event in your sample. (My sense is that this latter problem would also affect a Poisson regression, although I am open to arguments otherwise.)

There are several ways to deal with these issues; this page includes many solutions available in R, and the relogit function in the R zelig package was designed to handle this situation. The penalized-likelihood estimation already provided by the lrm package might be adequate, although this is getting a bit beyond my practical experience. This recent thesis compares several glm approaches (including asymmetric link functions), SVM, and Random Forest on 4 test data sets. Some competent local statistical support thus would be important for your project; you might find it useful (or at least amusing) to "please read this comment".

Finally, if you have the data, consider using a survival model rather than logistic regression. Even if your interest is in reportable readmissions within 30 days of discharge, you are dealing with some continuing risk of readmission. Patients readmitted after 30 days could provide useful information for getting a better estimate of the 30-day risk, which could be incorporated into a survival model but not so well into a logistic model.

Answer 2

I don't necessarily agree that logistic regression is that appropriate. The reason I say this is because it is well known that the logistic distribution does not fit infrequent or rare events -- the extremes of the distribution -- at all well. A better choice would be to use poisson regression, which is the basis for, e.g., the law of small numbers, low valued integer or count data where the variance equals the mean as well as being designed for predicting rare events. In addition, there are extensions and adaptations for zero-heavy event data as in zero-inflated poisson models.