5
$\begingroup$

I am building a binary classification model to predict patient admission with respiratory issue in R. Each row in my data set is a patient record. The dependent variable is admit or not(1 or 0), and the features including age, gender, weather info, and air quality info. All the variables are in numeric type. The data set contains 70,000 records with admit rate around 3%.

I searched online for possible technique to deal with rare event problem. Like using xgboost, or resampling data set and combine with ML algorithms.

I am using three algorithms to compare the results. 1. xgboost. My codes are:

   pos_weight <- sum(training_df$resp_admit==0)/sum(training_df$resp_admit==1)
   xgb_mod <- xgboost(data = dtrain,
                      eta = 0.01,
                      max_depth = 9,  nround=3000, nthread = 2,
                      subsample = 0.9, colsample_bytree = 0.9,
                      eval_metric = "error", eval_metric = "auc",
                      objective = "binary:logistic",
                      max_delta_step = 6,
                      scale_pos_weight = pos_weight,
                      verbose = 1)

2. logistic regression and decision tree with resampling method(ovun.sample with method = both)

However, with parameter tuning and resampling method, the classifier performance is still bad for me.

   metric      logistic regression     xgboost
Sensitivity       0.0240480962         0.34482759
Specificity       0.9703949693         0.61143868
Pos Pred Value    0.0278422274         0.02945508
Neg Pred Value    0.9657548696         0.96465116
Precision         0.0278422274         0.02945508
Recall            0.0240480962         0.34482759
F1                0.0258064516         0.05427408
Balanced Accuracy 0.4972215327         0.47813313
AUC               0.4901495211         0.45190509

I am new to rare event problem. My questions are:

  1. I know the standard for the performance is different for different problem. But is there a overall standard? Like at least recall or precision should above 0.5?
  2. What other techniques can I try to improve the performance?
  3. Did I miss anything I should pay attention when I built the model?
  4. I randomly picked some basic features that I thought are important to build a simple model. Should I also do feature engineering? Does this step matter a lot to the model performance?

The following is how I constructed my data set.

  1. From all the patient admission records, picked out the ones admitted with respiratory issues. These records are associated with admission date. One patient may have multiple records at different admission dates. (Patients with multiple admissions count 15% among all the patients.) Labeled dependent variable for these records as 1.
  2. Group the above records by year-month. For each day in a month, if the patient didn't admit on that day, replicates patient info, and label as 0. Do this for all the patients fall in that month, and repeat the procedure for each different year-month. The reason I didn't generate 0 records across the whole time period is that if I did so, the rare event rate will be around 0.1%.
  3. Combine all the 1 and 0 records, left join the weather and air quality info by date.

I am also concerning about the way I constructed my data set. Discussions about this are welcomed.

The final goal of this model is to make respiratory admission risk predictions on a patient list, based on patient info and current day weather and air quality info.

$\endgroup$
  • $\begingroup$ Is your data really binary? If we have weather and air quality, that's presumably for a particular day or week? Or are intervals of unequal length? If so, do you have more than one record per patient? $\endgroup$ – Björn Jul 11 '17 at 6:02
  • 1
    $\begingroup$ Did you try SMOTE? There is a related question about ROSE vs SMOTE. stats.stackexchange.com/questions/166458/… $\endgroup$ – user3611727 Jul 11 '17 at 8:03
  • $\begingroup$ @Björn I have added the description of how I constructed my data set at the end of my question. $\endgroup$ – Y.Li Jul 11 '17 at 11:23
  • 1
    $\begingroup$ @LmnICE Thank you for the article. I used to deal with unbalanced problems with the methods mentioned in the article. However, these methods didn't solve my issues with this problem. I assume that my problem is more complicated, other than a simple imbalanced problem. $\endgroup$ – Y.Li Jul 12 '17 at 1:33
  • 1
    $\begingroup$ Thinking about it you are also only looking at re-hospitalizations within 1 month, if you only consider days wi t hin 1 month (i.e. first discharge day + 28 or 30 or so days - I hope you are not looking at the calendar month of the hospitalization date). If that's what you want each new first hospitalization outside that period cannot count, but starts a new at risk period. I suspect you really have a serious issue with having a clear question and a data selection that is relevant to the question. $\endgroup$ – Björn Jul 12 '17 at 5:39
5
$\begingroup$

Casting this as a classification problem was a major misstep. This is inherently a "tendency estimation", i.e., probability estimation problem. That is what logistic regression is all about. And you've chosen improper accuracy scores - scores that are optimized by choosing the wrong features and giving them the wrong weights. For details see http://www.fharrell.com/2017/01/classification-vs-prediction.html and http://www.fharrell.com/2017/03/damage-caused-by-classification.html

$\endgroup$
  • 1
    $\begingroup$ I have read the two articles. They are very inspiring to me. I can't fully understand the knowledge and techniques in the articles currently, but I will keep learning. You mentioned different logistic regression methods in your article's comments. I was wondering for imbalanced data set, is penalized likelihood logistic regression better than resampling method? Could you suggest more on this kind issue? $\endgroup$ – Y.Li Jul 12 '17 at 3:33
  • $\begingroup$ When I performed logistic regression and xgboost, the output probability are very small. The median of the probability is around 0,05. For this case, how to define a standard for high/low risk properly? $\endgroup$ – Y.Li Jul 12 '17 at 3:35
3
$\begingroup$

In addition to Frank Harrell's important point about classification versus prediction, you might need to consider that you don't have the information needed to judge the probability of admission. AUC is the one measure in your list that isn't subject to arbitrary choices of cutoffs for classification, and it is very close to the value of 0.5 seen when your model is no better than random. It might just be too hard to determine the precise day when an individual will be admitted, particularly for a first admission, which seems to be what your data-set construction and analysis are examining.

Dealing with re-admission characteristics might be possible with a survival model. The starting date in each case would be the date of discharge from the first admission. Then you could incorporate weather and air-quality measures as time-dependent covariates in a model of time-to-readmission.

$\endgroup$
  • $\begingroup$ Could you explain the reason you don't have the information needed to judge the probability of admission? $\endgroup$ – Y.Li Jul 12 '17 at 1:44
  • $\begingroup$ Logistic regression provides perhaps the best approach for predicting admissions. Yet the AUC with the predictors you included in the logistic model showed no better performance than chance. The simplest explanation is that the predictors you included in your model don't really help predict admission probability, at least in the way that you modeled it. $\endgroup$ – EdM Jul 12 '17 at 2:26
  • $\begingroup$ If I perform analysis on readmission problem, besides survival analysis, is logistic regression still an option(if the AUC score is much better than 0.5)? $\endgroup$ – Y.Li Jul 12 '17 at 3:21
  • $\begingroup$ For analysis of the time it takes for an event to occur (like time to re-admission), survival analysis is typically better and provides more power. See this answer for example. $\endgroup$ – EdM Jul 12 '17 at 7:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.