1
$\begingroup$

I am building a healthcare readmission model. It is a binary classification task. I had around 90K observations with close 500 features. Except 9-10 features, rest all are binary features.

I did 5 fold cross validation and ran three algos. Stochastic gradient descent SVM, Vanilla Logistic regression, Logistic regression with elastic net regularization.

There is a big class imbalance problem and it is around 8:1 in favor of negative class. Here is the count of the dependent variable values.

model_data1.readmission.value_counts()
Out[101]:
0    80571
1     9717
dtype: int64

Because the class imbalance is there I used the class_weights parameter in sklearn algos and weighted it 1:8 in favor of positive class. Here is one example code:

sgd_lr=SGDClassifier(loss='log',penalty='elasticnet',alpha=0.002,l1_ratio=0.70,class_weight={0:1,1:8})

And Here is the diagnostic statistics of all the three algorithms:

('The mean accuracy of Stochastic Gradient Descent SVM on CV data is:', 0.99944621379697762)
('The mean accuracy of Logistic regression on CV data is:', 1.0)
('The mean accuracy of Stochastic Gradient Descent Logistic on CV data is:', 0.99968355807253673)
('The accuracy of SGD SVM on test data is:', 0.99929855650311961)
Classification Metrics for
             precision    recall  f1-score   support

          0       1.00      1.00      1.00     24134
          1       0.99      1.00      1.00      2953

avg / total       1.00      1.00      1.00     27087

Confusion matrix
[[24117    17]
 [    2  2951]]
('The accuracy of Logistic with Elastic Net on test data is:', 0.99963081921216823)
Classification Metrics for
             precision    recall  f1-score   support

          0       1.00      1.00      1.00     24134
          1       1.00      1.00      1.00      2953

avg / total       1.00      1.00      1.00     27087

Confusion matrix
[[24127     7]
 [    3  2950]]
('The accuracy of Logistic Regression on test data is:', 1.0)
Classification Metrics for
             precision    recall  f1-score   support

          0       1.00      1.00      1.00     24134
          1       1.00      1.00      1.00      2953

avg / total       1.00      1.00      1.00     27087

Confusion matrix
[[24134     0]
 [    0  2953]]
ROC Curve for Stochastic Gradient Descent SVM:


ROC Curve for Stochastic Gradient Descent Logistic Regression with Elastic Net:


ROC Curve for Stochastic Gradient Descent Logistic Regression with L1:



In [ ]:

And here is the ROC curve.

enter image description here

I am not sure this can be true. One cannot get 100% accuracy on test data in a machine learning exercise. That is very unbelievable. Please advise what is going wrong here?.

Thanks

$\endgroup$
  • 1
    $\begingroup$ Did you check if your independent variables did not contain the dependent or a proxy for it? $\endgroup$ – spdrnl Dec 3 '15 at 10:59
  • $\begingroup$ No I removed the dependent variable from the design matrix. So dependent variable is not present. There are other features but I don't think any is proxy. Those are normal features $\endgroup$ – Baktaawar Dec 3 '15 at 12:18
  • $\begingroup$ I see you use python. You could try to fit a tree and get the variable importance graphically: scikit-learn.org/stable/auto_examples/ensemble/… Then you could start to eliminate variables to get more insight into what is going on. $\endgroup$ – spdrnl Dec 3 '15 at 12:21
  • $\begingroup$ See this old post by Brian Ripley: math.yorku.ca/Who/Faculty/Monette/S-news/0027.html $\endgroup$ – kjetil b halvorsen Feb 9 '17 at 20:33
3
$\begingroup$

This situation can occur in logistic regression with a large number of candidate predictors, even with thousands of cases as you have here. With 500 binary predictor variables you have $2^{500} = 3.3 \times 10^{150} $ possible combinations of predictors. This is approximately the square of the number of atoms in the universe. You have found a particular combination of predictors that completely distinguish the two groups in this particular data set. This result is unlikely to generalize to new cases, as you recognize.

For ideas about how to proceed, follow the hauck-donner-effect tag on this site. This page and this page are good places to start.

In response to comment:

If you set aside separate training and test sets, either you got really lucky in finding a reproducible linear separation in your training set, or some predictor is acting as a proxy for readmission. You are unlikely, unfortunately, to have solved the vexing problem of predicting readmission, where an AUC of 0.8 might be considered spectacular.

Make sure that the "normal features" you are examining are only those with data available at the time of prior discharge (before the readmission/no-readmission cutoff time). With 500 features (a lot for clinical data) it's possible that one slipped through that would only have a particular value for readmitted patients. Follow the suggestions by @spdrnl in the comments to examine the individual predictors and combinations in more detail.

$\endgroup$
  • $\begingroup$ Here is my question. When you say it won't generalize to new cases why is that?. This AUC and everything I am seeing is on new cases only. Test data and not training data. So there is no overfitting happening here if you think that ways. $\endgroup$ – Baktaawar Dec 3 '15 at 20:47
  • $\begingroup$ Please see the edit to my post. $\endgroup$ – EdM Dec 3 '15 at 22:07
  • $\begingroup$ Do you suggest me looking at the features which are having non zero weights and see if any of them is proxy for re-admission?. Or do you want to look at all the features. Those which have zero weight would anyway not helping in classification right?. So then only those which have weights can be checked if there is any which is a proxy for re-admission. Does this sound good? $\endgroup$ – Baktaawar Dec 3 '15 at 22:30
  • $\begingroup$ Restricting to predictors with non-zero coefficients should suffice. $\endgroup$ – EdM Dec 3 '15 at 23:05
  • $\begingroup$ A really excellent answer. $\endgroup$ – rolando2 Dec 4 '15 at 3:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.