I am building a healthcare readmission model. It is a binary classification task. I had around 90K observations with close 500 features. Except 9-10 features, rest all are binary features.
I did 5 fold cross validation and ran three algos. Stochastic gradient descent SVM, Vanilla Logistic regression, Logistic regression with elastic net regularization.
There is a big class imbalance problem and it is around 8:1 in favor of negative class. Here is the count of the dependent variable values.
model_data1.readmission.value_counts()
Out[101]:
0 80571
1 9717
dtype: int64
Because the class imbalance is there I used the class_weights parameter in sklearn algos and weighted it 1:8 in favor of positive class. Here is one example code:
sgd_lr=SGDClassifier(loss='log',penalty='elasticnet',alpha=0.002,l1_ratio=0.70,class_weight={0:1,1:8})
And Here is the diagnostic statistics of all the three algorithms:
('The mean accuracy of Stochastic Gradient Descent SVM on CV data is:', 0.99944621379697762)
('The mean accuracy of Logistic regression on CV data is:', 1.0)
('The mean accuracy of Stochastic Gradient Descent Logistic on CV data is:', 0.99968355807253673)
('The accuracy of SGD SVM on test data is:', 0.99929855650311961)
Classification Metrics for
precision recall f1-score support
0 1.00 1.00 1.00 24134
1 0.99 1.00 1.00 2953
avg / total 1.00 1.00 1.00 27087
Confusion matrix
[[24117 17]
[ 2 2951]]
('The accuracy of Logistic with Elastic Net on test data is:', 0.99963081921216823)
Classification Metrics for
precision recall f1-score support
0 1.00 1.00 1.00 24134
1 1.00 1.00 1.00 2953
avg / total 1.00 1.00 1.00 27087
Confusion matrix
[[24127 7]
[ 3 2950]]
('The accuracy of Logistic Regression on test data is:', 1.0)
Classification Metrics for
precision recall f1-score support
0 1.00 1.00 1.00 24134
1 1.00 1.00 1.00 2953
avg / total 1.00 1.00 1.00 27087
Confusion matrix
[[24134 0]
[ 0 2953]]
ROC Curve for Stochastic Gradient Descent SVM:
ROC Curve for Stochastic Gradient Descent Logistic Regression with Elastic Net:
ROC Curve for Stochastic Gradient Descent Logistic Regression with L1:
In [ ]:
And here is the ROC curve.
I am not sure this can be true. One cannot get 100% accuracy on test data in a machine learning exercise. That is very unbelievable. Please advise what is going wrong here?.
Thanks