I have an imbalanced dataset with around 200k instances and 50 predictors. The imbalance has a 4:1 ratio for the negative class (i.e class 0). In other words the negative class makes around 80% of the samples and the positive just 20% of the samples.
It's a binary classification problem where I have a target vector with 0's and 1's.
I have been trying to fit several classifiers like logistic regression and random forest.
I evaluate them with cross -validation
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=999)and ROC
roc_curve from Python
My ROC curve for each validation fold are the same and I have no idea why. The
AUC is also always absurdly good (
0.9). Although the Precision-Recall Curve shows worse
AUC=0.74 (which I think it's more accurate).
I tried following this example for ROC with cross-validation: http://lijiancheng0614.github.io/scikit-learn/auto_examples/model_selection/plot_roc_crossval.html#example-model-selection-plot-roc-crossval-py
ROC curves Logistic Regression
The question: Why does the performance of the model seems to be similar on each fold? shouldn't they differ at least slightly?
clasifier = linear_model.LogisticRegression(class_weight = "balanced") clasifier.fit(X,y) fig, ax1 = plt.subplots(figsize=(12, 8)) mean_tpr = 0.0 mean_fpr = linspace(0, 1, 100) skf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=999) for i, (train_index, test_index) in enumerate(skf.split(X,y)): # calculate the probability of each class assuming it to be positive probas_ = classifier.fit(X[train_index], y[train_index]).predict_proba(X[test_index]) # Compute ROC curve and area under the curve fpr, tpr, thresholds = roc_curve(y[test_index], probas_[:, 1], pos_label=1) mean_tpr += interp(mean_fpr, fpr, tpr) mean_tpr = 0.0 roc_auc = auc(fpr, tpr) plt.plot(fpr, tpr, lw=1, label='ROC fold %d (area = %0.2f)' % (i+1, roc_auc)) plt.plot([0, 1], [0, 1], '--', color=(0.6, 0.6, 0.6), label='Random', lw=2) mean_tpr /= n_folds mean_tpr[-1] = 1.0 mean_auc = auc(mean_fpr, mean_tpr) plt.plot(mean_fpr, mean_tpr, 'k--', label='Mean ROC (area = %0.2f)' % mean_auc, lw=3) plt.xlim([-0.05, 1.05]) plt.ylim([-0.05, 1.05]) plt.xlabel('False Positive Rate (1- specificity)', fontsize=18) plt.ylabel('True Positive Rate (sensitivity)', fontsize=18)