1
$\begingroup$

UPDATE Confidence Intervals enter image description here

I have an imbalanced dataset with around 200k instances and 50 predictors. The imbalance has a 4:1 ratio for the negative class (i.e class 0). In other words the negative class makes around 80% of the samples and the positive just 20% of the samples.

It's a binary classification problem where I have a target vector with 0's and 1's.

I have been trying to fit several classifiers like logistic regression and random forest.

I evaluate them with cross -validation skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=999)and ROC roc_curve from Python sklearn v.018

My Problem

My ROC curve for each validation fold are the same and I have no idea why. The AUC is also always absurdly good (0.9). Although the Precision-Recall Curve shows worse AUC=0.74 (which I think it's more accurate).

I tried following this example for ROC with cross-validation: http://lijiancheng0614.github.io/scikit-learn/auto_examples/model_selection/plot_roc_crossval.html#example-model-selection-plot-roc-crossval-py

ROC curves Logistic Regression

enter image description here Precision Recall curves PR curve

ROC curves Random Forestsenter image description here

The question: Why does the performance of the model seems to be similar on each fold? shouldn't they differ at least slightly?

Code Below

clasifier = linear_model.LogisticRegression(class_weight = "balanced")
clasifier.fit(X,y)
fig, ax1 = plt.subplots(figsize=(12, 8))
    mean_tpr = 0.0
    mean_fpr = linspace(0, 1, 100)

    skf = StratifiedKFold(n_splits=n_folds, shuffle=True, random_state=999)

    for i, (train_index, test_index) in enumerate(skf.split(X,y)):
        # calculate the probability of each class assuming it to be positive
        probas_ = classifier.fit(X[train_index], y[train_index]).predict_proba(X[test_index])
        # Compute ROC curve and area under the curve
        fpr, tpr, thresholds = roc_curve(y[test_index], probas_[:, 1], pos_label=1)
        mean_tpr += interp(mean_fpr, fpr, tpr)
        mean_tpr[0] = 0.0
        roc_auc = auc(fpr, tpr)

        plt.plot(fpr, tpr, lw=1, label='ROC fold %d (area = %0.2f)' % (i+1, roc_auc))

    plt.plot([0, 1], [0, 1], '--', color=(0.6, 0.6, 0.6), label='Random', lw=2)


    mean_tpr /= n_folds
    mean_tpr[-1] = 1.0
    mean_auc = auc(mean_fpr, mean_tpr)

    plt.plot(mean_fpr, mean_tpr, 'k--',
         label='Mean ROC (area = %0.2f)' % mean_auc, lw=3)
    plt.xlim([-0.05, 1.05])
    plt.ylim([-0.05, 1.05])
    plt.xlabel('False Positive Rate (1- specificity)', fontsize=18)
    plt.ylabel('True Positive Rate (sensitivity)', fontsize=18)
$\endgroup$
  • $\begingroup$ What classifier are you using $\endgroup$ – Alex R. Mar 18 '17 at 16:21
  • $\begingroup$ How large is your data set? $\endgroup$ – Calimo Mar 19 '17 at 7:14
  • $\begingroup$ @Calimo 200k rows and 50 predictors/columns. $\endgroup$ – Roxanne Mar 19 '17 at 9:52
  • $\begingroup$ @AlexR. logistic regression like this : clasifier = linear_model.LogisticRegression(class_weight = "balanced") $\endgroup$ – Roxanne Mar 19 '17 at 9:53
  • $\begingroup$ Sorry I missed that. Now you can calculate the standard error and confidence intervals of your AUC with 200/k = 40 000. I bet it's below 0.01. $\endgroup$ – Calimo Mar 19 '17 at 10:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.