5
$\begingroup$

I have three datasets as below. They are about the same prediction task.

Data 1:

  • Training data: 95% positive instances and 5% negative instances

  • Test data: 95% positive instances and 5% negative instances

Data 2:

  • Training data: 50% positive instances and 50% negative instances

  • Test data: 50% positive instances and 50% negative instances

Data 3:

  • Training data: 5% positive instances and 95% negative instances

  • Test data: 5% positive instances and 95% negative instances

I use logistic regression to fit the three datasets respectively. I use the AUC function provided by scikit-learn to measure the prediction outcome.

Surprisingly, the AUC of the Data 1 is 0.92, the AUC of the Data 2 is 0.75, the AUC fo the Data 3 is 0.68.

Why does the AUC monotonically decrease? Is there any possible reason? I would like to have some kinds of numbers to prove the reason.

======================================================================== Below is some descriptive analysis about the prediction

Data 1: TRUTH: 
AUC: 0.925431817577
CLASS 1:
prediction mean: 0.999236699597 # This is the mean of the prediction outcomes which in fact are in class 1.
prediction std: 0.00360159690023
CLASS 0:
prediction mean: 0.993764261308
prediction std: 0.0198596992508
Effect Size of the two groups: 0.18829037121178865

Data 2: TRUTH: 
AUC: 0.755869422091
CLASS 1:
prediction mean: 0.953888184569
prediction std: 0.0538778221966
CLASS 0:
prediction mean: 0.908878619149
prediction std: 0.0738915897689
Effect Size of the two groups: 0.3286908171245826

Data 3: 
AUC: 0.681406318885
CLASS 1:
prediction mean: 0.121257041956
prediction std: 0.090470866846
CLASS 0:
prediction mean: 0.070718980391
prediction std: 0.0579126148936
Effect Size of the two groups: 0.31566692944753133

It seems that the effect sizes tell me the difference of the predictions in the Data 3 is more significant than that in the Data 2. However, the AUC of Data 1 is higher. This makes me very confused.

The distributions of the prediction outcomes are as below: enter image description here

Zoom in the upper left corner. enter image description here

Below is the ROC. I still do not understand why the ROC of data 1 is the best.

enter image description hereenter image description hereenter image description here

$\endgroup$

closed as unclear what you're asking by usεr11852, mkt, mdewey, whuber Feb 3 at 15:02

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ From your description, it appears you are fitting a model to each dataset and then computing AUC on on the same dataset. In other words, no cross-validation is occurring. (Is that correct? Or is the AUC you report the average over k-fold cross-validation?) Unfortunately, without cross-validation, all the AUC tells us is the tendency of the model to over fit. I encourage you to repeat your experiment and calculate AUC using cross-validation. scikit-learn.org/stable/auto_examples/model_selection/… $\endgroup$ – olooney Aug 19 '18 at 23:43
  • 1
    $\begingroup$ I'm voting to close this question as off-topic because it was abandoned by the OP after a request for clarification. $\endgroup$ – mkt Feb 3 at 8:46