5
$\begingroup$

If I swap positive class and negative class, then train a model again (I tried decision tree, adaboost, svm from scikit-learn built-in package) for a two class classification problem. Sometimes, I can see AUC slightly change (around 1-2%). Anyone have any ideas why there are such changes?

For ROC curve, x-axis is false positive rate, and y-axis and true positive rate. When prediction model gives prediction scores, we will order the scores from higher value to lower value, and then choose threshold according to the sorted values and calculate at the specific threshold point, what is the fpr and tpr. AUC is the area under ROC.

BTW, for swap, I mean manually assign negative label to be 1 and manually assign positive label as 0. I am asking if I swap, whether area of AUC may change?

Edit 1, here is how adaboost works, confused why it is not converged? From the formula, it should be converged. Refer from this book

enter image description here enter image description here enter image description here

$\endgroup$
  • 3
    $\begingroup$ This should have no effect on anything whatsoever. Your model is now predicting the "zero" class instead of the "one" class, which is just a matter of what names we're using for things. $\endgroup$ – dsaxton Sep 12 '16 at 0:45
  • 5
    $\begingroup$ Is there some randomness involved in the model fitting procedure? That could explain it, otherwise there should be no difference. That's because by flipping the labels the model is solving the exact same problem as before, which is classifying things into one of two classes. All that's changed is what we're calling an event and how we should interpret the model scores. $\endgroup$ – dsaxton Sep 12 '16 at 3:16
  • 3
    $\begingroup$ Yes. Random forests for instance use random sampling of the data / predictors, so fitting the model twice will result in slightly different models. $\endgroup$ – dsaxton Sep 12 '16 at 12:17
  • 1
    $\begingroup$ adaboost is not convergent, it random walks around in a small region of the solution space. $\endgroup$ – Carl Sep 15 '16 at 6:52
  • 1
    $\begingroup$ Well, random noise. That is adaboost never settles down, it takes a walk in solution space, if you restart with new initial conditions, it will not wind up in exactly the same place as the last time around. $\endgroup$ – Carl Sep 15 '16 at 7:20
1
+50
$\begingroup$

I believe one can show that the algorithms are not convergent. They get to close to a solution and then do a random walk.

$\endgroup$
  • $\begingroup$ Thanks Carl, but it cannot explain, if I do not swap positive and native label, AUC never changed. If there are randomness in adaboost Sci-kit learn implementation, even if I do not swap positive/negative label, result of AUC should also diff each time I run, correct? $\endgroup$ – Lin Ma Sep 18 '16 at 0:26
  • 1
    $\begingroup$ Well, if the same randomization (or its equivalent as inflexible algorithmic choice) is being repeated each time, then the runs will be identical, not correct, just identical. If one chooses different start conditions, then even the same randomization will lead to different answers. $\endgroup$ – Carl Sep 19 '16 at 16:30
  • $\begingroup$ Thanks Carl, vote up for your reply. I think adaboost should be converged. I post how it works in my original post, maybe it is specific implementation issue it is not converged? Could you help to summarize why it is not converged? $\endgroup$ – Lin Ma Sep 20 '16 at 6:09
  • 1
    $\begingroup$ And if the training interval is extended, what then? There is nothing sacrosanct about the training interval. $\endgroup$ – Carl Sep 20 '16 at 16:14
  • 1
    $\begingroup$ @LinMa The training interval looks like $1$ to $M$ in your equations. Increase $M$ and watch what happens. $\endgroup$ – Carl Sep 22 '16 at 1:39
3
$\begingroup$

I reproduced this so we can use my notebook as a reference:

https://github.com/csizsek/crossvalidated/blob/03ea088e0805bf550750d27735b38ebe1c9b567a/changing_roc_auc_score.ipynb

The data set I used is a simple Ecoli classification data set. You can see that I run the same classificator two times with the label unchanged, then I swap the labels and run it twice again and the ROC AUC score is always slightly different.

The reason why this is happening is exactly what @dsaxton said: that most classification algorithms (in this case a Random Forest) use some kind of random bootstrapping or something else that is random and the result model is always slightly different thus it's predictions and the ROC AUC score is different as well.

$\endgroup$
  • $\begingroup$ Thanks Peter, nice advice and vote up. Wondering if training and testing data are fixed, if there are still some randomness possible? $\endgroup$ – Lin Ma Sep 18 '16 at 0:24
  • $\begingroup$ BTW, Peter, but it cannot explain, if I do not swap positive and native label, AUC never changed. If there are randomness in random forest Sci-kit learn implementation, even if I do not swap positive/negative label, result of AUC should also diff each time I run, correct? $\endgroup$ – Lin Ma Sep 18 '16 at 0:27
  • $\begingroup$ 1) There is randomness in random forests, wiki Random Forest. 2) Scikit-learn's implementation of random forest without random_state parameter gives me slightly different result each time. modified notebook from the answer 3) If you can share data and/or implementation it would be better. $\endgroup$ – Michail L Sep 20 '16 at 11:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.