question regarding ROC curve

Question

When I was learning data-analysis course online, the lecturer spoke of two advantages of ROC curve. He said "that AUC results do not change with changes in the incidents of the actual condition, nor is AUC affected by changes in the relative cost of the two different types of binary classification errors, false positives and false negatives. Therefore, when either future incidents or the cost of classification errors or both are unstable or cannot be known, the AUC is generally the best possible performance metric available."

One thing I was confused about was why ROC curve will not be affected by the changes in the incidents of the actual condition. Based on what I know, ROC is mapped out from the data people have been collected, or the curve's x and y location is the true negative rate and the true positive rate at different thresholds, which are all related to events.

Therefore, could anyone explain what the lecturer's meaning? Thanks a lot.

Answer 1

You can look at the definition of the two axes to understand this. The two axes for an ROC curve: X-axis : FPR (False positive rate) Y-axis : TPR (True positive rate)

In equation form:

FPR = FP/(FP + TN)

TPR = TP/(TP + FN)

So say you have a binary target and your ratio of 0s to 1s changes in the dataset. Then for both equations above, the numerator and the denominator scale together. (FP and TN are both cases where ground truth is 0, and TP and FN where the ground truth is 1).

For some intuition, say your dataset today has a much larger number/proportion of 0s as your data set yesterday, and you are measuring ROC on a daily basis. You'd expect your number of false positives to go up (since these are 0s being mistakenly predicted by your model as 1), but FPR would stay stable since your number of 0s has also gone up proportionally. Similar story for TPR.