Search Results

ROC curves are not informative in 99% of the cases I've seen over the past few years. They seem to be thought of as obligatory by many statisticians and even more machine learning practitioners. … ROC curves cannot be used to find optimum tradeoffs except in very special cases where users of a decision rule abdicate their loss (cost; utility) function to the analyst. …

Once you have accurate calibration, you can also assess predictive discrimination using e.g. the area under the ROC curve or $c$-index which don't require consideration of any cutpoints at all. …

the probability of an event, not for classification Data splitting as a validation method (whether using 2 or 3 subsets) only works when you start with an enormous sample size, otherwise it is volatile ROC … curves are not compatible with decision making - see the Diagnosis chapter of BBR ROC curves should not be used for obtaining a cutoff value - see the Information Loss chapter of BBR as well as http:/ …

Many researchers have tried to make decisions on the basis of shape differences in ROC curves, but in my view this has been futile. … And areas under ROC are not sensitive enough measures for comparing two models. …

ROC curves are not something you validate. Instead validate absolute predictive accuracy by estimating a smooth calibration curve (predicted probability vs. actual probability that Y=1). … You can validate the ROC area (c-index; Wilcoxon-Mann-Whitney concordance probability) as a measure of pure discrimination. But other indexes are better. …

Since all points on an ROC curve condition on Y, the distribution of Y is necessarily irrelevant for the points. … This also points out why ROC curves should not be used except in a retrospective case-control study where samples are taken from Y=0 and Y=1 observations. …

As detailed here and in the Diagnosis chapter of BBR, ROC curves are inefficient, non-enlightening, and inconsistent with optimal decision making. … Comparison of ROC curves and AUROC is statistically non-powerful. …

You are making the assumption that the ROC curve is informative and leads to good decisions. Neither is true. I have yet to see an ROC curve that provided useful insight. … There is no need to present an ROC curve. Besides having low information yield, ROC curves invite analysts to seek cutpoints on predicted probabilities, which is a decision-making disaster. …

The ROC curve is not helpful here; neither are sensitivity or specificity which, like overall classification accuracy, are improper accuracy scoring rules that are optimized by a bogus model not fitted … Note that you achieve high predictive discrimination (high $c$-index (ROC area)) by overfitting the data. …

Drawing the ROC curve leads to more confusion. Think about what is at the root of your issue. You can't interpret pieces of the ROC curve without having a utility/cost/loss function. … The area under the ROC curve happens to equal the $c$-index (concordance probability) which is a simple interpretable pure measure of predictive discrimination. …

Note that whenever you use an ROC curve to choose a threshold on a predictor or on predicted probability, you are turning the analysis into a decision problem that is not using the appropriate utility/ …

You didn't state the ultimate goal of the exercise, hence the choice of ROC curves was not well motivated. …

In addition make sure that the area of the ROC curve ($c$-index; concordance probability) is really what you need. …

ROC curves are very poor methods for comparing models. The are insensitive, tempt one to make arbitrary dichotomizations, and do not provide hints of corrective actions. …

First of all, as detailed here you cannot get ROC curves from a classifier because a classifier is a 0-1 output. So you must not really be using a classifier but rather a probability estimator. … Then you get a single concordance probability (c-index; AUROC) and don't look so much at the individual concordance probabilities/ROC curves. …