Search Results

Concordance probabilities, AKA $c$-index or AUROC are decent measures of pure predictive discrimination but are not recommended for comparison of two models, because of low power. Taking the differen …

The concordance probability (AUROC) is not sensitive enough for comparing models. Use a sensitive measure such as mean squared error, log-likelihood, AIC. Related information is here.

Comparing AUROC curves is equivalent to comparing two Wilcoxon test statistics, as the AUROC with binary Y is the $c$-index, i.e., concordance probability that is a unitless version of the Wilcoxon te …

Concordance probabilities ($c$ index; AUROC) are not sensitive enough to lead to good power for comparing models. Differences in AUROC are the same as differences in two Wilcoxon-Mann-Whitney test st …

ROC, sensitivity, and specificity have nothing to do with decision making as they are all in reverse time and information-flow order. …

The area under the ROC curve (AUROC) equals the Wilcoxon-Mann-Whitney-Somers concordance probability, a $U$-statistic, i.e., take all possible pairs of an observation with Y=0 and an observation with Y … Likewise every point on the ROC curve conditions on Y so the entire curve is conditional on Y. Each point is made up of probabilities like $\Pr(X > x | Y=y)$ ($y=0$ for x-axis, $y=1$ for y-axis). …

Choosing cutoffs in general, and in particular based on indexes derived from retrospective sampling (ROC curve, sens, spec), is a process that is completely at odds with decision making, which is a fully …

The AUROC, better understood as the concordance probability between predicted and observed values, has no problem with highly imbalanced data other than having a higher standard error than when the ou …

In extending If you like the Wilcoxon test you must like the proportional odds model I have simulations studying the concordance probabilities arising from the Kruskal-Wallis multi-group rank ANOVA s …

ROC curves are incompatible with optimal decision making because each point on the curve conditions on the future to predict the past. See https://www.fharrell.com/post/mlconfusion. …

Since all points on an ROC curve condition on Y, the distribution of Y is necessarily irrelevant for the points. … This also points out why ROC curves should not be used except in a retrospective case-control study where samples are taken from Y=0 and Y=1 observations. …

See Dana Quade's pair chart which relates to the Wilcoxon-Mann-Whitney two-sample rank-sum test.

ROC curves are not something you validate. Instead validate absolute predictive accuracy by estimating a smooth calibration curve (predicted probability vs. actual probability that Y=1). … You can validate the ROC area (c-index; Wilcoxon-Mann-Whitney concordance probability) as a measure of pure discrimination. But other indexes are better. …

Plotting the ROC will not help in any way. …

Concordance probability (AUROC; c-index) are not appropriate for comparing two models, because of lack of statistical power. See fharrell.com/post/addvalue for measures that are sensitive enough to b …