When are ROC curves to compare imaging tests valid? (Focus on the example below)

Question

I would like to ask what criticism could be raised in the following case example: In this paper they test a way of detecting narrowing of the cervical canal on radiographs using a ratio of measurements. In the subjects studied they do have a CT-scan as "gold standard": it is well-established that a spinal canal measured on CT of $<12$ mm is diagnostic of narrowing ($1$ label in logistic regression).

Here is the stated aim of the study:

Ratios with significant correlation coefficients [with the CT measurement] were evaluated with receiver operating characteristic (ROC) curve analysis to define the cutoff ratio value that optimized the sensitivity and false-positive rate (1 - specificity) for indicating developmental cervical stenosis, which was defined as a sagittal canal diameter of <12 mm on CT.

with the following conclusion:

However, ROC curve analysis showed that only an LM/CD ratio [one of the methods of measuring the diameter on radiographs] of $\geq 0.735$ indicated a canal diameter of $<12$ mm (developmental cervical stenosis). [$<12$ mm is the agreed upon threshold on the "gold standard"].

1. Is it methodologically correct to compare these different ratios of measurements of the spinal canal (LM/CD, SL/LM, etc) for accuracy using ROC's? Under what criteria is it OK in general?
2. Is it correct to derive a cutoff point of $0.735$ from the ROC curves?
3. And, much less important but curious, wouldn't SL/VB be just as good an (inverse) classifier as LM/CD, indicating a widely open spinal canal?

Regarding point (2) the use of ROC curves to establish thresholds in medicine is very problematic, and as explained in Scientific American, "Which threshold is optimal for a given population depends on such factors as the seriousness of the condition being diagnosed, the prevalence of the condition in a population, the availability of corrective measures for those who are diagnosed, and the financial, emotional and other costs of false alarms.".

Unfortunately, the paper is sparse in statistical details, but I presume the cutoff value indicated was chosen as to maximize the Youden's J statistic (lr.eta). Without positive evidence that this is the method that was used, it has problems because the cost ratio varies with prevalence, as in this article kindly shared by @Scortchi.

Regarding point (3) I wonder if this measurement should have somehow been inverted to place it back into the competition for the "best ratio" as a negative predictor, since "any classifier that produces a point in the lower right triangle can be negated to produce a point in the upper left triangle."

As an illustration, and using a PSA (prostatic specific antigen) dataset available for download here, the total PSA could be considered a good indicator of prostate cancer. The ROC plot exhibits a convex hull with an AUC of $0.85,$ and p-value of $<0.01$; however, just changing the sign (or inverting the value of the concentration of $\text{[PSA]}$ to $1/\text{[PSA]}$) resulted in a mirror image:

One may conclude that the ratios that predicted a large diameter of the canal (and therefore could act as good indicators as to the absence of canal stenosis) were not considered in the study, because no simple transformation was applied to for instance SL/VB (in yellow) to flip it onto the upper triangle:

After @Carl's observation about the fact that the positive correlation of SL/VB with the canal diameter of $0.652$ does not jibe with the concave curve and low AUC, the point (3) is reinforced - they are throwing in the same bag and comparing measures of wide cervical canal (healthy) with a positive correlation together with a measure of narrow canal (disease) with the only negative correlation:

---

In general, the accuracy or performance of different diagnostic tests is ubiquitously analyzed in ROC curves, and the AUC's reported with or without CI's. Combinations of different classifiers are also often compared. This is such common practice that it is difficult for me to tell whether it is just something that is perhaps abused but not going away (as p values), and therefore not worth mentioning on peer-reviews, or a practice that is acceptable under some conditions that avoid the many criticisms that ROC curves have received, including AUC factoring in segments of little to no interest in the curve. In this regard, should the ROC curves be presented as supplemental, and avoid presenting "ROC analysis" as the method?

Answer 1

The particular paper in question, P.H. Horne et al, A Novel Radiographic Indicator of Developmental Cervical Stenosis, J Bone Joint Surg Am. (2016) 98:1206-14, seems to be an unfortunate example of what one might call "premature dichotomization." There is an established cutoff of <12 mm in saggital spinal canal diameter to classify someone as having "cervical stenosis," based on reconstruction from 3D imaging (like computed tomography scans). The authors examined four measurements from more readily available 2D imaging (which is also less expensive and involves much lower radiation doses) in patients who also had canal diameters determined from 3D imaging. The authors examined whether those measurements in 2D could be used to predict cervical stenosis.

This study would have been a great opportunity to model saggital canal diameter as a function of all these 2D measurements, and see how well true canal diameter could be modeled. Unfortunately, the authors only examined individual correlations of each of those 4 measurements with canal diameter to start, and then looked at correlations of canal diameter with a set of pairwise ratios of 2D measurements. That approach thus threw away the more detailed information that a multiple-regression approach involving all 4 measurements together might have provided.

Then, to evaluate these less-than-ideal pairwise ratios, the authors seem to have ignored the actual measurements of canal diameter, and only tried to predict the 3D-based classifications into stenosis/normal. The receiver operating characteristic (ROC) curves shown in the paper and in this question show how changing the cutoff for each of those ratios affects the sensitivity and specificity of identifying stenosis. A model in which all measurements were used to estimate canal diameter (along with an error estimate), and only then making the call of < 12 mm diameter, would probably have been much more useful.

Although this isn't a great paper from a statistical perspective, the questions raised about it are of general interest and deserve discussion. D. Hand, in Measuring classifier performance: a coherent alternative to the area under the ROC curve, Mach Learn (2009) 77: 103–123 (referenced in this related question) provides an important key.

Hand considers two classes labeled $k=0$ and $k=1$, prevalences $\pi_k$, and density functions $f_k(s)$ describing the distribution within each class of a score $s$ that is monotonically increasing with the probability of membership in class $1$. The cost of misclassification into class $k$ is $c_k$, with $c$ the cost ratio for misclassification into class $0$, $c =c_0/(c_0+c_1)$.

When the cost ratio is expressed this way and you have the correct model for the probability of class membership, the cost-optimal probability cutoff for class assignment is $c$. Thus a generic measure of model quality might not provide much guidance in applying the model. What's critical is having a well calibrated model of class membership probability, particularly for probabilities near the ultimate decision point if the relative misclassification costs are known. Put another way, any choice of a probability or score cutoff is making an implicit choice about those relative costs.

Hand shows (page 111) that the area under the ROC curve, the AUC,

is equivalent to taking an average of the losses corresponding to different cost ratios $c$, where the average is calculated according to the distribution:

$$w(c) = \pi_0 f_0 (P_1^{-1}(c)) \left| \frac{dP_1^{-1}(c)}{dc} \right| + \pi_1 f_1 (P_1^{-1}(c)) \left| \frac{dP_1^{-1}(c)}{dc} \right|.$$

Here, $P_1^{-1}(c)$ represents the cost-optimal score/probability threshold for classification. This illustrates two problems with using the AUC to compare different classifiers. First, as Hand continues;

The implication of this is that the weight distribution over cost ratios $c$, implicitly used in calculating the AUC, depends on the empirical score distributions $f_k$. That is, the weight distribution used to combine different cost ratios c, will vary from classifier to classifier. But this is absurd. The beliefs about likely values of $c$ must be obtained from considerations separate from the data: they are part of the problem definition. One cannot change one’s mind about how important one regards a misclassification according to which tool one uses to make that classification. Nevertheless, this is effectively what the AUC does—-it evaluates different classifiers using different metrics.

Second, the weighted average further depends on the class prevalences, $\pi_0$ and $\pi_1$. That can lead to further confusion, described for example by T.M. Hamill and J. Juras, Measuring forecast skill: is it real skill or is it the varying climatology?, Q. J. R. Meteorol. Soc. (2006), 132: 2905–2923.

Applying these principles to the 3 specific questions with respect to the Horne et al paper:

1. Is it methodologically correct to compare these different ratios of measurements of the spinal canal (LM/CD, SL/LM, etc) for accuracy using ROC's? Under what criteria is it OK in general?

For now, put aside the broader problems with experimental design raised at the beginning. If one takes "compare ... accuracy using ROC's" to mean comparing the AUC values, then that can be dangerous in general. In addition to ignoring relative costs of different misclassifications and the problems of different distributions of within-class scores among the classification schemes that Hand discusses, there is a potentially big problem here arising from the prevalence $\pi$ of stenosis. The population in the Horne et al paper consisted of individuals who already had 2D and 3D imaging for some clinical indication. One probably would not want to apply the same criteria to a broader population in which the prevalence of stenosis might be much lower and relative misclassification costs might differ.

Furthermore, even if one chooses to ignore these problems, the AUC is not very sensitive for distinguishing among models. Again, calibration is key. With the sample sizes typical of such clinical studies, comparisons of model performance are better based on resampling, for example repeating the modeling on multiple bootstrap samples from the data and evaluating on the full data set.

2, Is it correct to derive a cutoff point of 0.735 from the ROC curves?

That choice seems to be made for the point on the ROC that has the farthest perpendicular distance from the diagonal line representing no skill, called (among other things) the maximum Peirce skill score. In A Note On the Maximum Peirce Skill Score, Weather and Forecasting (2007) 22: 1148-1154, A. Manzato says: "it is the ROC point that maximizes the skill of the classifier." Nevertheless, that choice of cutoff does not take the relative misclassification costs into account, as Manzato goes on to demonstrate. Whether that choice is "correct" depends on the intended use of the scoring system and the relative misclassification costs, which Horne et al don't seem to discuss.

3. And, much less important but curious, wouldn't SL/VB be just as good an (inverse) classifier as LM/CD, indicating a widely open spinal canal?

In general, if a particular scoring system does that good a job of choosing the incorrect class, just choose the other class.

Note, however, that much of the above has to do with problems in comparing different scoring systems. For any one scoring system, the ROC curve still provides a convenient overview of the underlying sensitivity/specificity tradeoff, particularly if the curve is correspondingly labeled with scores. And for any one scoring system, the AUC provides the fraction of pairs of different-class cases for which the difference in relative scores agrees with class membership.