Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
When would you tend to use ROC curves over some other tests to determine the predictive ability of some measurement on an outcome?
When dealing with discrete outcomes (alive/dead, present/absent), what makes ROC curves more or less powerful than something like a chi-square?
The ROC function (it is not necessarily a curve) allows you to assess the discrimination ability provided by a a specific statistical model (comprised of a predictor variable or a set of them).
A main consideration of ROCs is that model predictions do not only stem from the model's ability to discriminate/make predictions based on the evidence provided by predictor variables. Also operating is a response criteria that defines how much evidence is necessary for the model to predict a response, and what is the outcome of these responses. The value that is established for the response criteria will greatly influence the model predictions, and ultimately the type of mistakes that it will make.
Consider a generic model with predictor variables and a response criteria. This model is trying to predict the Presence of X,by responding Yes or No. So you have the following confusion matrix:
**X present X absent**
**Model Predicts X Present** Hit False Alarm
**Model Predicts X Absent** Miss Correct Rejection
In this matrix, you only need to consider the proportion of Hits and the False Alarms (because the others can be derived from these, given that they have to some to 1). For each response criteria, you wil ave a different confusion matrix. The errors (Misses and False Alarms) are negatively related, which means that a response criteria that minimizes false alarms maximizes misses and vice-versa. The message is: there is no free lunch.
So, in order to understand how well the model discriminates cases/makes predictions, independently of the response criteria established, you plot the Hits and False rates produced across the range of possible response criteria.
What you get from this plot is the ROC function. The area under the function provides an unbiased, and non-parametric measure of the discrimination ability of the model. This measure is very important because it is free of any confounds that could have been produced by the response criteria.
A second important aspect, is that by analyzing the function, one can define what response criteria is better for your objectives. What types of errors you want to avoid, and what are errors are OK. For instance, consider an HIV test: it is a test that looks up some sort of evidence (in this case antibodies) and makes a discrimination/prediction based on the comparison of the evidence with response criterion. This response criterion is usually set very low, so that you minimize Misses. Of course this will result in more False Alarms, which have a cost, but a cost that is negligible when compared to the Misses.
With ROCs you can assess some model's discrimination ability, independently of the response criteria, and also establish the optimal response criteria, given the needs and constraints of whatever that you are measuring. Tests like hi-square cannot help at all in this because even if your testing if the predictions are at chance level, many different Hit-False Alarm pairs are consistent with chance level.
Some frameworks, like signal detection theory, assume a priori that the evidence available for discrimination has specific distribuiton (e.g., normal distribution, or gamma distribution). When these assumptions hold (or are pretty close), some really nice measures are available that make your life easier.
hope this helps to elucidate you on the advantages of ROCs
An ROC curve is used when the predictor is continuous and the outcome is discrete, so a chi-square test would not be applicable. In fact, ROC analysis is in some sense equivalent to the Mann-Whitney test: the area under the curve is P(X>Y) which is the quantity being tested by the M-W test. However Mann-Whitney analysis does not emphasize selecting a cutoff, while that is the main point of the ROC analysis. Additionally, ROC curves are often used as just a visual display of the predictive ability of a covariate.
The shortest answer is that traditional tests of signal detection only give you a single point on the ROC (receiver operating characteristic) while the curve allows you to see responses through a range of values. It's possible that the criteria and d' shift as one shifts throughout the curve. It's like the difference between a t-test generated by selecting two classes of predictor variables and two regression lines generated by looking at parametric manipulations of each predictor variable.
In case you're interested in further references, an extensive list of papers is available on K.H. Zou's website, Receiver Operating Characteristic (ROC) Literature Research.
ROC curves are also used when one is interested in comparing different classifiers performance, with wide applications in biomedical research and bioinformatics.
In many ways ROCs are a diversion away from primary inference and estimation tools for models. I can't see much value there.
