# What do ROC curves tell you that traditional inference wouldn't?

When would you tend to use ROC curves over some other tests to determine the predictive ability of some measurement on an outcome?

When dealing with discrete outcomes (alive/dead, present/absent), what makes ROC curves more or less powerful than something like a chi-square?

The ROC function (it is not necessarily a curve) allows you to assess the discrimination ability provided by a a specific statistical model (comprised of a predictor variable or a set of them).

A main consideration of ROCs is that model predictions do not only stem from the model's ability to discriminate/make predictions based on the evidence provided by predictor variables. Also operating is a response criteria that defines how much evidence is necessary for the model to predict a response, and what is the outcome of these responses. The value that is established for the response criteria will greatly influence the model predictions, and ultimately the type of mistakes that it will make.

Consider a generic model with predictor variables and a response criteria. This model is trying to predict the Presence of X,by responding Yes or No. So you have the following confusion matrix:

                                **X present               X absent**
**Model Predicts X Present**       Hit                   False Alarm

**Model Predicts X Absent**      Miss                 Correct Rejection


In this matrix, you only need to consider the proportion of Hits and the False Alarms (because the others can be derived from these, given that they have to some to 1). For each response criteria, you wil ave a different confusion matrix. The errors (Misses and False Alarms) are negatively related, which means that a response criteria that minimizes false alarms maximizes misses and vice-versa. The message is: there is no free lunch.

So, in order to understand how well the model discriminates cases/makes predictions, independently of the response criteria established, you plot the Hits and False rates produced across the range of possible response criteria.

What you get from this plot is the ROC function. The area under the function provides an unbiased, and non-parametric measure of the discrimination ability of the model. This measure is very important because it is free of any confounds that could have been produced by the response criteria.

A second important aspect, is that by analyzing the function, one can define what response criteria is better for your objectives. What types of errors you want to avoid, and what are errors are OK. For instance, consider an HIV test: it is a test that looks up some sort of evidence (in this case antibodies) and makes a discrimination/prediction based on the comparison of the evidence with response criterion. This response criterion is usually set very low, so that you minimize Misses. Of course this will result in more False Alarms, which have a cost, but a cost that is negligible when compared to the Misses.

With ROCs you can assess some model's discrimination ability, independently of the response criteria, and also establish the optimal response criteria, given the needs and constraints of whatever that you are measuring. Tests like hi-square cannot help at all in this because even if your testing if the predictions are at chance level, many different Hit-False Alarm pairs are consistent with chance level.

Some frameworks, like signal detection theory, assume a priori that the evidence available for discrimination has specific distribuiton (e.g., normal distribution, or gamma distribution). When these assumptions hold (or are pretty close), some really nice measures are available that make your life easier.

hope this helps to elucidate you on the advantages of ROCs

• If we believe in models, then model-based estimates are ideal, and they are the most powerful/sensitive/precise. There are various classes of measures, such as explained variation measures like $R^2$ and generalizations of that. Other measures concentrate on the variety of predictions achieved by the model. A histogram of predicted values goes a long way. ROC curves envision different cutoffs. Cutoffs are misleading and dangerous; they give rise to categorical thinking, i.e., treating all persons in a group as if they have identical characteristics. Another approach: partitioning deviance. – Frank Harrell Oct 25 '11 at 22:52