# Tagged Questions

Receiver Operating Characteristic, also known as ROC curve.

1k views

### Adjusting for covariates in ROC curve analysis

This question is about estimating cut-off scores on a multi-dimensional screening questionnaire to predict a binary endpoint, in the presence of correlated scales. I was asked about the interest of ...
3k views

### ROC vs precision-and-recall curves

I understand the formal differences between them, what I want to know is when it is more relevant to use one vs. the other. Do they always provide complementary insight about the performance of a ...
7k views

### How to plot ROC curves in multiclass classification?

In other words, instead of having a two class problem I am dealing with 4 classes and still would like to assess performance using AUC.
396 views

### Combining classifiers by flipping a coin

I am studying a machine learning course and the lecture slides contain information what I find contradicting with the recommended book. The problem is the following: there are three classifiers: ...
556 views

What is the advantages of the ROC curves? For example I am classifying some images which is a binary classification problem. I extracted about 500 features and applied a features selection algorithm ...
1k views

### What do ROC curves tell you that traditional inference wouldn't?

When would you tend to use ROC curves over some other tests to determine the predictive ability of some measurement on an outcome? When dealing with discrete outcomes (alive/dead, present/absent), ...
4k views

### How to determine best cutoff point and its confidence interval using ROC curve in R?

I have the data of a test that could be used to distinguish normal and tumor cells. According to ROC curve it looks good for this purpose (area under curve is 0.9): My questions are: How to ...
696 views

### Optimising for Precision-Recall curves under class imbalance

I have a classification task where I have a number of predictors (one of which is the most informative), and I am using the MARS model to construct my classifier (I am interested in any simple model, ...
2k views

### What is the difference in what AIC and c-statistic (AUC) actually measure for model fit?

Akaike Information Criterion (AIC) and the c-statistic (area under ROC curve) are two measures of model fit for logistic regression. I am having trouble explaining what is going on when the results of ...
1k views

### How to do cross-validation with a Cox proportional hazards model?

Suppose I have constructed a prediction model for the occurrence of a particular disease in one dataset (the model building dataset) and now want to check how well the model works in a new dataset ...
221 views

### Random generation of scores similar to those of a classification model

Hello fellow number crunchers I want to generate n random scores (together with a class label) as if they had been produced by a binary classification model. In detail, the following properties are ...
329 views

### Analogues of sensitivity and specificity for continuous outcomes

How can I calculate the sensitivity and specificity (or analogous measures) of a continuous diagnostic test in predicting a continuous outcome (e.g., blood pressure) without dichotomizing the outcome? ...
818 views

### How to calculate sample size for comparing the area under the curve of two models?

Because I would like to calculate the sample size for comparing the area under the curve (AUC) of 2 models (cross-sectional study, predictor = continuous variable). Can you point me which function in ...
2k views

### Relation between true positive, false positive, false negative and true negative

I have values for True Positive (TP) and False Negative (FN) as follows: ...
138 views

### Evaluating a logistic regression

Recently I’ve been working on a logistic model and I´m having some difficulties evaluating the results. I am still learning all this, so I apologise in advance for the mistakes. I would still ...
766 views

### Evaluating and combining methods based on ROC and PR curves

I am evaluating and combining a few binary classification models. I am using the ROC and PR curves to evaluate their performance. The problem I am having is that as I try to improve the method, I am ...
368 views

### AUC in ordinal logistic regression

I'm using 2 kind of logistic regression - one is the simple type, for binary classification, and the other is ordinal logistic regression. For calculating the accuracy of the first, I used ...
385 views

### ROC surfaces in R

If my response variable has 2 levels, 0 and 1, I can use a ROC curve to assess the accuracy of my predictive model. But what if my response variable has 3 levels, -1, 0, and 1? Is there a way to ...
3k views

### Calculate ROC curve for data

So, I have 16 trials in which I am trying to authenticate a person from a biometric trait using Hamming Distance. My threshold is set to 3.5. My data is below and only trial 1 is a True Positive: ...
172 views

### Summary statistics of the precision-recall curve

From what I understand, one can use the AUC of the ROC curve as a summary statistic of the full curve. Q1. Are there any similar summary statistics that one can use on a single precision-recall ...
282 views

### Evaluation of classifiers: learning curves vs ROC curves

I would like to compare 2 different classifiers for a multiclass text classification problem that use large training datasets. I am doubting whether I should use ROC curves or learning curves to ...
721 views

### Is it valid to select a model based upon AUC?

I have plot ROC for several models. These models were used to classify my samples into 2 classes. Using these commands, I can obtain sensitivity vs. specificity plots for each model: ...
208 views

### Why does my ROC curve look like this (is it correct?)

I have a ROC curve generated for a multivariate logistic regression. Does it look correct? This is what I've done: Solve $\theta_0 + \theta_1X_1 + \theta_2X_2 ... = Y$ for the $\theta$s Iterate ...
985 views

### Choosing the right threshold for a biometric trait authentication system

I have a biometric authentication system that is using a person's gait to authenticate them. I extract features from gait, run it through a comparison versus a template and produce a similarity score ...
104 views

### Why do you maximize specificity for circumstantial findings?

Tests for a medical condition can be classified into two categories: Tests on an asymptomatic (no reason to suspect condition) patient where the result is a circumstantial finding; and Tests on a ...
650 views

### P-value of a survival ROC c-index

It is possible to calculate the c-index for time dependent outcomes (such as disease) using the survivalROC package in R. My question is : is it possible to produce a p-value for the c-index that is ...
2k views

### How to test the statistical significance of AUC?

(Step 1) Using my predictive model, I predicted 1000 scores for my sample dataset. (Step 2) I then calculate the random score using the same method for a randomized dataset. I firstly fit the ...
529 views

### Based only on these sensitivity and specificity values, what is the best decision method?

If I have the following sensitivity and specificity values, what is the best decision we can say in this case? ...
203 views

### Area under curve of ROC vs. overall accuracy

I am a little bit confusing about the Area Under Curve (AUC) of ROC and the overall accuracy. Will the AUC be proportional to the overall accuracy? In other words, when we have a larger overall ...
85 views

### CAP and ROC measures

Why is the area under the curve measure better than the accuracy ratio? I know that the AR lies between 0,5 and 1 and the relationship $AUC=\frac{1}{2}(AR+1)$ holds, so the AUC lies between 0,75 and ...
2k views

### ROC curve for discrete classifiers like SVM: Why do we still call it a “curve”?, Isn't it just a “point”?

In the discussion : how to generate a roc curve for binary classification, I think that the confusion was that a "binary classifier" (which is any classifier that separates 2 classes) was for Yang ...
102 views

### In R how to compute the p-value for area under ROC

I struggle to find a way to compute the p-value for the area under a receiver operator characteristic (ROC). I have a continuous variable and a diagnostic test result. I want to see if AUROC is ...
4k views

### How to do ROC-analysis in R with a Cox model

I've created a few Cox regression models and I would like to see how well these models perform and I thought that perhaps a ROC-curve or a c-statistic might be useful similar to this articles use: J. ...
625 views

### Plotting overlaid ROC curves

I'm trying to make overlaid ROC curves to represent successive improvements in model performance when particular predictors are added one at a time to the model. I want one ROC curve for each of about ...
1k views

### Lorenz curve and Gini coefficient for measuring classifier performance

I often use a ROC curve and the area under that curve as a measure of classifier accuracy in 2-class problems, e.g: ...
58 views

### Estimating ROC/AUC on large data sets?

Plotting an ROC curve of a classifier compared to cases requires that the data set be sorted first on the classifier score. I am in a position where I need to calculate ROC on a large data set very ...
81 views

### pattern of ROC curve and choice of AUC

I am using ROC curves and full AUC values to compare different models, using simulated data. Now I think I am confused with the interpretations of ROC curves and AUC values. Please see the figure ...
125 views

### How can I tell if my binary classifier is any good?

Say I have a data set with 10,000 rows and the target is a binary variable with 1500 positives (1's) and 8500 negatives (0's). I run a model and get predictions on the 0-1 interval. My question is ...
44 views

### Cutoff on ROC curve

@cbeleites mentions in response to the question How to determine best cutoff point and its confidence interval using ROC curve in R? that if one could specify the relationship between a false ...
94 views

### Alternating AUC curve. What does it mean?

Why do I see my ROC curve crossing the line from (0,0) to (1,1) (i.e. the 0-1 line)? I have the following test data as a tab-separated testdata.txt file. Running my R code (given below) multiple ...
386 views

### Fastest way to compare ROC curves

I have a set of true positive (TP) values which are used to train a model. I am using 5-fold cross validation to train my model (i.e. split my true positives into 5, use 4/5ths for training and ...
802 views

### Interpreting logistic regression

I need to perform a logistic regression to to see if a group of variables which are found to be significantly associated with an outcome (by univariate tests) have significant impact on the outcome ...
345 views

### Evaluation of classifier using ROC curve in the presence of rare events

For binary classification problems where the data is highly imbalanced, i.e. much more negative samples than positive samples, it is recommended to evaluate the performance of a classifier using the ...
339 views

### Rationale of using AUC?

Especially in the computer-science oriented side of the machine learning literature, AUC (area under the receiver operator characteristic curve) is a popular criterion for evaluating classifiers. ...
286 views

### Decision threshold for a 3-class Naive Bayes ROC curve

I have some doubts regarding how a ROC curve for a 3-class classifier (Naive Bayes) can be built. Basically, given some test data, the classifier outputs the probabilities for each of the 3 possible ...
122 views

### Predictive Probabilities

Other than a calibration plot, is there a way to decide how good one models' predictive probabilities as compared to another model. I'm not interested in error rates as I find them ineffective for ...
469 views

### ROC vs Accuracy

I've designed a 4 classifiers which perform pretty decently (all of them are above 90% in accuracy). However, they don't have similar AUC for their respective ROC curves (obviously, it doesn't have ...
381 views

### ROC plot for continuous data in R

I am currently estimating a bunch of ARMA models, and using them to predict subsets of my data. In order to evaluate their predictive accuracy I would like to make some ROC plots, however since all of ...