Receiver Operating Characteristic, also known as ROC curve.
15
votes
2answers
974 views
Adjusting for covariates in ROC curve analysis
This question is about estimating cut-off scores on a multi-dimensional screening questionnaire to predict a binary endpoint, in the presence of correlated scales.
I was asked about the interest of ...
14
votes
4answers
5k views
How to plot ROC curves in multiclass classification?
In other words, instead of having a two class problem I am dealing with 4 classes and still would like to assess performance using AUC.
14
votes
2answers
331 views
Combining classifiers by flipping a coin
I am studying a machine learning course and the lecture slides contain information what I find contradicting with the recommended book.
The problem is the following: there are three classifiers:
...
12
votes
1answer
2k views
ROC vs precision-and-recall curves
I understand the formal differences between them, what I want to know is when it is more relevant to use one vs. the other.
Do they always provide complementary insight about the performance of a ...
11
votes
2answers
424 views
Advantages of ROC curves
What is the advantages of the ROC curves?
For example I am classifying some images which is a binary classification problem.
I extracted about 500 features and applied a features selection algorithm ...
10
votes
5answers
1k views
What do ROC curves tell you that traditional inference wouldn't?
When would you tend to use ROC curves over some other tests to determine the predictive ability of some measurement on an outcome?
When dealing with discrete outcomes (alive/dead, present/absent), ...
9
votes
1answer
518 views
Optimising for Precision-Recall curves under class imbalance
I have a classification task where I have a number of predictors (one of which is the most informative), and I am using the MARS model to construct my classifier (I am interested in any simple model, ...
8
votes
1answer
881 views
How to do cross-validation with a Cox proportional hazards model?
Suppose I have constructed a prediction model for the occurrence of a particular disease in one dataset (the model building dataset) and now want to check how well the model works in a new dataset ...
7
votes
1answer
211 views
Random generation of scores similar to those of a classification model
Hello fellow number crunchers
I want to generate n random scores (together with a class label) as if they had been produced by a binary classification model. In detail, the following properties are ...
7
votes
1answer
675 views
How to calculate sample size for comparing the area under the curve of two models?
Because I would like to calculate the sample size for comparing the area under the curve (AUC) of 2 models (cross-sectional study, predictor = continuous variable). Can you point me which function in ...
6
votes
1answer
439 views
What is the difference in what AIC and c-statistic (AUC) actually measure for model fit?
Akaike Information Criterion (AIC) and the c-statistic (area under ROC curve) are two measures of model fit for logistic regression. I am having trouble explaining what is going on when the results of ...
6
votes
4answers
2k views
How to determine best cutoff point and it`s confidence interval using ROC curve in R?
I have the data of a test that could be used to distinguish normal and tumor cells. According to ROC curve it looks good for this purpose (area under curve is 0.9):
My questions are:
How to ...
6
votes
4answers
631 views
Evaluating and combining methods based on ROC and PR curves
I am evaluating and combining a few binary classification models. I am using the ROC and PR curves to evaluate their performance. The problem I am having is that as I try to improve the method, I am ...
5
votes
1answer
352 views
ROC surfaces in R
If my response variable has 2 levels, 0 and 1, I can use a ROC curve to assess the accuracy of my predictive model. But what if my response variable has 3 levels, -1, 0, and 1? Is there a way to ...
5
votes
0answers
300 views
AUC in ordinal logistic regression
I'm using 2 kind of logistic regression - one is the simple type, for binary classification, and the other is ordinal logistic regression. For calculating the accuracy of the first, I used ...
4
votes
2answers
2k views
Calculate ROC curve for data
So, I have 16 trials in which I am trying to authenticate a person from a biometric trait using Hamming Distance. My threshold is set to 3.5. My data is below and only trial 1 is a True Positive:
...
4
votes
2answers
117 views
Summary statistics of the precision-recall curve
From what I understand, one can use the AUC of the ROC curve as a summary statistic of the full curve.
Q1. Are there any similar summary statistics that one can use on a single precision-recall ...
4
votes
1answer
177 views
Evaluation of classifiers: learning curves vs ROC curves
I would like to compare 2 different classifiers for a multiclass text classification problem that use large training datasets. I am doubting whether I should use ROC curves or learning curves to ...
4
votes
3answers
562 views
Is it valid to select a model based upon AUC?
I have plot ROC for several models. These models were used to classify my samples into 2 classes.
Using these commands, I can obtain sensitivity vs. specificity plots for each model:
...
4
votes
1answer
846 views
Choosing the right threshold for a biometric trait authentication system
I have a biometric authentication system that is using a person's gait to authenticate them. I extract features from gait, run it through a comparison versus a template and produce a similarity score ...
4
votes
1answer
92 views
Why do you maximize specificity for circumstantial findings?
Tests for a medical condition can be classified into two categories:
Tests on an asymptomatic (no reason to suspect condition) patient where the result is a circumstantial finding; and
Tests on a ...
4
votes
0answers
2k views
How to test the statistical significance of AUC?
(Step 1) Using my predictive model, I predicted 1000 scores for my sample dataset.
(Step 2) I then calculate the random score using the same method for a randomized dataset. I firstly fit the ...
3
votes
1answer
54 views
CAP and ROC measures
Why is the area under the curve measure better than the accuracy ratio?
I know that the AR lies between 0,5 and 1 and the relationship
$AUC=\frac{1}{2}(AR+1)$ holds, so the AUC lies between 0,75 and ...
3
votes
2answers
1k views
ROC curve for discrete classifiers like SVM: Why do we still call it a “curve”?, Isn't it just a “point”?
In the discussion : how to generate a roc curve for binary classification, I think that the confusion was that a "binary classifier" (which is any classifier that separates 2 classes) was for Yang ...
3
votes
2answers
3k views
How to do ROC-analysis in R with a Cox model
I've created a few Cox regression models and I would like to see how well these models perform and I thought that perhaps a ROC-curve or a c-statistic might be useful similar to this articles use:
J. ...
3
votes
1answer
426 views
Plotting overlaid ROC curves
I'm trying to make overlaid ROC curves to represent successive improvements in model performance when particular predictors are added one at a time to the model. I want one ROC curve for each of about ...
3
votes
1answer
502 views
P-value of a survival ROC c-index
It is possible to calculate the c-index for time dependent outcomes (such as disease) using the survivalROC package in R. My question is : is it possible to produce a p-value for the c-index that is ...
2
votes
2answers
408 views
Based only on these sensitivity and specificity values, what is the best decision method?
If I have the following sensitivity and specificity values, what is the best decision we can say in this case?
...
2
votes
2answers
221 views
Fastest way to compare ROC curves
I have a set of true positive (TP) values which are used to train a model.
I am using 5-fold cross validation to train my model (i.e. split my true positives into 5, use 4/5ths for training and ...
2
votes
2answers
620 views
Interpreting logistic regression
I need to perform a logistic regression to to see if a group of variables which are found to be significantly associated with an outcome (by univariate tests) have significant impact on the outcome ...
2
votes
1answer
239 views
Evaluation of classifier using ROC curve in the presence of rare events
For binary classification problems where the data is highly imbalanced, i.e. much more negative samples than positive samples, it is recommended to evaluate the performance of a classifier using the ...
2
votes
1answer
220 views
Rationale of using AUC?
Especially in the computer-science oriented side of the machine learning literature, AUC (area under the receiver operator characteristic curve) is a popular criterion for evaluating classifiers. ...
2
votes
1answer
206 views
Decision threshold for a 3-class Naive Bayes ROC curve
I have some doubts regarding how a ROC curve for a 3-class classifier (Naive Bayes) can be built.
Basically, given some test data, the classifier outputs the probabilities for each of the 3 possible ...
2
votes
1answer
97 views
Predictive Probabilities
Other than a calibration plot, is there a way to decide how good one models' predictive probabilities as compared to another model.
I'm not interested in error rates as I find them ineffective for ...
2
votes
2answers
290 views
ROC vs Accuracy
I've designed a 4 classifiers which perform pretty decently (all of them are above 90% in accuracy).
However, they don't have similar AUC for their respective ROC curves (obviously, it doesn't have ...
2
votes
1answer
349 views
ROC plot for continuous data in R
I am currently estimating a bunch of ARMA models, and using them to predict subsets of my data. In order to evaluate their predictive accuracy I would like to make some ROC plots, however since all of ...
2
votes
1answer
37 views
Computing by hand the optimal threshold value for a biomarker using the Youden Index
I have an empirical estimate of a ROC curve, that is, a plot of the sensitivity versus 1-specificity over all possible cut-off values of the marker.
Based on an empirical ROC curve, I would like to ...
2
votes
1answer
161 views
What exactly the ROC curve can tell us or can be inferred?
(I post this originally at http://stackoverflow.com/questions/15477282/what-exactly-the-roc-curve-can-tell-us-or-can-be-inferred, but people directed me to here. Sorry about posting this twice.)
I ...
2
votes
1answer
180 views
Recall and AUC of a binary classifier
Is it possible for a binary classifier to have a recall of 0.0 for one of the classes and
at the same time an area under the ROC curve (AUC) of ...
2
votes
1answer
111 views
Replicating a plot from the ROCR Website
I'm trying to reproduce this plot from the ROCR website:
I can get something similar as follows:
...
2
votes
1answer
659 views
Lorenz curve and Gini coefficient for measuring classifier performance
I often use a ROC curve and the area under that curve as a measure of classifier accuracy in 2-class problems, e.g:
...
2
votes
1answer
204 views
Precision - Recall: Graphical Representation
I'm a little bit confused with precision recall. I read some papers about recommender systems, where in the one paper they have a graphical representation and in the other papers they don't (they just ...
2
votes
1answer
109 views
Analogues of sensitivity and specificity for continuous outcomes
How can I calculate the sensitivity and specificity (or analogous measures) of a continuous diagnostic test in predicting a continuous outcome (e.g., blood pressure) without dichotomizing the outcome? ...
2
votes
2answers
573 views
Creating ROC curve for multi-level logistic regression model in R
I used the functions from this link for creating ROC curve for logistic regression model. Since the object produced by glmer in ...
2
votes
0answers
43 views
Consistent ranked list for ROC interpolation
For classifiers with binary outputs, their performance is summarized by a true positive rate and false positive rate. To interpolate the performance between two classifiers $A$ and $B$ with their ...
2
votes
0answers
78 views
Validating a model for a set of DNA sequences
Consider a set of DNA sequences i.e. strings composed from the four
letters A, C, G, T.
We model these sequences as a random vector, $\mathbf{X}=(X_1,\dots,
X_n)$. Each member of the set of ...
1
vote
1answer
2k views
How do you generate ROC curves for leave-one-out cross validation?
When performing 5-fold cross-validation (for example), it is typical to compute a separate ROC curve for each of the 5 folds and often times a mean ROC curve with std. dev. shown as curve thickness.
...
1
vote
1answer
706 views
Is it necessary to plot ROC curve in a leave-one-out cross validation?
I have two classes and, in total, 40 subjects (20,20). I used an SVM to implement classification and did a leave-one-out cross validation to get an overall ...
1
vote
1answer
301 views
Binary classifier - dividing dataset into training and evaluation sets
I have a Hidden Markov Model for binary classification and two datasets:
positive instances
negative instances (way more data than the positive ones)
In order to evaluate the performance of the ...
1
vote
1answer
67 views
ROC curve crossing the diagonal
I am running a binary classifier at the moment. When I plot the ROC curve I get a good lift at the beginning then it changes direction and crosses the diagonal then of course back up, making the curve ...
