Receiver Operating Characteristic, also known as ROC curve.

learn more… | top users | synonyms

2
votes
1answer
13 views

How to compare predictive power of sports odds

For my MSc Thesis I would like to compare the predictive power of classic bookmakers on the one hand and a betting exchange on the other. I have a lot of data on both types of betting, mostly on ...
2
votes
1answer
70 views

Comparing logistic regression models with AUC ROC in R vs Stata

I am fitting a logistic regression model for the likelihood of patients suffering morbidity after surgery. The most commonly used prediction tool at the moment is POSSUM (Physiological and Operative ...
0
votes
0answers
23 views

ROC calculation in LOOCV context - caret

I am not sure how caret handle the ROC calculation when used with LOOCV. From what I understand, in the more common case where a 10-fold cross validation is used, the ROC value is calculated for each ...
0
votes
1answer
35 views

Relationship between AUC and U Mann-Whitney statistic

Recently I learned about the relationship between Area Under (ROC) Curve and $U$ statistic of the Wilcoxon-Mann-Whitney test. It is supposed to follow the following rule (got it from this nice post on ...
0
votes
0answers
12 views

How is it correct to optimize a binary classifier output threshold with ROC and LPOCV?

Hello everyone and thank you in advance for you help! I'm building a screening tool with a machine learning algorithm. The model provides a probabilistic prediction (i.e. logistic regression, ...
0
votes
0answers
16 views

Scenario of how to compare models

I have a binary classification problem where the distribution of classes is skewed. I've already trained some scoring models with logistic regression. Now I would like to compare them. How to do this ...
0
votes
0answers
20 views

Calculating Efficiency in ROC analyses

I don't have a detailed understanding of statistics and need help. I'm working on determining cutoff scores for a brief measure using SPSS. Background: we administered one measure at three different ...
1
vote
1answer
59 views

100% training accuracy despite a low cv score

I am working on an assignment where we have to study the affect of gamma and C parameters on SVM with RBF kernel. I use python's sklearn library and grid search with 10 fold cross validation (with a ...
1
vote
0answers
24 views

AUC for ROC on predictors

During preprocessing how good is AUC ROC used on predictors as a way to eliminate predictors? What size of AUC is too low to be considered useful?
0
votes
0answers
23 views

Optimizing a model for a limited budget

I am building models to predict probability of failures against a list of approximately 500K assets. I want to optimize my models for maximum predictive performance on a fixed (limited) number of ...
1
vote
1answer
37 views

Optimal cut-off calculation in logistic regression

I am building a logistic regression model and am using k-fold cross-validation for model selection. My doubt is with regards to misclassification rate. For measuring that, i will have to first find ...
4
votes
2answers
77 views

“Good” classifier destroyed my Precision-Recall curve. What happened?

I'm working with imbalanced data, where there are about 40 class=0 cases for every class=1. I can reasonably discriminate between the classes using individual features, and training a naive Bayes and ...
0
votes
0answers
9 views

Cross-Validation for single feature with two labels

In systems neuroscience it is common to report the area under ROC when comparing signals between two different outcomes (two labels). For example you can compare the number of spikes (one feature) ...
0
votes
0answers
23 views

How do I judge low probability model quality?

I'm modeling loan defaults - an unlikely occurrence. My logistic regression model predicts probabilities ranging from 0.001% (squeaky clean) to 44% (hinky). Each specific prediction is "no, this ...
0
votes
0answers
23 views

AUROC for SVM two class classification [duplicate]

I'm trying to compute the ROC and AUROC of a binary svm classification. I already looked up a code in the internet and it's working: ...
0
votes
0answers
46 views

Accuracy Ratio Brute-force vs Logistic Regression

We want to model a binary dependent variable $Y$ with 0 or 1 values (e.g. whether a loan defaults or not) based on 3 independent variables $X_1$, $X_2$ and $X_3$. I have the following 2 methods and I ...
0
votes
0answers
15 views

Signal Detection Theory, How do I create a ROC curve from 2 distributions?

I have 2 samples and I have to decide using the Signal Detection Theory which one is the true distribution. This is my data: ...
0
votes
1answer
43 views

Deep Neural Networks: AUCROC Values Consistently = 0.5 even though RMS Error on Test Set ~10%

I am new to Neural Networks and but I have built a multi-classifier using the FANN neural network package. My multi-classifier, regardless of the network hyperparameters, consistently gives an error ...
0
votes
0answers
15 views

How to plot ROC for LOOCV by hand based on output probabilities returned by SVM? [duplicate]

I want to plot ROC for Leave-one-out (LOO) cross-validation (CV). I have the decision values as well as probability values for each class from SVM classifier. I did see the answer to this post; ...
1
vote
0answers
27 views

Can I draw a single ROC curve for a test method applied onto a group of people with different levels (i.e., light, mild,sever) of sickness

Can I draw a single ROC curve for a test method applied onto a group of people of different levels of sickness severity (i.e., light, mild, severe)? Or I need to draw three different ROC curves, one ...
3
votes
1answer
77 views

AUC values for different sets of features

I have a dataset with two groups of features, set 1 and set2. I have traind and tested SVM classifiers in three different settings: 1) only set 1 features, 2) only set 2 features, and 3) union of set ...
1
vote
1answer
73 views

Why compare AUC's in binary classification?

I understand that a common metric for comparing binary classifiers is the AUC of the ROC curve. But, after this is computed, only one threshold is actually chosen for classifying negative and ...
1
vote
0answers
39 views

AUC for binary ROC curve

I am using the ROCR package in R to calculate ROC and associated AUC for an arbitrary continuous data set with labels coded as 0 or 1. In case A, I have some set of labels for each entry in the data ...
0
votes
1answer
48 views

Performance of a classifier change heavily

I'm using a data set of 32 face persons and a svm-rbf to classify and a random group of 70% for train and 30% for test. The problem is that my results are heavily dependent of the random group used ...
0
votes
0answers
24 views

Reporting AUC on training or testing data

I have a really simple question. I am writing an article to submit to a conference. I have used SVM classifier in it. I have seen in many papers which report ROC and AUC for their classifiers, and I ...
0
votes
0answers
27 views

ROC change after variable selection with glmnet

I was using glmnet in caret to select important variables. The code is like ...
0
votes
2answers
62 views

R Caret - Repeated CrossValidation, finalModel and ROC curves

I got a problem understanding the meaning of the finalModel when using a repeated CV. ...
0
votes
0answers
13 views

Were there a diagnostic value if ROC analysis showed a high AUC but a low specificity?

Were there a diagnostic value if ROC analysis showed a high AUC but a low specificity? for example, spcificity below 0.7.
1
vote
1answer
55 views

Does dice coefficient same as accuracy?

I come across dice coefficient for volume similarity (https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient) and accuracy (https://en.wikipedia.org/wiki/Accuracy_and_precision). It ...
1
vote
0answers
26 views

How to plot ROC for knn (and potentially kernel spectral regression)

I understand how to plot ROC for logistic classifier (like varies the probability cutoff). For KNN, how can I find the ROC? Also, what about kernel spectral regression?
0
votes
0answers
13 views

Can two Classifier with different number of False Positives have exactly same ROC curve?

There are two classifiers, the first one return Scores ...
0
votes
0answers
30 views

The meaning of “training accuracy”?

If I split my data set into testing, training (further separated into subtraining and validation data set in cross-validation). In the context of machine learning and esp. in those ROC comparing the ...
3
votes
1answer
84 views

ROC curve drawbacks

In the class yesterday, we were taught about logistic and subsequently the ROC curve and how to use it. My questions are: Is this the most common way to identify if the logistic model is the ...
1
vote
0answers
24 views

Equivalence of dominant classifier performance in ROC and PR spaces

In their article of 2006, The Relationship Between Precision-Recall and ROC Curves, J. Davis & M. Goadrich show that ROC (TPR vs FPR) and Precision-Recall (PR) curves have one-to-one ...
0
votes
0answers
13 views

Minimum sample size to establish a cutoff point

I conducted a case-control study (obese and normal body weight). I want to conduct ROC curve between my variables to establish a cutoff point for vitamin depending on insulin resistance risk. Does it ...
0
votes
1answer
34 views

Performance evaluation measures for binary random forest classifier

I am using Random forest (Matlab) to classify the binary data. Broadly, the input to the random forest is number of features and class label. And random forest, after training, return the labels for ...
0
votes
0answers
122 views

Leave-one-out cross validation output interpretation and ROC curve

I have taken plenty of time to try and help myself, but I keep reaching dead ends. I have a dataset consisting of body measurements collected from a bird species, and the sex of each bird (known by ...
0
votes
1answer
360 views

R - Plotting a ROC curve for a Naive Bayes classifier using ROCR. Not sure if I'm plotting it correctly

I have a Naive Bayes classifiers that I'm using to try to predict whether a game is going to win or lose based on historical data. The model has 25 variables in total, all of which are categorical ...
1
vote
0answers
66 views

A strange ROC curve

I produced the ROC curve for a continuous biomarker used as a classifier toward a binary class (absence/presence of a disease according to another biomarker), using the ROCR package in R. I have a ...
21
votes
4answers
3k views

What is the name of this chart showing false and true positive rates and how is it generated?

The image below shows a continuous curve of false positive rates vs. true positive rates: However, what I don't immediately get is how these rates are being calculated. If a method is applied to a ...
3
votes
2answers
110 views

Diagonal in ROC plot?

I've used the R ROCR package to generate an ROC plot. I can see a diagonal in my plot. However, the ROC curve that I see other people generated has only horizontal and vertical lines. Why would I get ...
0
votes
0answers
48 views

Use AUC for model comparison but what is the optimal threshold for final prediction

We can compare the performance of different models using AUC ROC and pick the one with large AUC. Then, we still need to choose and use specific threshold to predict the label for the test data. I ...
0
votes
0answers
20 views

In the classification framework, is AUROC a performance measure or metric?

I guess, the title is self-explaining. I have seen both so far and was wondering if there is a correct term or whether it does not really matter.
4
votes
1answer
70 views

Why two interpretations of AUC(area under the ROC curver) Equivalent

I found there are two ways to understand what AUC stands for but I couldn't get why these two interpretations are equivalent mathematically. In the first interpretation, AUC is the area under the ...
0
votes
0answers
15 views

Can I use p-value as the criteria to generate ROC cuvers

I developed an algorithm to predict positive and negative samples. The algorithm returns a p-value for each sample. Can I use the p-value (ranging from 0 to 1) as the criteria to generate a ROC curve? ...
0
votes
0answers
40 views

ROC curve smoothening

I conducted a logistic regression analysis in SPSS which resulted with a very good model. However when I wanted to do the ROC curve, it was drawn using only three points (see figure). All the examples ...
0
votes
1answer
37 views

Why is ROC curve always inreasing (or non-decreasing)?

As far as I understood, we plot the (False positive Rate, True Positive Rate) points for all values of threshold. Why should this curve be increasing always? (or non-decreasing)
17
votes
1answer
721 views

Did I just invent a Bayesian method for analysis of ROC curves?

Preamble This is a long post. If you're re-reading this, please note that I've revised the question portion, though the background material remains the same. Additionally, I believe that I've devised ...
0
votes
1answer
41 views

Is it legitimate to use sensitivity and specificity next to more proper performance measures to compare classifiers?

Clearly, Brier Score and AUROC are better performance measures to compare classifiers. However, besides that, I am interested in a let's call it more economic view. I could imagine a classifier being ...
0
votes
1answer
74 views

How to draw the ROC curve?

Currently I'm asking me how to draw the ROC curve (Receiver Operating Characteristic curve). On the x-axis there is the false positive rate (1-specificity) and on the y-axis there is the true positive ...