Receiver Operating Characteristic, also known as ROC curve.

learn more… | top users | synonyms

1
vote
1answer
19 views

ROC/AUC Confidence Interval

For a single ROC curve (with relevant AUC score), how can you calculate the confidence interval? (The data used to generate this ROC/AUC is available) Given my relatively limited background in this ...
0
votes
1answer
13 views

How can I get cut-off point in multivariated ROC analysis

If I have 1 independent variable (continues) and 1 dependent variable (binary), I can conduct logistic regression and ROC analysis, and I can get a cut-off point of independent variable using ROC ...
2
votes
1answer
39 views

Sample size calculation for ROC/AUC analysis

As a background, I am not familiar with stats except on a basic level. I have been tasked with doing some analysis that is out of my comfort zone. I am trying to figure out how to compute necessary ...
0
votes
0answers
19 views

Computing baseline probability

What is the meaning of the term Baseline probability in an experiment? How is it computed, say for a binary classifier? How to measure the performance of a classifier according to a given Baseline ...
2
votes
1answer
42 views

How we can draw an ROC curve for decision trees?

Normally we cannot draw an ROC curve for the discrete classifiers like decision trees. Am I right? Is there any way to draw an ROC curve for Dtrees?
2
votes
1answer
69 views

Understanding ROC curve[edited]

I'm having trouble understanding ROC curve. Is there any advantage/improvement in area under the ROC curve if I build different models from each unique subset of train set and use it to produce ...
0
votes
1answer
48 views

Confidence intervals for predictors in multivariate logistic regression

I've got a question. I am dealing with medical data which contain 5 predictors and 1 binary outcome. When I try to classify the data using all 5 predictors I get 0.84 area under roc-curve which is ...
0
votes
0answers
21 views

recursive feature elemination in R with caret

i work with R caret software package to select the most important features from some set of data. My response is a factor of multiple classes (e.g. nominal ...
0
votes
1answer
42 views

Confused with ROC curve and interpretation

The following figures show examples of ROC curves: First of all ignoring the picture, from a logical point one can say: When the cutoff value decreases, more and more cases are allocated to class 1 ...
0
votes
0answers
25 views

ROC and false positive rate with over sampling

I'm modelling a rare event (say 1 in 10000) and I'm using an over sampled train set to cross validate and train my model. I'm using ROC as a global performance metric but there are business reasons ...
1
vote
0answers
48 views

Comparing ROC-curves

I would like to find if there is a significant difference between two ROC-curves. I've found the roc.test in the pROC package. However, I cannot seem to find any information on how this test is ...
1
vote
1answer
48 views

Reverse AUC interpretation

Given a classifier (SVM) classifying in 2 'classes' (+1 or -1) for prediction purposes. It has an AUC score of 0.28, meaning its success rate is lower than just random predictions. If I just do the ...
0
votes
1answer
41 views

Problem with ROC curve in R

I am trying to plot the ROC curve for a random forest model (ROCR package), and I am getting weird results. I have double-checked the code several times, but I ...
1
vote
0answers
24 views

How to visualize the effect of a regression parameter

I am arguing that I can control error vs. coverage by modifying a certain parameter. After running an experiment with leave-many-out validation I have a set of errors along with the parameter value ...
3
votes
0answers
49 views

Internal validation via bootstrap: What ROC curve to present?

I am using the bootstrap approach for internal validation of a multivariate model built with either standard logistic regression OR elastic net. The procedure I use is as follows: 1) build model ...
1
vote
0answers
39 views

Comparing AUC vs accuracy

I understand this question has been asked many times however, i am unable to understand the answers well enough and apply to my situation. I have attached 2 screenshots of my model. There are 5 class ...
0
votes
0answers
89 views

Writing the objective function and constraints for scipy.optimize.minimize from matrices

I'm trying to find the optimal threshold point from a ROC curve. I have two main constraints : tpr >=80 and fpr <=60. I tried three main minimization functions : ...
0
votes
0answers
50 views

Confidence Intervals for AUC using cross-validation

I am analyzing the performance of a predictive model with the AUC, area under the ROC curve. I repeat several times cross-validation, and I have different estimations of the AUC in each folder. For ...
0
votes
1answer
28 views

Which quality measures are available for non-binary problems?

I'm wondering which quality measures are available for non-binary classifiers. I've read this article https://en.wikipedia.org/wiki/Receiver_operating_characteristic I understand, that the idea of ...
1
vote
1answer
47 views

Interpretation of the area under the PR curve

I'm currently comparing three methods and I have the Accuracy, auROC and auPR as metrics. And I have the following results : Method A - acc: 0.75, auROC: 0.75, auPR: 0.45 Method B - acc: 0.65, ...
0
votes
0answers
52 views

DeLong's Test to compare ROC curves

I am trying to compare the performance of a short and long form of a questionnaire. The long form has 25 items and the short form has 8 of the 25 items. Can DeLong's Test be used to compare the ROC ...
3
votes
2answers
136 views

Confused about sensitivity, specificity and area under ROC curve (AUC)

Just read a unpublished paper for review purpose. The reported results like Leave-one-out cross validation sensitivity is 95%. Leave-one-out cross validation specificity is 100%. Leave-one-out cross ...
0
votes
1answer
44 views

Drawing ROC plot for SDT in R

I am trying to plot an ideal ROC plot as predicted by Signal Detection Theory. Here are the formulas I try to plot: hi = Φ((d′−ci)/σ) (3) fi = Φ(−ci) (4)  ...
0
votes
0answers
61 views

Calibration curve in spss

I couldn't find any tutorials on this on youtube or otherwise. I am validating a clinical prediction model and I have a set of predicted outcomes and the real outcome. I have built a ROC curve but I ...
1
vote
0answers
35 views

Model validation for multilevel logistic regrssion?

I have designed a multilevel logistic model using PROC GLIMMIX in SAS 9.3 for hierachical data based on pupil attainment where level-two is the school the pupil attends. I'm quite sure that my model ...
0
votes
0answers
29 views

ROC curve and its function beginner

I have 3 features of a signal (example: amplitude, frequency, energy). I want to check which feature is the best to represent that particular signal. That signal is classified into two categories ...
1
vote
1answer
90 views

I want to learn about ROC curve — what is the canonical textbook?

I want to learn about Receiver-Operator-Characteristic curves, and metrics. I have read through online webpages with some basics, and I have used MATLAB built-ins to create ROC plots. It tells me ...
0
votes
0answers
32 views

How do the ROC cutoffs relate to predictors?

Apologies for this rather simple question, but I haven't been able to find a definition online. What does the ROC cutoffs represent for the AUC package? Specifically, how does it relate to the ...
2
votes
3answers
129 views

The value of adding the ROC graph if the AUC is given

I always see in papers that when they want to show how they classifiers performed, they provide ROC graph and the AUC score. Now as far as I know only the AUC tells how well the classifier performed, ...
3
votes
0answers
217 views

How to choose the cutoff probability for a rare event Logistic Regression

I have 100,000 observations (9 dummy indicator variables) with 1000 positives. Logistic Regression should work fine in this case but the cutoff probability puzzles me. In common literature, we ...
2
votes
2answers
114 views

Area under the ROC curve or area under the PR curve for imbalanced data?

I have some doubts about which performance measure to use, area under the ROC curve (TPR as a function of FPR) or area under the precision-recall curve (precision as a function of recall). My data is ...
5
votes
3answers
333 views

Why is AUC higher for a classifier that is less accurate than for one that is more accurate?

I have two classifiers A: naive Bayesian network B: tree (singly-connected) Bayesian network In terms of accuracy and other measures, A performs comparatively worse than B. However, when I use the ...
3
votes
1answer
110 views

Understanding random forest, gini, and KS

I'm a beginner machine learning user, doing my first predictive model using random forest. I have some questions regarding the way to measure how good a model is (Gini area from roc curve, and KS), ...
5
votes
2answers
133 views

How to do external validation of logistic regression models and perform model benchmarking

Quality assessment in trauma has for > 25 years been done with the US derived logistic regression model, the TRISS model. DV: survival/death and IVs: physiologic derangement (continuous), anatomic ...
0
votes
0answers
65 views

Random forest highly imbalanced dataset: how to test _ create ROC curve?

I have a dataset containing 10,000 examples with 8 features. I would like to create a random forest to classify this dataset in two classes, more specific: substrate / no substrate. In this dataset ...
1
vote
1answer
146 views

Why use d-prime instead of percent correct?

In signal detection theory, people often use $d'$ to assess performance. Apart from the fact that $d'$ is in $z$ units (units of measurement transformed to standard deviation units, i.e., $z$ scores), ...
2
votes
2answers
252 views

How to create ROC curve to assess the performance of regression models?

I knew that, ROC curve are use to assess the performance of classifiers. But is it possible to generate ROC curve for the regression model? If yes, How?
3
votes
1answer
148 views

Statistics for Area under the ROC curve

I have a question regarding statistical evaluation of the AUC. In their paper (http://www.jstor.org/stable/2531595), DeLong et al. describe a method to evaluate AUC curves. (Another good explanation ...
0
votes
1answer
116 views

LIBSVM ROC PLOt [closed]

my question is "What is command used to generate ROC PLOT in libsvm as well as for specificity , sensitivity and precision what is the command in libsvm " i Am using ubuntu
0
votes
0answers
21 views

Measuring the performance of a classifier that estimates bin counts for a Poisson process

I have some data that results from the convolution of a Poisson random process with a decaying exponential, plus some additive Gaussian noise. I'm working on designing a classifier that returns a ...
0
votes
0answers
22 views

VUS (volume under the surface)

After some research I did with the VUS(volume under the surface) for 3 or more class ordinal regression,I have this question. Let Y be a continuous out come from a 3 or more classifier. For example ...
3
votes
2answers
79 views

ROC graph interpretation

I'm reading Fawcett's 2004 paper on ROC graphs for machine learning algorithms, which can be found here. On page 7-8 he shows a simple ROC example and makes some interpretations that I don't ...
1
vote
2answers
140 views

Bootstrap to evaluate variance of AUC ROC

I have a toy dataset and want to eval the AUC - ROC with bootstrap ...
2
votes
1answer
107 views

Using predict_proba with sklearn's multiclass SVC

I'm using python's sklearn for multi-class classification (SVC) When using the predict method, i get very high scores with my dataset, However, I want to plot ROC curves for each of my classes. That ...
2
votes
0answers
81 views

Does it make sense to plot train and test results on a common ROC graph?

I am building a model using SPSS with 80% of my data. I can plot a ROC curve for both the training set and separately for the test set. I can´t find a way to plot both on the same graph. Since both ...
1
vote
0answers
62 views

Leave one subject out: Evaluating classifier performance

I have a small dataset, and I am trying to train 1-vs-all classifiers (class X or not class X) using a linear SVM. The dataset is simply too small to split up 60/40, 80/20 or anything like that. Also, ...
0
votes
0answers
64 views

How to compare several learning algorithms in a single domain

I want to compare several learning algorithms in a single domain. In the book " Evaluating Learning Algorithms" I saw the procedures to compare two learning algorithms in a single domain or multiple ...
0
votes
1answer
32 views

How to compute for the verification rate at 0.001 FAR?

I want to test my pair matching method based on this evaluation metric but unsure how to do this? I only have positive (matching) test samples.
1
vote
0answers
36 views

Calculate AUC of a logistic regression model [duplicate]

I have a data sample of a bank loan history of customers. I have performed logistic regression testing on the sample for finding out how the loan repayment(YES/NO) is dependent on various factors. I ...
1
vote
1answer
48 views

which performance metrics to classify model

I wonder between two performance metrics for classification models: accuracy and area under ROC curve (AUC), which one is to be preferred in which conditions? examples appreciated