Receiver Operating Characteristic, also known as ROC curve.

learn more… | top users | synonyms

0
votes
0answers
26 views

Is it logical to stand on the chance-line (50-50 %) when we don't know a-priori probabilities?

Let we have two hypotheses $H_0$ and $H_1$ and we don't know their a-priori probabilities. If we wish to calculate the average probability of error, does it makes sense to assume 50-50 % chance of ...
0
votes
0answers
9 views

How to evaluate a trained model using parameters other than AUC in RapidMiner?

I am using RapidMiner to build predictive models trained and cross-validated by a set of medical data(65 cases. 18 attributes), I am now running trials by trying different combinations of learners and ...
0
votes
0answers
35 views

Cutoff and precision values of a binary classifier

Let's say I have fitted a binary classifier to some data and I'm varying the cutoff value, effectively producing a ROC-curve. Knowing the true proportions of positives and negatives, I can calculate ...
2
votes
2answers
67 views

ROC for more than 2 outcome categories

How do you construct ROC Curves when there are more than two outcome categories (in my case, I have four)? I've heard you should do this for the most popular group. Are there any other ideas? Are ...
0
votes
0answers
34 views

Selecting individuals from a population using a binary classifier

I have a dataset consisting of around 200 individuals, whose outcome is either of state $0$ or $1$. I am able to make binary classifiers and predictors on this set and build ROC-curves for them just ...
1
vote
3answers
116 views

ROC graph shape

Could you explain to me how the shape of a ROC curve is determined? From the following illustration, it seems that for every time the actual class (C) is positive, it goes up and when it's negative, ...
2
votes
0answers
42 views

Unbalanced dataset - ROC curve to compare classifiers?

I use the machine learning software WEKA for data mining on biological data. I would describe my dataset as unbalanced: It comprises around 2000 instances, ...
2
votes
1answer
62 views

Is it reasonable for a classifier to obtain a high AUC and a low MCC? Or the opposite?

Let's say I have 2 models: 1) High Matthew's correlation coefficient (MCC) score, low area under the curve (AUC) 2) Low MCC, high AUC When I say high and low, I mean relatively to the other model. ...
-1
votes
1answer
23 views

Which is better ROC curve or Confusion matrix?

anyone know about the predictive model evaluation? I'm confused about the ROC curve and confusion matrix. The area under the curve for ROC is represent about the accuracy of the classifier. But what ...
0
votes
1answer
43 views

Using the Caret package is it possible to obtain Confusion Matrices for specific threshold values?

I've obtained a logistic regression model (via train) for a binary response, and I've obtained the logistic confusion matrix via ...
0
votes
0answers
21 views

Multi-class AUC in Matlab

I would like to compute the area under the ROC-courve (AUC) metric for a classifier with multiple classes. Do you know (reliable) functions for Matlab that implement methods for that, like e.g. in ...
2
votes
1answer
43 views

Differences in AUC calculation between pROC and ROCR

Does anyone know the difference in calculation between these two AUC packages? They get different results when I add in positives with predicted value of 0 (simulating a prob model where many outputs ...
3
votes
1answer
49 views

ROC-AUC and Precision-Recall for random classifiers in class imbalanced problems

I have always always understood the diagonal of the ROC plot to represent the performance of a "random" classifier (corresponding to an AUC of 0.5). Is this still the case for highly imbalanced ...
2
votes
2answers
117 views

ROC vs. Accuracy [duplicate]

If you want to compare two learning algorithms, which metric is better to use in general: ROC or accuracy? I understand that in ROC, you get both the sensitivity and specificity?
1
vote
1answer
34 views

ROC/AUC Confidence Interval

For a single ROC curve (with relevant AUC score), how can you calculate the confidence interval? (The data used to generate this ROC/AUC is available) Given my relatively limited background in this ...
0
votes
1answer
19 views

How can I get cut-off point in multivariated ROC analysis

If I have 1 independent variable (continues) and 1 dependent variable (binary), I can conduct logistic regression and ROC analysis, and I can get a cut-off point of independent variable using ROC ...
2
votes
1answer
76 views

Sample size calculation for ROC/AUC analysis

As a background, I am not familiar with stats except on a basic level. I have been tasked with doing some analysis that is out of my comfort zone. I am trying to figure out how to compute necessary ...
0
votes
0answers
20 views

Computing baseline probability

What is the meaning of the term Baseline probability in an experiment? How is it computed, say for a binary classifier? How to measure the performance of a classifier according to a given Baseline ...
2
votes
2answers
81 views

How we can draw an ROC curve for decision trees?

Normally we cannot draw an ROC curve for the discrete classifiers like decision trees. Am I right? Is there any way to draw an ROC curve for Dtrees?
2
votes
1answer
113 views

Understanding ROC curve[edited]

I'm having trouble understanding ROC curve. Is there any advantage/improvement in area under the ROC curve if I build different models from each unique subset of train set and use it to produce ...
0
votes
1answer
56 views

Confidence intervals for predictors in multivariate logistic regression

I've got a question. I am dealing with medical data which contain 5 predictors and 1 binary outcome. When I try to classify the data using all 5 predictors I get 0.84 area under roc-curve which is ...
0
votes
0answers
39 views

recursive feature elemination in R with caret

i work with R caret software package to select the most important features from some set of data. My response is a factor of multiple classes (e.g. nominal ...
0
votes
1answer
54 views

Confused with ROC curve and interpretation

The following figures show examples of ROC curves: First of all ignoring the picture, from a logical point one can say: When the cutoff value decreases, more and more cases are allocated to class 1 ...
0
votes
0answers
31 views

ROC and false positive rate with over sampling

I'm modelling a rare event (say 1 in 10000) and I'm using an over sampled train set to cross validate and train my model. I'm using ROC as a global performance metric but there are business reasons ...
1
vote
0answers
62 views

Comparing ROC-curves

I would like to find if there is a significant difference between two ROC-curves. I've found the roc.test in the pROC package. However, I cannot seem to find any information on how this test is ...
1
vote
1answer
50 views

Reverse AUC interpretation

Given a classifier (SVM) classifying in 2 'classes' (+1 or -1) for prediction purposes. It has an AUC score of 0.28, meaning its success rate is lower than just random predictions. If I just do the ...
0
votes
1answer
54 views

Problem with ROC curve in R

I am trying to plot the ROC curve for a random forest model (ROCR package), and I am getting weird results. I have double-checked the code several times, but I ...
1
vote
0answers
27 views

How to visualize the effect of a regression parameter

I am arguing that I can control error vs. coverage by modifying a certain parameter. After running an experiment with leave-many-out validation I have a set of errors along with the parameter value ...
3
votes
0answers
89 views

Internal validation via bootstrap: What ROC curve to present?

I am using the bootstrap approach for internal validation of a multivariate model built with either standard logistic regression OR elastic net. The procedure I use is as follows: 1) build model ...
1
vote
0answers
41 views

Comparing AUC vs accuracy

I understand this question has been asked many times however, i am unable to understand the answers well enough and apply to my situation. I have attached 2 screenshots of my model. There are 5 class ...
0
votes
0answers
158 views

Writing the objective function and constraints for scipy.optimize.minimize from matrices

I'm trying to find the optimal threshold point from a ROC curve. I have two main constraints : tpr >=80 and fpr <=60. I tried three main minimization functions : ...
1
vote
0answers
66 views

Confidence Intervals for AUC using cross-validation

I am analyzing the performance of a predictive model with the AUC, area under the ROC curve. I repeat several times cross-validation, and I have different estimations of the AUC in each folder. For ...
0
votes
1answer
31 views

Which quality measures are available for non-binary problems?

I'm wondering which quality measures are available for non-binary classifiers. I've read this article https://en.wikipedia.org/wiki/Receiver_operating_characteristic I understand, that the idea of ...
1
vote
1answer
53 views

Interpretation of the area under the PR curve

I'm currently comparing three methods and I have the Accuracy, auROC and auPR as metrics. And I have the following results : Method A - acc: 0.75, auROC: 0.75, auPR: 0.45 Method B - acc: 0.65, ...
0
votes
0answers
77 views

DeLong's Test to compare ROC curves

I am trying to compare the performance of a short and long form of a questionnaire. The long form has 25 items and the short form has 8 of the 25 items. Can DeLong's Test be used to compare the ROC ...
3
votes
2answers
173 views

Confused about sensitivity, specificity and area under ROC curve (AUC)

Just read a unpublished paper for review purpose. The reported results like Leave-one-out cross validation sensitivity is 95%. Leave-one-out cross validation specificity is 100%. Leave-one-out cross ...
0
votes
1answer
51 views

Drawing ROC plot for SDT in R

I am trying to plot an ideal ROC plot as predicted by Signal Detection Theory. Here are the formulas I try to plot: hi = Φ((d′−ci)/σ) (3) fi = Φ(−ci) (4)  ...
0
votes
0answers
87 views

Calibration curve in spss

I couldn't find any tutorials on this on youtube or otherwise. I am validating a clinical prediction model and I have a set of predicted outcomes and the real outcome. I have built a ROC curve but I ...
1
vote
0answers
47 views

Model validation for multilevel logistic regrssion?

I have designed a multilevel logistic model using PROC GLIMMIX in SAS 9.3 for hierachical data based on pupil attainment where level-two is the school the pupil attends. I'm quite sure that my model ...
0
votes
0answers
30 views

ROC curve and its function beginner

I have 3 features of a signal (example: amplitude, frequency, energy). I want to check which feature is the best to represent that particular signal. That signal is classified into two categories ...
1
vote
1answer
98 views

I want to learn about ROC curve — what is the canonical textbook?

I want to learn about Receiver-Operator-Characteristic curves, and metrics. I have read through online webpages with some basics, and I have used MATLAB built-ins to create ROC plots. It tells me ...
0
votes
0answers
34 views

How do the ROC cutoffs relate to predictors?

Apologies for this rather simple question, but I haven't been able to find a definition online. What does the ROC cutoffs represent for the AUC package? Specifically, how does it relate to the ...
2
votes
3answers
137 views

The value of adding the ROC graph if the AUC is given

I always see in papers that when they want to show how they classifiers performed, they provide ROC graph and the AUC score. Now as far as I know only the AUC tells how well the classifier performed, ...
3
votes
0answers
289 views

How to choose the cutoff probability for a rare event Logistic Regression

I have 100,000 observations (9 dummy indicator variables) with 1000 positives. Logistic Regression should work fine in this case but the cutoff probability puzzles me. In common literature, we ...
2
votes
2answers
141 views

Area under the ROC curve or area under the PR curve for imbalanced data?

I have some doubts about which performance measure to use, area under the ROC curve (TPR as a function of FPR) or area under the precision-recall curve (precision as a function of recall). My data is ...
5
votes
3answers
386 views

Why is AUC higher for a classifier that is less accurate than for one that is more accurate?

I have two classifiers A: naive Bayesian network B: tree (singly-connected) Bayesian network In terms of accuracy and other measures, A performs comparatively worse than B. However, when I use the ...
3
votes
1answer
144 views

Understanding random forest, gini, and KS

I'm a beginner machine learning user, doing my first predictive model using random forest. I have some questions regarding the way to measure how good a model is (Gini area from roc curve, and KS), ...
5
votes
2answers
157 views

How to do external validation of logistic regression models and perform model benchmarking

Quality assessment in trauma has for > 25 years been done with the US derived logistic regression model, the TRISS model. DV: survival/death and IVs: physiologic derangement (continuous), anatomic ...
0
votes
0answers
74 views

Random forest highly imbalanced dataset: how to test _ create ROC curve?

I have a dataset containing 10,000 examples with 8 features. I would like to create a random forest to classify this dataset in two classes, more specific: substrate / no substrate. In this dataset ...
1
vote
1answer
177 views

Why use d-prime instead of percent correct?

In signal detection theory, people often use $d'$ to assess performance. Apart from the fact that $d'$ is in $z$ units (units of measurement transformed to standard deviation units, i.e., $z$ scores), ...