Receiver Operating Characteristic, also known as ROC curve.

learn more… | top users | synonyms

0
votes
0answers
4 views

Equal Error Rate (EER) and Receiver Operating Characteristic (ROC) curve

I have a one-vs-all classifier set. This set consists of, let's say, 3 classifiers (LibSVM SVMs) each trained on data for a class and all other class data. The current setup for a sample is that the ...
0
votes
0answers
21 views

ROCR package… what am I not getting? [migrated]

I am testing a simple case using ROCR package in R. Basically, here is my code. I have a set of true values, and for each value, I have a set of predictions, and my labels are 1 if the prediction is ...
0
votes
0answers
5 views

random selection of raters [closed]

I have a group of 11 raters that I am comparing to a gold standard. I would like to know the sensitivity and specificity if 5 (also 7 and 9) of those raters are randomly selected. I could do this ...
0
votes
1answer
57 views

ROC Curve for different classifiers

I am trying to compare the classification performance of different classifiers. So far, I am using SVM, Random forest, Adaboost.M1, and Naive Bayes. 70% of data is used as training (and then plotting ...
1
vote
1answer
53 views

What is the best way to calculate the AUC of a ROC curve?

I have a ROC curve for which I'd like to calculate the AUC. I'm getting different values using the trapezoidal and rank-based approaches. What I'm noticing is that the two values actually add to 1.0 ...
0
votes
1answer
18 views

ROC plot not in one curve but scattered

I'm new to ensemble learning and am using three classifiers to identify anomalies using majority voting. I plotted the ROC of each classifier and then created an ensemble by varying the cutoff of each ...
3
votes
2answers
71 views

Do I do threshold selection for my logit model on the testing or training subset?

I have data with a binary outcome and I am doing logit model selection using AIC and BIC. I have already withheld 30% of the data as a holdout sample (testing subset) and used the remainder (training ...
0
votes
2answers
71 views

How to choose between ROC AUC and F1 score?

I recently completed a Kaggle competition in which roc auc score was used as per competition requirement. Before this project, I normally used f1 score as the metric to measure model performance. ...
2
votes
0answers
47 views

Interpret ROC/AUC values with respect data

I am using R to plot ROC curves. I have a prediction matrix, where each column shows the prediction values corresponding to different approaches. Also, I have a ...
2
votes
1answer
19 views

How to compare predictive power of sports odds

For my MSc Thesis I would like to compare the predictive power of classic bookmakers on the one hand and a betting exchange on the other. I have a lot of data on both types of betting, mostly on ...
2
votes
1answer
84 views

Comparing logistic regression models with AUC ROC in R vs Stata

I am fitting a logistic regression model for the likelihood of patients suffering morbidity after surgery. The most commonly used prediction tool at the moment is POSSUM (Physiological and Operative ...
0
votes
0answers
27 views

ROC calculation in LOOCV context - caret

I am not sure how caret handle the ROC calculation when used with LOOCV. From what I understand, in the more common case where a 10-fold cross validation is used, the ROC value is calculated for each ...
0
votes
1answer
49 views

Relationship between AUC and U Mann-Whitney statistic

Recently I learned about the relationship between Area Under (ROC) Curve and $U$ statistic of the Wilcoxon-Mann-Whitney test. It is supposed to follow the following rule (got it from this nice post on ...
0
votes
0answers
12 views

How is it correct to optimize a binary classifier output threshold with ROC and LPOCV?

Hello everyone and thank you in advance for you help! I'm building a screening tool with a machine learning algorithm. The model provides a probabilistic prediction (i.e. logistic regression, ...
0
votes
0answers
18 views

Scenario of how to compare models

I have a binary classification problem where the distribution of classes is skewed. I've already trained some scoring models with logistic regression. Now I would like to compare them. How to do this ...
0
votes
0answers
24 views

Calculating Efficiency in ROC analyses

I don't have a detailed understanding of statistics and need help. I'm working on determining cutoff scores for a brief measure using SPSS. Background: we administered one measure at three different ...
1
vote
1answer
71 views

100% training accuracy despite a low cv score

I am working on an assignment where we have to study the affect of gamma and C parameters on SVM with RBF kernel. I use python's sklearn library and grid search with 10 fold cross validation (with a ...
1
vote
0answers
25 views

AUC for ROC on predictors

During preprocessing how good is AUC ROC used on predictors as a way to eliminate predictors? What size of AUC is too low to be considered useful?
0
votes
0answers
27 views

Optimizing a model for a limited budget

I am building models to predict probability of failures against a list of approximately 500K assets. I want to optimize my models for maximum predictive performance on a fixed (limited) number of ...
1
vote
1answer
43 views

Optimal cut-off calculation in logistic regression

I am building a logistic regression model and am using k-fold cross-validation for model selection. My doubt is with regards to misclassification rate. For measuring that, i will have to first find ...
4
votes
2answers
103 views

“Good” classifier destroyed my Precision-Recall curve. What happened?

I'm working with imbalanced data, where there are about 40 class=0 cases for every class=1. I can reasonably discriminate between the classes using individual features, and training a naive Bayes and ...
0
votes
0answers
10 views

Cross-Validation for single feature with two labels

In systems neuroscience it is common to report the area under ROC when comparing signals between two different outcomes (two labels). For example you can compare the number of spikes (one feature) ...
0
votes
0answers
23 views

How do I judge low probability model quality?

I'm modeling loan defaults - an unlikely occurrence. My logistic regression model predicts probabilities ranging from 0.001% (squeaky clean) to 44% (hinky). Each specific prediction is "no, this ...
0
votes
0answers
25 views

AUROC for SVM two class classification [duplicate]

I'm trying to compute the ROC and AUROC of a binary svm classification. I already looked up a code in the internet and it's working: ...
0
votes
0answers
51 views

Accuracy Ratio Brute-force vs Logistic Regression

We want to model a binary dependent variable $Y$ with 0 or 1 values (e.g. whether a loan defaults or not) based on 3 independent variables $X_1$, $X_2$ and $X_3$. I have the following 2 methods and I ...
0
votes
0answers
15 views

Signal Detection Theory, How do I create a ROC curve from 2 distributions?

I have 2 samples and I have to decide using the Signal Detection Theory which one is the true distribution. This is my data: ...
0
votes
1answer
50 views

Deep Neural Networks: AUCROC Values Consistently = 0.5 even though RMS Error on Test Set ~10%

I am new to Neural Networks and but I have built a multi-classifier using the FANN neural network package. My multi-classifier, regardless of the network hyperparameters, consistently gives an error ...
0
votes
0answers
16 views

How to plot ROC for LOOCV by hand based on output probabilities returned by SVM? [duplicate]

I want to plot ROC for Leave-one-out (LOO) cross-validation (CV). I have the decision values as well as probability values for each class from SVM classifier. I did see the answer to this post; ...
1
vote
0answers
27 views

Can I draw a single ROC curve for a test method applied onto a group of people with different levels (i.e., light, mild,sever) of sickness

Can I draw a single ROC curve for a test method applied onto a group of people of different levels of sickness severity (i.e., light, mild, severe)? Or I need to draw three different ROC curves, one ...
3
votes
1answer
80 views

AUC values for different sets of features

I have a dataset with two groups of features, set 1 and set2. I have traind and tested SVM classifiers in three different settings: 1) only set 1 features, 2) only set 2 features, and 3) union of set ...
1
vote
1answer
79 views

Why compare AUC's in binary classification?

I understand that a common metric for comparing binary classifiers is the AUC of the ROC curve. But, after this is computed, only one threshold is actually chosen for classifying negative and ...
1
vote
0answers
43 views

AUC for binary ROC curve

I am using the ROCR package in R to calculate ROC and associated AUC for an arbitrary continuous data set with labels coded as 0 or 1. In case A, I have some set of labels for each entry in the data ...
0
votes
1answer
48 views

Performance of a classifier change heavily

I'm using a data set of 32 face persons and a svm-rbf to classify and a random group of 70% for train and 30% for test. The problem is that my results are heavily dependent of the random group used ...
0
votes
0answers
29 views

Reporting AUC on training or testing data

I have a really simple question. I am writing an article to submit to a conference. I have used SVM classifier in it. I have seen in many papers which report ROC and AUC for their classifiers, and I ...
0
votes
0answers
29 views

ROC change after variable selection with glmnet

I was using glmnet in caret to select important variables. The code is like ...
0
votes
2answers
84 views

R Caret - Repeated CrossValidation, finalModel and ROC curves

I got a problem understanding the meaning of the finalModel when using a repeated CV. ...
0
votes
0answers
14 views

Were there a diagnostic value if ROC analysis showed a high AUC but a low specificity?

Were there a diagnostic value if ROC analysis showed a high AUC but a low specificity? for example, spcificity below 0.7.
1
vote
1answer
61 views

Does dice coefficient same as accuracy?

I come across dice coefficient for volume similarity (https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient) and accuracy (https://en.wikipedia.org/wiki/Accuracy_and_precision). It ...
1
vote
0answers
31 views

How to plot ROC for knn (and potentially kernel spectral regression)

I understand how to plot ROC for logistic classifier (like varies the probability cutoff). For KNN, how can I find the ROC? Also, what about kernel spectral regression?
0
votes
0answers
13 views

Can two Classifier with different number of False Positives have exactly same ROC curve?

There are two classifiers, the first one return Scores ...
0
votes
0answers
33 views

The meaning of “training accuracy”?

If I split my data set into testing, training (further separated into subtraining and validation data set in cross-validation). In the context of machine learning and esp. in those ROC comparing the ...
3
votes
1answer
84 views

ROC curve drawbacks

In the class yesterday, we were taught about logistic and subsequently the ROC curve and how to use it. My questions are: Is this the most common way to identify if the logistic model is the ...
1
vote
0answers
25 views

Equivalence of dominant classifier performance in ROC and PR spaces

In their article of 2006, The Relationship Between Precision-Recall and ROC Curves, J. Davis & M. Goadrich show that ROC (TPR vs FPR) and Precision-Recall (PR) curves have one-to-one ...
0
votes
0answers
14 views

Minimum sample size to establish a cutoff point

I conducted a case-control study (obese and normal body weight). I want to conduct ROC curve between my variables to establish a cutoff point for vitamin depending on insulin resistance risk. Does it ...
0
votes
1answer
38 views

Performance evaluation measures for binary random forest classifier

I am using Random forest (Matlab) to classify the binary data. Broadly, the input to the random forest is number of features and class label. And random forest, after training, return the labels for ...
0
votes
0answers
164 views

Leave-one-out cross validation output interpretation and ROC curve

I have taken plenty of time to try and help myself, but I keep reaching dead ends. I have a dataset consisting of body measurements collected from a bird species, and the sex of each bird (known by ...
0
votes
1answer
533 views

R - Plotting a ROC curve for a Naive Bayes classifier using ROCR. Not sure if I'm plotting it correctly

I have a Naive Bayes classifiers that I'm using to try to predict whether a game is going to win or lose based on historical data. The model has 25 variables in total, all of which are categorical ...
1
vote
0answers
72 views

A strange ROC curve

I produced the ROC curve for a continuous biomarker used as a classifier toward a binary class (absence/presence of a disease according to another biomarker), using the ROCR package in R. I have a ...
21
votes
4answers
3k views

What is the name of this chart showing false and true positive rates and how is it generated?

The image below shows a continuous curve of false positive rates vs. true positive rates: However, what I don't immediately get is how these rates are being calculated. If a method is applied to a ...
3
votes
2answers
113 views

Diagonal in ROC plot?

I've used the R ROCR package to generate an ROC plot. I can see a diagonal in my plot. However, the ROC curve that I see other people generated has only horizontal and vertical lines. Why would I get ...