Receiver Operating Characteristic, also known as the ROC curve.

learn more… | top users | synonyms

1
vote
0answers
20 views

Proportion of variance explained, prediction accuracy, and noisy criterion variables

A given criterion measure is corrupted by noise such that only 60% of its variance is attributable to a true signal. This noisy measure of outcome is regressed against some number of predictors and a ...
4
votes
1answer
29 views

Calculating Costs for ROC Curves

I am trying to calculate the optimal threshold for a binary classifier using Receiver operating characteristic (ROC) Curves. Currently I am assigning a cost for each false negative and another cost ...
0
votes
0answers
9 views

Bayes factors and ROC curves

The question comes from Kevin Murphy's book, Ch 5, Ex 5.6. Could somebody suggest a solution? Let $B=p(D|H_1)/p(D|H_0)$ be the bayes factor in favor of model 1. Suppose we plot two ROC curves, one ...
1
vote
0answers
15 views

Parametric versus nonparametric AUC from ROC curve

SPSS offers two ways to estimate Area Under the Curve (AUC) and its standard error, that is nonparametricly using trapezoidal rule or parametrically using binegative exponential distribution. I have ...
0
votes
1answer
22 views

How do you use a ROC curve to optimize a random forest classifier? [duplicate]

I am using a random forest for a binary classification problem using sklearn. The sklearn implementation outputs both a predicted class, and a probability for each class. The sklearn implementation ...
0
votes
0answers
16 views

Compare predictive power of 1 model on different data

I would like to test if the odds of bets on football matches with higher betting volume are more efficient (i.e. predict the result better) than bets with lower betting volume. I use a probit model ...
0
votes
0answers
12 views

Alter X axis labels on ROC curve [migrated]

I have performed a logistic regression and have plotted a ROC curve in R. I would like to alter the x and y axis labels however, am having trouble doing so. I have managed to suppress the previous ...
2
votes
1answer
43 views

How to compute the AUROC for a single categorical variable

I am building new features for a binary classifier. The new features fall into two categories: categorical and ordinal. An example of the first feature would be the colours ...
0
votes
0answers
28 views

Interpretation of a strange part in ROC curve

I generated the following ROC curve using Rapid Miner to compare few binary classifers but I don't know how to interpret the curve of "Random Forest" model It doesn't look like a curve
0
votes
1answer
16 views

Optimal cut-off point with minimum sensitivity

I have a logistic regression model. I'm looking for a non-graphical way to find the optimal cut-off where sensitivity is above a threshold(say 0.95) and maximizes sensitivity+specificity. I don't have ...
4
votes
1answer
103 views

outlier detection: area under precision recall curve

I would like to compare outlier detection algorithms. I am not sure if area under roc or under precision recall curve is the measure to use. A quick test in matlab gives me strange results. I try to ...
1
vote
1answer
70 views

The distribution of the AUC

I am wondering how the confidence interval for the Area under the Curve statistic (ROC curves) is derived. I have heard that the AUC can be assumed to be normally distributed, but I am looking for a ...
0
votes
0answers
9 views

Receiver operating characteristics curves in R - dichotomous predictors

I have performed a binary logistic regression using RStudio software with whether or not a player is re-contracted or not as the dependent variable. There are 7 predictor variables with 3 of these ...
1
vote
2answers
52 views

How much is ROC biased towards the minority class?

It's known that ROC is overly optimistic in case of imbalanced data sets. How big can this bias be? For example if I read paper where they report 0.75 ROC on a dataset with 5 percent of samples being ...
1
vote
0answers
19 views

Increasing PR curve?

Usually PR curves are decreasing (like as precision decreases, recall increases). How would you justify/interpret an increasing precision recall curve? The only way I can explain this is that the ...
0
votes
0answers
18 views

Equal Error Rate (EER) and Receiver Operating Characteristic (ROC) curve

I have a one-vs-all classifier set. This set consists of, let's say, 3 classifiers (LibSVM SVMs) each trained on data for a class and all other class data. The current setup for a sample is that the ...
0
votes
1answer
70 views

ROC Curve for different classifiers

I am trying to compare the classification performance of different classifiers. So far, I am using SVM, Random forest, Adaboost.M1, and Naive Bayes. 70% of data is used as training (and then plotting ...
1
vote
1answer
59 views

What is the best way to calculate the AUC of a ROC curve?

I have a ROC curve for which I'd like to calculate the AUC. I'm getting different values using the trapezoidal and rank-based approaches. What I'm noticing is that the two values actually add to 1.0 ...
0
votes
1answer
18 views

ROC plot not in one curve but scattered

I'm new to ensemble learning and am using three classifiers to identify anomalies using majority voting. I plotted the ROC of each classifier and then created an ensemble by varying the cutoff of each ...
3
votes
2answers
82 views

Do I do threshold selection for my logit model on the testing or training subset?

I have data with a binary outcome and I am doing logit model selection using AIC and BIC. I have already withheld 30% of the data as a holdout sample (testing subset) and used the remainder (training ...
0
votes
2answers
82 views

How to choose between ROC AUC and F1 score?

I recently completed a Kaggle competition in which roc auc score was used as per competition requirement. Before this project, I normally used f1 score as the metric to measure model performance. ...
2
votes
0answers
56 views

Interpret ROC/AUC values with respect data

I am using R to plot ROC curves. I have a prediction matrix, where each column shows the prediction values corresponding to different approaches. Also, I have a ...
2
votes
1answer
31 views

How to compare predictive power of sports odds

For my MSc Thesis I would like to compare the predictive power of classic bookmakers on the one hand and a betting exchange on the other. I have a lot of data on both types of betting, mostly on ...
2
votes
1answer
102 views

Comparing logistic regression models with AUC ROC in R vs Stata

I am fitting a logistic regression model for the likelihood of patients suffering morbidity after surgery. The most commonly used prediction tool at the moment is POSSUM (Physiological and Operative ...
0
votes
0answers
30 views

ROC calculation in LOOCV context - caret

I am not sure how caret handle the ROC calculation when used with LOOCV. From what I understand, in the more common case where a 10-fold cross validation is used, the ROC value is calculated for each ...
0
votes
1answer
95 views

Relationship between AUC and U Mann-Whitney statistic

Recently I learned about the relationship between Area Under (ROC) Curve and $U$ statistic of the Wilcoxon-Mann-Whitney test. It is supposed to follow the following rule (got it from this nice post on ...
0
votes
0answers
14 views

How is it correct to optimize a binary classifier output threshold with ROC and LPOCV?

Hello everyone and thank you in advance for you help! I'm building a screening tool with a machine learning algorithm. The model provides a probabilistic prediction (i.e. logistic regression, ...
0
votes
0answers
19 views

Scenario of how to compare models

I have a binary classification problem where the distribution of classes is skewed. I've already trained some scoring models with logistic regression. Now I would like to compare them. How to do this ...
0
votes
0answers
28 views

Calculating Efficiency in ROC analyses

I don't have a detailed understanding of statistics and need help. I'm working on determining cutoff scores for a brief measure using SPSS. Background: we administered one measure at three different ...
1
vote
1answer
81 views

100% training accuracy despite a low cv score

I am working on an assignment where we have to study the affect of gamma and C parameters on SVM with RBF kernel. I use python's sklearn library and grid search with 10 fold cross validation (with a ...
0
votes
0answers
31 views

Optimizing a model for a limited budget

I am building models to predict probability of failures against a list of approximately 500K assets. I want to optimize my models for maximum predictive performance on a fixed (limited) number of ...
1
vote
1answer
50 views

Optimal cut-off calculation in logistic regression

I am building a logistic regression model and am using k-fold cross-validation for model selection. My doubt is with regards to misclassification rate. For measuring that, i will have to first find ...
4
votes
2answers
143 views

“Good” classifier destroyed my Precision-Recall curve. What happened?

I'm working with imbalanced data, where there are about 40 class=0 cases for every class=1. I can reasonably discriminate between the classes using individual features, and training a naive Bayes and ...
0
votes
0answers
10 views

Cross-Validation for single feature with two labels

In systems neuroscience it is common to report the area under ROC when comparing signals between two different outcomes (two labels). For example you can compare the number of spikes (one feature) ...
0
votes
0answers
24 views

How do I judge low probability model quality?

I'm modeling loan defaults - an unlikely occurrence. My logistic regression model predicts probabilities ranging from 0.001% (squeaky clean) to 44% (hinky). Each specific prediction is "no, this ...
0
votes
0answers
25 views

AUROC for SVM two class classification [duplicate]

I'm trying to compute the ROC and AUROC of a binary svm classification. I already looked up a code in the internet and it's working: ...
0
votes
0answers
56 views

Accuracy Ratio Brute-force vs Logistic Regression

We want to model a binary dependent variable $Y$ with 0 or 1 values (e.g. whether a loan defaults or not) based on 3 independent variables $X_1$, $X_2$ and $X_3$. I have the following 2 methods and I ...
0
votes
0answers
16 views

Signal Detection Theory, How do I create a ROC curve from 2 distributions?

I have 2 samples and I have to decide using the Signal Detection Theory which one is the true distribution. This is my data: ...
0
votes
1answer
59 views

Deep Neural Networks: AUCROC Values Consistently = 0.5 even though RMS Error on Test Set ~10%

I am new to Neural Networks and but I have built a multi-classifier using the FANN neural network package. My multi-classifier, regardless of the network hyperparameters, consistently gives an error ...
0
votes
0answers
16 views

How to plot ROC for LOOCV by hand based on output probabilities returned by SVM? [duplicate]

I want to plot ROC for Leave-one-out (LOO) cross-validation (CV). I have the decision values as well as probability values for each class from SVM classifier. I did see the answer to this post; ...
1
vote
0answers
28 views

Can I draw a single ROC curve for a test method applied onto a group of people with different levels (i.e., light, mild,sever) of sickness

Can I draw a single ROC curve for a test method applied onto a group of people of different levels of sickness severity (i.e., light, mild, severe)? Or I need to draw three different ROC curves, one ...
3
votes
1answer
85 views

AUC values for different sets of features

I have a dataset with two groups of features, set 1 and set2. I have traind and tested SVM classifiers in three different settings: 1) only set 1 features, 2) only set 2 features, and 3) union of set ...
1
vote
1answer
89 views

Why compare AUC's in binary classification?

I understand that a common metric for comparing binary classifiers is the AUC of the ROC curve. But, after this is computed, only one threshold is actually chosen for classifying negative and ...
1
vote
0answers
48 views

AUC for binary ROC curve

I am using the ROCR package in R to calculate ROC and associated AUC for an arbitrary continuous data set with labels coded as 0 or 1. In case A, I have some set of labels for each entry in the data ...
0
votes
1answer
50 views

Performance of a classifier change heavily

I'm using a data set of 32 face persons and a svm-rbf to classify and a random group of 70% for train and 30% for test. The problem is that my results are heavily dependent of the random group used ...
0
votes
0answers
41 views

Reporting AUC on training or testing data

I have a really simple question. I am writing an article to submit to a conference. I have used SVM classifier in it. I have seen in many papers which report ROC and AUC for their classifiers, and I ...
0
votes
0answers
32 views

ROC change after variable selection with glmnet

I was using glmnet in caret to select important variables. The code is like ...
0
votes
2answers
100 views

R Caret - Repeated CrossValidation, finalModel and ROC curves

I got a problem understanding the meaning of the finalModel when using a repeated CV. ...
0
votes
0answers
15 views

Were there a diagnostic value if ROC analysis showed a high AUC but a low specificity?

Were there a diagnostic value if ROC analysis showed a high AUC but a low specificity? for example, spcificity below 0.7.