Receiver Operating Characteristic, also known as the ROC curve.

learn more… | top users | synonyms

4
votes
2answers
44 views

Accuracy vs. area under the ROC curve

I constructed an ROC curve for a diagnostic system. The area under the curve was then non-parametrically estimated to be AUC = 0.89. When I tried to calculate the accuracy at the optimum threshold ...
0
votes
0answers
13 views

should i normalize for size? using roc-curve to compare classifiers

background: I plan to use a roc-curve to compare 4 classifiers. My data is made up of 2000 family members (x1, x2, ... x2000) and each member is part of a family of a certain size (y1,y2...y215). ...
0
votes
0answers
18 views

Support Vector Classifier - Opposite Classification

I'm using a support vector machine (SVC from the SKLearn library in Python) to classify some time series data, aiming to predict the severity of injury to babies. A rough summary of the method is ...
0
votes
0answers
14 views

Macro-averaged Specificity

I have read the paper "A systematic analysis of performance measures for classification tasks" in which different statistics for binary classification as well as their extension to the multi-class ...
0
votes
1answer
20 views

ROC of multi-class classification from confusion matrix only

I've been looking into using ROC curves as a evaluation tool of a multi-class classification. The only data I have about this classification is in form of 7-by-7 confusion matrix. Visualisation of ...
1
vote
1answer
45 views

Calculate true positive rate (TPR) and false positive rate (FPR) from prediction values to form ROC curve

I have the prediction values of different algorithms, as shown below. My question is: how can I get the FPR and TPR to form a ROC curve? If I compute the confusion matrix I am getting only one point, ...
0
votes
1answer
31 views

Calculate EER from FAR and FRR?

I'm wondering if we have FAR and FRR scores for each threshold if we can compute an EER programatically? Say we have: ...
0
votes
1answer
30 views

How to calculate average EER and ROC?

If I test a system on 3 users and I obtain FARs and FRRs for thresholds and an EER for each user, how do I obtain an average EER and plot a ROC? As I see it, I could average the 3 EERs I have ...
0
votes
1answer
37 views

How to compare the results of two hierarchical classification results?

How could I compare two classification results from two implementations of e.g. using different methods/features for pairwise similarity calculation? If one method classified e.g. 3000 data points ...
1
vote
0answers
25 views

Proportion of variance explained, prediction accuracy, and noisy criterion variables

A given criterion measure is corrupted by noise such that only 60% of its variance is attributable to a true signal. This noisy measure of outcome is regressed against some number of predictors and a ...
4
votes
1answer
32 views

Calculating Costs for ROC Curves

I am trying to calculate the optimal threshold for a binary classifier using Receiver operating characteristic (ROC) Curves. Currently I am assigning a cost for each false negative and another cost ...
0
votes
0answers
9 views

Bayes factors and ROC curves

The question comes from Kevin Murphy's book, Ch 5, Ex 5.6. Could somebody suggest a solution? Let $B=p(D|H_1)/p(D|H_0)$ be the bayes factor in favor of model 1. Suppose we plot two ROC curves, one ...
1
vote
0answers
18 views

Parametric versus nonparametric AUC from ROC curve

SPSS offers two ways to estimate Area Under the Curve (AUC) and its standard error, that is nonparametricly using trapezoidal rule or parametrically using binegative exponential distribution. I have ...
0
votes
1answer
33 views

How do you use a ROC curve to optimize a random forest classifier? [duplicate]

I am using a random forest for a binary classification problem using sklearn. The sklearn implementation outputs both a predicted class, and a probability for each class. The sklearn implementation ...
0
votes
0answers
18 views

Compare predictive power of 1 model on different data

I would like to test if the odds of bets on football matches with higher betting volume are more efficient (i.e. predict the result better) than bets with lower betting volume. I use a probit model ...
2
votes
1answer
54 views

How to compute the AUROC for a single categorical variable

I am building new features for a binary classifier. The new features fall into two categories: categorical and ordinal. An example of the first feature would be the colours ...
0
votes
0answers
29 views

Interpretation of a strange part in ROC curve

I generated the following ROC curve using Rapid Miner to compare few binary classifers but I don't know how to interpret the curve of "Random Forest" model It doesn't look like a curve
0
votes
1answer
17 views

Optimal cut-off point with minimum sensitivity

I have a logistic regression model. I'm looking for a non-graphical way to find the optimal cut-off where sensitivity is above a threshold(say 0.95) and maximizes sensitivity+specificity. I don't have ...
4
votes
1answer
110 views

outlier detection: area under precision recall curve

I would like to compare outlier detection algorithms. I am not sure if area under roc or under precision recall curve is the measure to use. A quick test in matlab gives me strange results. I try to ...
1
vote
1answer
83 views

The distribution of the AUC

I am wondering how the confidence interval for the Area under the Curve statistic (ROC curves) is derived. I have heard that the AUC can be assumed to be normally distributed, but I am looking for a ...
0
votes
0answers
12 views

Receiver operating characteristics curves in R - dichotomous predictors

I have performed a binary logistic regression using RStudio software with whether or not a player is re-contracted or not as the dependent variable. There are 7 predictor variables with 3 of these ...
1
vote
2answers
54 views

How much is ROC biased towards the minority class?

It's known that ROC is overly optimistic in case of imbalanced data sets. How big can this bias be? For example if I read paper where they report 0.75 ROC on a dataset with 5 percent of samples being ...
1
vote
0answers
19 views

Increasing PR curve?

Usually PR curves are decreasing (like as precision decreases, recall increases). How would you justify/interpret an increasing precision recall curve? The only way I can explain this is that the ...
0
votes
0answers
37 views

Equal Error Rate (EER) and Receiver Operating Characteristic (ROC) curve

I have a one-vs-all classifier set. This set consists of, let's say, 3 classifiers (LibSVM SVMs) each trained on data for a class and all other class data. The current setup for a sample is that the ...
0
votes
1answer
80 views

ROC Curve for different classifiers

I am trying to compare the classification performance of different classifiers. So far, I am using SVM, Random forest, Adaboost.M1, and Naive Bayes. 70% of data is used as training (and then plotting ...
1
vote
1answer
65 views

What is the best way to calculate the AUC of a ROC curve?

I have a ROC curve for which I'd like to calculate the AUC. I'm getting different values using the trapezoidal and rank-based approaches. What I'm noticing is that the two values actually add to 1.0 ...
0
votes
1answer
20 views

ROC plot not in one curve but scattered

I'm new to ensemble learning and am using three classifiers to identify anomalies using majority voting. I plotted the ROC of each classifier and then created an ensemble by varying the cutoff of each ...
3
votes
2answers
88 views

Do I do threshold selection for my logit model on the testing or training subset?

I have data with a binary outcome and I am doing logit model selection using AIC and BIC. I have already withheld 30% of the data as a holdout sample (testing subset) and used the remainder (training ...
0
votes
2answers
97 views

How to choose between ROC AUC and F1 score?

I recently completed a Kaggle competition in which roc auc score was used as per competition requirement. Before this project, I normally used f1 score as the metric to measure model performance. ...
2
votes
0answers
59 views

Interpret ROC/AUC values with respect data

I am using R to plot ROC curves. I have a prediction matrix, where each column shows the prediction values corresponding to different approaches. Also, I have a ...
2
votes
1answer
33 views

How to compare predictive power of sports odds

For my MSc Thesis I would like to compare the predictive power of classic bookmakers on the one hand and a betting exchange on the other. I have a lot of data on both types of betting, mostly on ...
2
votes
1answer
118 views

Comparing logistic regression models with AUC ROC in R vs Stata

I am fitting a logistic regression model for the likelihood of patients suffering morbidity after surgery. The most commonly used prediction tool at the moment is POSSUM (Physiological and Operative ...
0
votes
0answers
31 views

ROC calculation in LOOCV context - caret

I am not sure how caret handle the ROC calculation when used with LOOCV. From what I understand, in the more common case where a 10-fold cross validation is used, the ROC value is calculated for each ...
0
votes
1answer
131 views

Relationship between AUC and U Mann-Whitney statistic

Recently I learned about the relationship between Area Under (ROC) Curve and $U$ statistic of the Wilcoxon-Mann-Whitney test. It is supposed to follow the following rule (got it from this nice post on ...
0
votes
0answers
20 views

How is it correct to optimize a binary classifier output threshold with ROC and LPOCV?

Hello everyone and thank you in advance for you help! I'm building a screening tool with a machine learning algorithm. The model provides a probabilistic prediction (i.e. logistic regression, ...
0
votes
0answers
21 views

Scenario of how to compare models

I have a binary classification problem where the distribution of classes is skewed. I've already trained some scoring models with logistic regression. Now I would like to compare them. How to do this ...
0
votes
0answers
32 views

Calculating Efficiency in ROC analyses

I don't have a detailed understanding of statistics and need help. I'm working on determining cutoff scores for a brief measure using SPSS. Background: we administered one measure at three different ...
1
vote
1answer
87 views

100% training accuracy despite a low cv score

I am working on an assignment where we have to study the affect of gamma and C parameters on SVM with RBF kernel. I use python's sklearn library and grid search with 10 fold cross validation (with a ...
0
votes
0answers
33 views

Optimizing a model for a limited budget

I am building models to predict probability of failures against a list of approximately 500K assets. I want to optimize my models for maximum predictive performance on a fixed (limited) number of ...
1
vote
1answer
54 views

Optimal cut-off calculation in logistic regression

I am building a logistic regression model and am using k-fold cross-validation for model selection. My doubt is with regards to misclassification rate. For measuring that, i will have to first find ...
4
votes
2answers
182 views

“Good” classifier destroyed my Precision-Recall curve. What happened?

I'm working with imbalanced data, where there are about 40 class=0 cases for every class=1. I can reasonably discriminate between the classes using individual features, and training a naive Bayes and ...
0
votes
0answers
10 views

Cross-Validation for single feature with two labels

In systems neuroscience it is common to report the area under ROC when comparing signals between two different outcomes (two labels). For example you can compare the number of spikes (one feature) ...
0
votes
0answers
24 views

How do I judge low probability model quality?

I'm modeling loan defaults - an unlikely occurrence. My logistic regression model predicts probabilities ranging from 0.001% (squeaky clean) to 44% (hinky). Each specific prediction is "no, this ...
0
votes
0answers
27 views

AUROC for SVM two class classification [duplicate]

I'm trying to compute the ROC and AUROC of a binary svm classification. I already looked up a code in the internet and it's working: ...
0
votes
0answers
60 views

Accuracy Ratio Brute-force vs Logistic Regression

We want to model a binary dependent variable $Y$ with 0 or 1 values (e.g. whether a loan defaults or not) based on 3 independent variables $X_1$, $X_2$ and $X_3$. I have the following 2 methods and I ...
0
votes
0answers
17 views

Signal Detection Theory, How do I create a ROC curve from 2 distributions?

I have 2 samples and I have to decide using the Signal Detection Theory which one is the true distribution. This is my data: ...
0
votes
1answer
64 views

Deep Neural Networks: AUCROC Values Consistently = 0.5 even though RMS Error on Test Set ~10%

I am new to Neural Networks and but I have built a multi-classifier using the FANN neural network package. My multi-classifier, regardless of the network hyperparameters, consistently gives an error ...
0
votes
0answers
17 views

How to plot ROC for LOOCV by hand based on output probabilities returned by SVM? [duplicate]

I want to plot ROC for Leave-one-out (LOO) cross-validation (CV). I have the decision values as well as probability values for each class from SVM classifier. I did see the answer to this post; ...
1
vote
0answers
28 views

Can I draw a single ROC curve for a test method applied onto a group of people with different levels (i.e., light, mild,sever) of sickness

Can I draw a single ROC curve for a test method applied onto a group of people of different levels of sickness severity (i.e., light, mild, severe)? Or I need to draw three different ROC curves, one ...
3
votes
1answer
87 views

AUC values for different sets of features

I have a dataset with two groups of features, set 1 and set2. I have traind and tested SVM classifiers in three different settings: 1) only set 1 features, 2) only set 2 features, and 3) union of set ...