# Tagged Questions

Receiver Operating Characteristic, also known as the ROC curve.

44 views

### Accuracy vs. area under the ROC curve

I constructed an ROC curve for a diagnostic system. The area under the curve was then non-parametrically estimated to be AUC = 0.89. When I tried to calculate the accuracy at the optimum threshold ...
13 views

### should i normalize for size? using roc-curve to compare classifiers

background: I plan to use a roc-curve to compare 4 classifiers. My data is made up of 2000 family members (x1, x2, ... x2000) and each member is part of a family of a certain size (y1,y2...y215). ...
18 views

### Support Vector Classifier - Opposite Classification

I'm using a support vector machine (SVC from the SKLearn library in Python) to classify some time series data, aiming to predict the severity of injury to babies. A rough summary of the method is ...
14 views

### Macro-averaged Specificity

I have read the paper "A systematic analysis of performance measures for classification tasks" in which different statistics for binary classification as well as their extension to the multi-class ...
20 views

### ROC of multi-class classification from confusion matrix only

I've been looking into using ROC curves as a evaluation tool of a multi-class classification. The only data I have about this classification is in form of 7-by-7 confusion matrix. Visualisation of ...
45 views

### Calculate true positive rate (TPR) and false positive rate (FPR) from prediction values to form ROC curve

I have the prediction values of different algorithms, as shown below. My question is: how can I get the FPR and TPR to form a ROC curve? If I compute the confusion matrix I am getting only one point, ...
31 views

### Calculate EER from FAR and FRR?

I'm wondering if we have FAR and FRR scores for each threshold if we can compute an EER programatically? Say we have: ...
30 views

### How to calculate average EER and ROC?

If I test a system on 3 users and I obtain FARs and FRRs for thresholds and an EER for each user, how do I obtain an average EER and plot a ROC? As I see it, I could average the 3 EERs I have ...
37 views

### How to compare the results of two hierarchical classification results?

How could I compare two classification results from two implementations of e.g. using different methods/features for pairwise similarity calculation? If one method classified e.g. 3000 data points ...
25 views

### Proportion of variance explained, prediction accuracy, and noisy criterion variables

A given criterion measure is corrupted by noise such that only 60% of its variance is attributable to a true signal. This noisy measure of outcome is regressed against some number of predictors and a ...
32 views

### Calculating Costs for ROC Curves

I am trying to calculate the optimal threshold for a binary classifier using Receiver operating characteristic (ROC) Curves. Currently I am assigning a cost for each false negative and another cost ...
9 views

### Bayes factors and ROC curves

The question comes from Kevin Murphy's book, Ch 5, Ex 5.6. Could somebody suggest a solution? Let $B=p(D|H_1)/p(D|H_0)$ be the bayes factor in favor of model 1. Suppose we plot two ROC curves, one ...
18 views

### Parametric versus nonparametric AUC from ROC curve

SPSS offers two ways to estimate Area Under the Curve (AUC) and its standard error, that is nonparametricly using trapezoidal rule or parametrically using binegative exponential distribution. I have ...
33 views

### How do you use a ROC curve to optimize a random forest classifier? [duplicate]

I am using a random forest for a binary classification problem using sklearn. The sklearn implementation outputs both a predicted class, and a probability for each class. The sklearn implementation ...
18 views

### Compare predictive power of 1 model on different data

I would like to test if the odds of bets on football matches with higher betting volume are more efficient (i.e. predict the result better) than bets with lower betting volume. I use a probit model ...
54 views

### How to compute the AUROC for a single categorical variable

I am building new features for a binary classifier. The new features fall into two categories: categorical and ordinal. An example of the first feature would be the colours ...
29 views

### Interpretation of a strange part in ROC curve

I generated the following ROC curve using Rapid Miner to compare few binary classifers but I don't know how to interpret the curve of "Random Forest" model It doesn't look like a curve
17 views

### Optimal cut-off point with minimum sensitivity

I have a logistic regression model. I'm looking for a non-graphical way to find the optimal cut-off where sensitivity is above a threshold(say 0.95) and maximizes sensitivity+specificity. I don't have ...
110 views

### outlier detection: area under precision recall curve

I would like to compare outlier detection algorithms. I am not sure if area under roc or under precision recall curve is the measure to use. A quick test in matlab gives me strange results. I try to ...
83 views

### The distribution of the AUC

I am wondering how the confidence interval for the Area under the Curve statistic (ROC curves) is derived. I have heard that the AUC can be assumed to be normally distributed, but I am looking for a ...
12 views

### Receiver operating characteristics curves in R - dichotomous predictors

I have performed a binary logistic regression using RStudio software with whether or not a player is re-contracted or not as the dependent variable. There are 7 predictor variables with 3 of these ...
54 views

### How much is ROC biased towards the minority class?

It's known that ROC is overly optimistic in case of imbalanced data sets. How big can this bias be? For example if I read paper where they report 0.75 ROC on a dataset with 5 percent of samples being ...
19 views

### Increasing PR curve?

Usually PR curves are decreasing (like as precision decreases, recall increases). How would you justify/interpret an increasing precision recall curve? The only way I can explain this is that the ...
37 views

### Equal Error Rate (EER) and Receiver Operating Characteristic (ROC) curve

I have a one-vs-all classifier set. This set consists of, let's say, 3 classifiers (LibSVM SVMs) each trained on data for a class and all other class data. The current setup for a sample is that the ...
80 views

### ROC Curve for different classifiers

I am trying to compare the classification performance of different classifiers. So far, I am using SVM, Random forest, Adaboost.M1, and Naive Bayes. 70% of data is used as training (and then plotting ...
65 views

### What is the best way to calculate the AUC of a ROC curve?

I have a ROC curve for which I'd like to calculate the AUC. I'm getting different values using the trapezoidal and rank-based approaches. What I'm noticing is that the two values actually add to 1.0 ...
20 views

### ROC plot not in one curve but scattered

I'm new to ensemble learning and am using three classifiers to identify anomalies using majority voting. I plotted the ROC of each classifier and then created an ensemble by varying the cutoff of each ...
88 views

### Do I do threshold selection for my logit model on the testing or training subset?

I have data with a binary outcome and I am doing logit model selection using AIC and BIC. I have already withheld 30% of the data as a holdout sample (testing subset) and used the remainder (training ...
97 views

### How to choose between ROC AUC and F1 score?

I recently completed a Kaggle competition in which roc auc score was used as per competition requirement. Before this project, I normally used f1 score as the metric to measure model performance. ...
59 views

### Interpret ROC/AUC values with respect data

I am using R to plot ROC curves. I have a prediction matrix, where each column shows the prediction values corresponding to different approaches. Also, I have a ...
33 views

### How to compare predictive power of sports odds

For my MSc Thesis I would like to compare the predictive power of classic bookmakers on the one hand and a betting exchange on the other. I have a lot of data on both types of betting, mostly on ...
118 views

### Comparing logistic regression models with AUC ROC in R vs Stata

I am fitting a logistic regression model for the likelihood of patients suffering morbidity after surgery. The most commonly used prediction tool at the moment is POSSUM (Physiological and Operative ...
31 views

### ROC calculation in LOOCV context - caret

I am not sure how caret handle the ROC calculation when used with LOOCV. From what I understand, in the more common case where a 10-fold cross validation is used, the ROC value is calculated for each ...
131 views

### Relationship between AUC and U Mann-Whitney statistic

Recently I learned about the relationship between Area Under (ROC) Curve and $U$ statistic of the Wilcoxon-Mann-Whitney test. It is supposed to follow the following rule (got it from this nice post on ...
20 views

### How is it correct to optimize a binary classifier output threshold with ROC and LPOCV?

Hello everyone and thank you in advance for you help! I'm building a screening tool with a machine learning algorithm. The model provides a probabilistic prediction (i.e. logistic regression, ...
21 views

### Scenario of how to compare models

I have a binary classification problem where the distribution of classes is skewed. I've already trained some scoring models with logistic regression. Now I would like to compare them. How to do this ...
32 views

### Calculating Efficiency in ROC analyses

I don't have a detailed understanding of statistics and need help. I'm working on determining cutoff scores for a brief measure using SPSS. Background: we administered one measure at three different ...
87 views

### 100% training accuracy despite a low cv score

I am working on an assignment where we have to study the affect of gamma and C parameters on SVM with RBF kernel. I use python's sklearn library and grid search with 10 fold cross validation (with a ...
33 views

### Optimizing a model for a limited budget

I am building models to predict probability of failures against a list of approximately 500K assets. I want to optimize my models for maximum predictive performance on a fixed (limited) number of ...
54 views

### Optimal cut-off calculation in logistic regression

I am building a logistic regression model and am using k-fold cross-validation for model selection. My doubt is with regards to misclassification rate. For measuring that, i will have to first find ...
182 views

### “Good” classifier destroyed my Precision-Recall curve. What happened?

I'm working with imbalanced data, where there are about 40 class=0 cases for every class=1. I can reasonably discriminate between the classes using individual features, and training a naive Bayes and ...
10 views

### Cross-Validation for single feature with two labels

In systems neuroscience it is common to report the area under ROC when comparing signals between two different outcomes (two labels). For example you can compare the number of spikes (one feature) ...
24 views

### How do I judge low probability model quality?

I'm modeling loan defaults - an unlikely occurrence. My logistic regression model predicts probabilities ranging from 0.001% (squeaky clean) to 44% (hinky). Each specific prediction is "no, this ...
27 views

### AUROC for SVM two class classification [duplicate]

I'm trying to compute the ROC and AUROC of a binary svm classification. I already looked up a code in the internet and it's working: ...
60 views

### Accuracy Ratio Brute-force vs Logistic Regression

We want to model a binary dependent variable $Y$ with 0 or 1 values (e.g. whether a loan defaults or not) based on 3 independent variables $X_1$, $X_2$ and $X_3$. I have the following 2 methods and I ...
17 views

### Signal Detection Theory, How do I create a ROC curve from 2 distributions?

I have 2 samples and I have to decide using the Signal Detection Theory which one is the true distribution. This is my data: ...
64 views

### Deep Neural Networks: AUCROC Values Consistently = 0.5 even though RMS Error on Test Set ~10%

I am new to Neural Networks and but I have built a multi-classifier using the FANN neural network package. My multi-classifier, regardless of the network hyperparameters, consistently gives an error ...
17 views

### How to plot ROC for LOOCV by hand based on output probabilities returned by SVM? [duplicate]

I want to plot ROC for Leave-one-out (LOO) cross-validation (CV). I have the decision values as well as probability values for each class from SVM classifier. I did see the answer to this post; ...