an acronym for Area Under Curve.

learn more… | top users | synonyms

0
votes
0answers
7 views

What are some offline metrics for sparse data set

I have a real world machine learning problem: Predicting whether user will buy a item on our website. The model we used is point wise logistic regression and the offline metric is AUC. With about ...
3
votes
2answers
71 views

Do I do threshold selection for my logit model on the testing or training subset?

I have data with a binary outcome and I am doing logit model selection using AIC and BIC. I have already withheld 30% of the data as a holdout sample (testing subset) and used the remainder (training ...
0
votes
0answers
19 views

Bad score for Area Under ROC, but Area Under Precision-Recall is high?

I'm doing some classification in Apache Spark, and I am unsure how to interpret my results. I get a very bad auROC (0.53), but a pretty high auPR (0.79). These results seem a bit contradictory to me, ...
2
votes
0answers
46 views

Interpret ROC/AUC values with respect data

I am using R to plot ROC curves. I have a prediction matrix, where each column shows the prediction values corresponding to different approaches. Also, I have a ...
1
vote
0answers
25 views

In glmnet, how do I identify Lamdba for a specific AUC value [closed]

I've got a large data set (n>11000) with a large number of predictor variables (~100) and the aim is to develop a satisfying species distribution model with as few of these predictors as possible. So ...
2
votes
1answer
84 views

Comparing logistic regression models with AUC ROC in R vs Stata

I am fitting a logistic regression model for the likelihood of patients suffering morbidity after surgery. The most commonly used prediction tool at the moment is POSSUM (Physiological and Operative ...
1
vote
1answer
80 views

Is AUC via CV a good procedure for selecting optimal model?

I'm fitting a logit classifier with LASSO and cross-validation, and struggling to select the optimal model using AUC -instead of the more usual loss like binomial deviance or classification error. I ...
0
votes
1answer
44 views

Relationship between AUC and U Mann-Whitney statistic

Recently I learned about the relationship between Area Under (ROC) Curve and $U$ statistic of the Wilcoxon-Mann-Whitney test. It is supposed to follow the following rule (got it from this nice post on ...
1
vote
0answers
25 views

AUC (and other measures) dependent on the way data is split

I am applying machine learning (XGBoost) to certain problem regarding time series classification, as input as uses some numerical values around 200 features and vectorized text (tfidf). The result I ...
0
votes
0answers
18 views

Impact of a mean increase in area under the curve

I am trying to get a handle on the impact some mean increase in area under the curve calculations that are statistically significant. What I'm trying to find is a way that can be translated into a ...
1
vote
1answer
71 views

100% training accuracy despite a low cv score

I am working on an assignment where we have to study the affect of gamma and C parameters on SVM with RBF kernel. I use python's sklearn library and grid search with 10 fold cross validation (with a ...
0
votes
0answers
15 views

testing AUC greater than training AUC? [duplicate]

I have about 30,000 samples with around 500 features. I randomly selected 10% as training dataset and another 10% as test1 and the remaining 80% as test2. I used randomForest to build model using ...
14
votes
3answers
683 views

Why AUC =1 even classifier has misclassified half of the samples?

I am using a classifier which returns probabilities. To calculate AUC, I am using pROC R-package. The output probabilities from classifier are: ...
0
votes
0answers
51 views

Accuracy Ratio Brute-force vs Logistic Regression

We want to model a binary dependent variable $Y$ with 0 or 1 values (e.g. whether a loan defaults or not) based on 3 independent variables $X_1$, $X_2$ and $X_3$. I have the following 2 methods and I ...
9
votes
1answer
180 views

Evaluate Random Forest: OOB vs CV

When we assess the quality of a Random Forest, for example using AUC, is it more appropriate to compute these quantities over the Out of Bag Samples or over the hold out set of cross validation? I ...
3
votes
1answer
80 views

AUC values for different sets of features

I have a dataset with two groups of features, set 1 and set2. I have traind and tested SVM classifiers in three different settings: 1) only set 1 features, 2) only set 2 features, and 3) union of set ...
1
vote
1answer
79 views

Why compare AUC's in binary classification?

I understand that a common metric for comparing binary classifiers is the AUC of the ROC curve. But, after this is computed, only one threshold is actually chosen for classifying negative and ...
1
vote
0answers
43 views

AUC for binary ROC curve

I am using the ROCR package in R to calculate ROC and associated AUC for an arbitrary continuous data set with labels coded as 0 or 1. In case A, I have some set of labels for each entry in the data ...
0
votes
0answers
29 views

Reporting AUC on training or testing data

I have a really simple question. I am writing an article to submit to a conference. I have used SVM classifier in it. I have seen in many papers which report ROC and AUC for their classifiers, and I ...
1
vote
1answer
61 views

Does dice coefficient same as accuracy?

I come across dice coefficient for volume similarity (https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient) and accuracy (https://en.wikipedia.org/wiki/Accuracy_and_precision). It ...
1
vote
0answers
31 views

How to plot ROC for knn (and potentially kernel spectral regression)

I understand how to plot ROC for logistic classifier (like varies the probability cutoff). For KNN, how can I find the ROC? Also, what about kernel spectral regression?
0
votes
0answers
10 views

Can I compute ROC AUC of F-measure for multi class classification? [duplicate]

I know ROC AUC is computed for binary classification, as well as F-score. But for multi - class classification, is it possible to calculate ROC AUC or F-score?
21
votes
4answers
3k views

What is the name of this chart showing false and true positive rates and how is it generated?

The image below shows a continuous curve of false positive rates vs. true positive rates: However, what I don't immediately get is how these rates are being calculated. If a method is applied to a ...
1
vote
0answers
13 views

how is it possible for a model with maximum AUC to not also have minumum misclassification error?

I have an elastic net model of a binary outcome where the lambda for max AUC is different than the lambda for min misclassification error. Shouldn't they be highly (inversely) correlated
0
votes
0answers
54 views

Use AUC for model comparison but what is the optimal threshold for final prediction

We can compare the performance of different models using AUC ROC and pick the one with large AUC. Then, we still need to choose and use specific threshold to predict the label for the test data. I ...
0
votes
0answers
20 views

In the classification framework, is AUROC a performance measure or metric?

I guess, the title is self-explaining. I have seen both so far and was wondering if there is a correct term or whether it does not really matter.
4
votes
1answer
78 views

Why two interpretations of AUC(area under the ROC curver) Equivalent

I found there are two ways to understand what AUC stands for but I couldn't get why these two interpretations are equivalent mathematically. In the first interpretation, AUC is the area under the ...
17
votes
1answer
746 views

Did I just invent a Bayesian method for analysis of ROC curves?

Preamble This is a long post. If you're re-reading this, please note that I've revised the question portion, though the background material remains the same. Additionally, I believe that I've devised ...
0
votes
1answer
41 views

Is it legitimate to use sensitivity and specificity next to more proper performance measures to compare classifiers?

Clearly, Brier Score and AUROC are better performance measures to compare classifiers. However, besides that, I am interested in a let's call it more economic view. I could imagine a classifier being ...
0
votes
2answers
59 views

How to interpret the AUROC curve for mortgage denial/approval?

My binary variable is whether a mortgage application is denied(1) or approved (0). Let's say I have two classifiers. One with AUROC = 0.75 and the other with AUROC = 0.85. Is it correct to state for ...
0
votes
0answers
132 views

How can we calculate ROC AUC for classification algorithm such as random forest?

As at In classification with 2 - classes, can a higher accuracy leads to a lower ROC - AUC?, AdamO said that for random forest ROC AUC is not available, because there is no cut-off value for this ...
0
votes
1answer
36 views

In classification with 2 - classes, can a higher accuracy leads to a lower ROC - AUC?

If I have a dataset with 2 possible outputs: Positive and Negative. I have 2 classification algorithms, each leads to a different predicting results. Is it possible if the algorithm 1 returns a ...
1
vote
1answer
114 views

Do these Precision-Recall (PR) curves indicate good classification performances?

I have trained a classifier for 3 different classes, the test datasets of which are imbalanced, and then plotted the PR curves (below) to evaluate their accuracies. The plots contain the number of ...
1
vote
1answer
49 views

Should PR AUC be used in cases where there is less than 5 positives vs 10000+ negatives?

I understand that the PR-AUC provides a better accuracy estimate than the ROC-AUC in the case of highly skewed datasets. But if I have a test dataset with less than 5 positives and 10000+ negatives, ...
0
votes
0answers
41 views

Binary Models - Deviance Logloss MSE AUC R2 Misclassification - Is there a defined choice?

For Binary Classification / Logistic Regression Models, Is there a specific preference or standard of what metric to be used for comparison of 2 models, especially when the model types are different - ...
0
votes
0answers
24 views

WEKA / Binary classifiers: Why two AUCs?

If I use different classification algorithms in WEKA, one possible output is the ROC-AUC. Why do I get two AUC indicators, one for the positive instances and one for the negative instances (besides ...
0
votes
1answer
354 views

Scoring a classifier with ROC AUC

I'm confused about how scikit-learn's roc_auc_score is working. As I understand it, an ROC AUC score for a classifier is obtained as follows: Sample from the parameter space Fit the model Make ...
2
votes
0answers
42 views

How to derive a mathematical formula for AUC?

Why the area under the ROC curve is the probability that a classifier will rank a randomly chosen "positive" instance (from the retrieved predictions) higher than a randomly chosen "positive" one ...
2
votes
2answers
110 views

SVM - can I use the decision function for calculating AUC?

An SVM returns a real-valued prediction for each of the input data samples, which corresponds to its distance from the separating hyperplane. Platt's scaling is often used to output a "probability" ...
2
votes
0answers
186 views

leave-one-out cross-validation on random data results in 0 AUC for glm model

I am running repeat simulations of leave-one-out cross-validation on glmnet models of randomly generated data, and collecting the AUC on left-out predictions (vs the full set of random targets). The ...
0
votes
1answer
118 views

Area Under the ROC Curve, a simple question

I split my dataset into 2 parts: 75% of it is the training set, 25% of it is the test set. Then I estimated the logistic regression parameters in the training set and I compute the Area Under the ROC ...
3
votes
1answer
84 views

Different ROC value for different packages in R, which one is correct?

I noticed that computing ROC with caret package and PROC packege sometimes gives different results. Usually they are the same, but if the predictions are worse than chance, caret will flip them and ...
0
votes
0answers
112 views

Is the ROC curve and estimated AUC from SPSS parametric or nonparametric?

I am using SPSS to generate some ROC curves, AUC and p values. According to SPSS manual, the AUC can be computed parametrically or nonparametrically. However, I do not see any option for that. There ...
0
votes
0answers
48 views

Confidence intervals for AUROC for repeated cross validation

I'm building a risk model using logistic regression in Stata. We are using $h \times k$ cross validation and calculating AUROC as part of the model validation procedure. Using Stata I've managed to ...
0
votes
1answer
63 views

GBM Performance on different sampling techniques

I am working on a healthcare data set for breast cancer patients. This data set is class imbalances and the distribution of positive and negative classes is 80%/20%. In order to deal with the class ...
1
vote
0answers
51 views

Does it make sense to minimize AUC when using GBM with weights?

I am using gbm(R's caret packages - using train function) on a class imbalanced data set with weights. So, class-1 has a weight of 1 and class-0 has a weight of 10. I am using parameter tuning and ...
2
votes
0answers
118 views

How to calculate AUC in Adaboost testing phase in R?

I'm using ROCR package in R for calculating AUC and drawing ROC curve using ...
0
votes
1answer
1k views

How can I calculate the AUC of combined variables using SPSS

thank you for taking time out to read this. I have previously ran ROC curves to get the AUCs for single test variables but I do not know how to derive the AUC for combined variables (2 test ...
1
vote
0answers
41 views

Diagnostic accuracy test done using mada package. How can I compare the AUC of SROC curves?

I am performing a meta-analysis to compare the diagnostic accuracy of different modalities on the same population. I have constructed SROC curves and have calculated the AUC values in each case. Is ...
2
votes
1answer
47 views

What should the AUROC be on the test set when no positive example is present?

Assume we have a probabilistic, binary classifier. We compute the AUROC on a test set in which no positive example is present (i.e. the ground truth is always 0). What should the AUROC be?