AUC stands for the Area Under the Curve.

learn more… | top users | synonyms

4
votes
3answers
270 views

Is overfitted model with higher AUC on test sample better than not overfitted one

i am participating in a challange in which I have created a model that performs 70% AUC on train set and 70% AUC on hold-out test set. The other participant has created a model that performs 96% AUC ...
1
vote
1answer
15 views

AUC seems too high, confusion matrix seems only slightly better than random

My confusion matrix looks as follows: > table(actual, predicted_all) predicted_all actual 0 1 0 1728 5261 1 2088 168 While the AUC ...
1
vote
0answers
15 views

Parametric versus nonparametric AUC from ROC curve

SPSS offers two ways to estimate Area Under the Curve (AUC) and its standard error, that is nonparametricly using trapezoidal rule or parametrically using binegative exponential distribution. I have ...
1
vote
1answer
41 views

Does AUC for multiple logistic regression make sense if prediction is not the goal?

Does it makes sense to calculate the AUC if I do not want to use my multiple logistic regression model for predictions? I only want to calculate some odds ratios and test if the variables in my model ...
2
votes
1answer
44 views

How to compute the AUROC for a single categorical variable

I am building new features for a binary classifier. The new features fall into two categories: categorical and ordinal. An example of the first feature would be the colours ...
4
votes
1answer
103 views

outlier detection: area under precision recall curve

I would like to compare outlier detection algorithms. I am not sure if area under roc or under precision recall curve is the measure to use. A quick test in matlab gives me strange results. I try to ...
1
vote
1answer
70 views

The distribution of the AUC

I am wondering how the confidence interval for the Area under the Curve statistic (ROC curves) is derived. I have heard that the AUC can be assumed to be normally distributed, but I am looking for a ...
1
vote
2answers
52 views

How much is ROC biased towards the minority class?

It's known that ROC is overly optimistic in case of imbalanced data sets. How big can this bias be? For example if I read paper where they report 0.75 ROC on a dataset with 5 percent of samples being ...
1
vote
1answer
57 views

Results from rfe function (caret) to compute average metrics - R

I am computing a SVM-RFE model with the rfe function of the caret package, but I am a bit confused about the results. My code is:...
0
votes
0answers
9 views

What are some offline metrics for sparse data set

I have a real world machine learning problem: Predicting whether user will buy a item on our website. The model we used is point wise logistic regression and the offline metric is AUC. With about ...
3
votes
2answers
82 views

Do I do threshold selection for my logit model on the testing or training subset?

I have data with a binary outcome and I am doing logit model selection using AIC and BIC. I have already withheld 30% of the data as a holdout sample (testing subset) and used the remainder (training ...
0
votes
0answers
33 views

Bad score for Area Under ROC, but Area Under Precision-Recall is high?

I'm doing some classification in Apache Spark, and I am unsure how to interpret my results. I get a very bad auROC (0.53), but a pretty high auPR (0.79). These results seem a bit contradictory to me, ...
2
votes
0answers
56 views

Interpret ROC/AUC values with respect data

I am using R to plot ROC curves. I have a prediction matrix, where each column shows the prediction values corresponding to different approaches. Also, I have a ...
1
vote
0answers
32 views

In glmnet, how do I identify Lamdba for a specific AUC value [closed]

I've got a large data set (n>11000) with a large number of predictor variables (~100) and the aim is to develop a satisfying species distribution model with as few of these predictors as possible. So ...
2
votes
1answer
104 views

Comparing logistic regression models with AUC ROC in R vs Stata

I am fitting a logistic regression model for the likelihood of patients suffering morbidity after surgery. The most commonly used prediction tool at the moment is POSSUM (Physiological and Operative ...
1
vote
1answer
87 views

Is AUC via CV a good procedure for selecting optimal model?

I'm fitting a logit classifier with LASSO and cross-validation, and struggling to select the optimal model using AUC -instead of the more usual loss like binomial deviance or classification error. I ...
0
votes
1answer
101 views

Relationship between AUC and U Mann-Whitney statistic

Recently I learned about the relationship between Area Under (ROC) Curve and $U$ statistic of the Wilcoxon-Mann-Whitney test. It is supposed to follow the following rule (got it from this nice post on ...
1
vote
0answers
27 views

AUC (and other measures) dependent on the way data is split

I am applying machine learning (XGBoost) to certain problem regarding time series classification, as input as uses some numerical values around 200 features and vectorized text (tfidf). The result I ...
0
votes
0answers
25 views

Impact of a mean increase in area under the curve

I am trying to get a handle on the impact some mean increase in area under the curve calculations that are statistically significant. What I'm trying to find is a way that can be translated into a ...
1
vote
1answer
81 views

100% training accuracy despite a low cv score

I am working on an assignment where we have to study the affect of gamma and C parameters on SVM with RBF kernel. I use python's sklearn library and grid search with 10 fold cross validation (with a ...
0
votes
0answers
15 views

testing AUC greater than training AUC? [duplicate]

I have about 30,000 samples with around 500 features. I randomly selected 10% as training dataset and another 10% as test1 and the remaining 80% as test2. I used randomForest to build model using ...
14
votes
3answers
692 views

Why AUC =1 even classifier has misclassified half of the samples?

I am using a classifier which returns probabilities. To calculate AUC, I am using pROC R-package. The output probabilities from classifier are: ...
0
votes
0answers
56 views

Accuracy Ratio Brute-force vs Logistic Regression

We want to model a binary dependent variable $Y$ with 0 or 1 values (e.g. whether a loan defaults or not) based on 3 independent variables $X_1$, $X_2$ and $X_3$. I have the following 2 methods and I ...
9
votes
1answer
201 views

Evaluate Random Forest: OOB vs CV

When we assess the quality of a Random Forest, for example using AUC, is it more appropriate to compute these quantities over the Out of Bag Samples or over the hold out set of cross validation? I ...
3
votes
1answer
85 views

AUC values for different sets of features

I have a dataset with two groups of features, set 1 and set2. I have traind and tested SVM classifiers in three different settings: 1) only set 1 features, 2) only set 2 features, and 3) union of set ...
1
vote
1answer
89 views

Why compare AUC's in binary classification?

I understand that a common metric for comparing binary classifiers is the AUC of the ROC curve. But, after this is computed, only one threshold is actually chosen for classifying negative and ...
1
vote
0answers
48 views

AUC for binary ROC curve

I am using the ROCR package in R to calculate ROC and associated AUC for an arbitrary continuous data set with labels coded as 0 or 1. In case A, I have some set of labels for each entry in the data ...
0
votes
0answers
41 views

Reporting AUC on training or testing data

I have a really simple question. I am writing an article to submit to a conference. I have used SVM classifier in it. I have seen in many papers which report ROC and AUC for their classifiers, and I ...
1
vote
1answer
99 views

Does dice coefficient same as accuracy?

I come across dice coefficient for volume similarity (https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient) and accuracy (https://en.wikipedia.org/wiki/Accuracy_and_precision). It ...
1
vote
0answers
39 views

How to plot ROC for knn (and potentially kernel spectral regression)

I understand how to plot ROC for logistic classifier (like varies the probability cutoff). For KNN, how can I find the ROC? Also, what about kernel spectral regression?
0
votes
0answers
10 views

Can I compute ROC AUC of F-measure for multi class classification? [duplicate]

I know ROC AUC is computed for binary classification, as well as F-score. But for multi - class classification, is it possible to calculate ROC AUC or F-score?
21
votes
4answers
3k views

What is the name of this chart showing false and true positive rates and how is it generated?

The image below shows a continuous curve of false positive rates vs. true positive rates: However, what I don't immediately get is how these rates are being calculated. If a method is applied to a ...
1
vote
0answers
13 views

how is it possible for a model with maximum AUC to not also have minumum misclassification error?

I have an elastic net model of a binary outcome where the lambda for max AUC is different than the lambda for min misclassification error. Shouldn't they be highly (inversely) correlated
0
votes
0answers
60 views

Use AUC for model comparison but what is the optimal threshold for final prediction

We can compare the performance of different models using AUC ROC and pick the one with large AUC. Then, we still need to choose and use specific threshold to predict the label for the test data. I ...
0
votes
0answers
20 views

In the classification framework, is AUROC a performance measure or metric?

I guess, the title is self-explaining. I have seen both so far and was wondering if there is a correct term or whether it does not really matter.
4
votes
1answer
84 views

Why two interpretations of AUC(area under the ROC curver) Equivalent

I found there are two ways to understand what AUC stands for but I couldn't get why these two interpretations are equivalent mathematically. In the first interpretation, AUC is the area under the ...
17
votes
1answer
773 views

Did I just invent a Bayesian method for analysis of ROC curves?

Preamble This is a long post. If you're re-reading this, please note that I've revised the question portion, though the background material remains the same. Additionally, I believe that I've devised ...
0
votes
1answer
46 views

Is it legitimate to use sensitivity and specificity next to more proper performance measures to compare classifiers?

Clearly, Brier Score and AUROC are better performance measures to compare classifiers. However, besides that, I am interested in a let's call it more economic view. I could imagine a classifier being ...
0
votes
2answers
65 views

How to interpret the AUROC curve for mortgage denial/approval?

My binary variable is whether a mortgage application is denied(1) or approved (0). Let's say I have two classifiers. One with AUROC = 0.75 and the other with AUROC = 0.85. Is it correct to state for ...
0
votes
0answers
163 views

How can we calculate ROC AUC for classification algorithm such as random forest?

As at In classification with 2 - classes, can a higher accuracy leads to a lower ROC - AUC?, AdamO said that for random forest ROC AUC is not available, because there is no cut-off value for this ...
0
votes
1answer
47 views

In classification with 2 - classes, can a higher accuracy leads to a lower ROC - AUC?

If I have a dataset with 2 possible outputs: Positive and Negative. I have 2 classification algorithms, each leads to a different predicting results. Is it possible if the algorithm 1 returns a ...
1
vote
1answer
119 views

Do these Precision-Recall (PR) curves indicate good classification performances?

I have trained a classifier for 3 different classes, the test datasets of which are imbalanced, and then plotted the PR curves (below) to evaluate their accuracies. The plots contain the number of ...
1
vote
1answer
52 views

Should PR AUC be used in cases where there is less than 5 positives vs 10000+ negatives?

I understand that the PR-AUC provides a better accuracy estimate than the ROC-AUC in the case of highly skewed datasets. But if I have a test dataset with less than 5 positives and 10000+ negatives, ...
0
votes
0answers
53 views

Binary Models - Deviance Logloss MSE AUC R2 Misclassification - Is there a defined choice?

For Binary Classification / Logistic Regression Models, Is there a specific preference or standard of what metric to be used for comparison of 2 models, especially when the model types are different - ...
0
votes
0answers
25 views

WEKA / Binary classifiers: Why two AUCs?

If I use different classification algorithms in WEKA, one possible output is the ROC-AUC. Why do I get two AUC indicators, one for the positive instances and one for the negative instances (besides ...
0
votes
1answer
418 views

Scoring a classifier with ROC AUC

I'm confused about how scikit-learn's roc_auc_score is working. As I understand it, an ROC AUC score for a classifier is obtained as follows: Sample from the parameter space Fit the model Make ...
2
votes
0answers
44 views

How to derive a mathematical formula for AUC?

Why the area under the ROC curve is the probability that a classifier will rank a randomly chosen "positive" instance (from the retrieved predictions) higher than a randomly chosen "positive" one (...
2
votes
2answers
139 views

SVM - can I use the decision function for calculating AUC?

An SVM returns a real-valued prediction for each of the input data samples, which corresponds to its distance from the separating hyperplane. Platt's scaling is often used to output a "probability" ...
2
votes
0answers
190 views

leave-one-out cross-validation on random data results in 0 AUC for glm model

I am running repeat simulations of leave-one-out cross-validation on glmnet models of randomly generated data, and collecting the AUC on left-out predictions (vs the full set of random targets). The ...
0
votes
1answer
130 views

Area Under the ROC Curve, a simple question

I split my dataset into 2 parts: 75% of it is the training set, 25% of it is the test set. Then I estimated the logistic regression parameters in the training set and I compute the Area Under the ROC ...