# Questions tagged [auc]

AUC stands for the Area Under the Curve and usually refers to the area under the receiver operator characteristic (ROC) curve.

352 questions
1answer
42 views

### What is metrics.roc_curve and metrics.auc measuring when I'm comparing binary data with probability estimates?

I was working on a challenge, and I was excited because the metric.auc for my predicted values compared to my test values was very high. This was for a binary selection process. However, when I ...
1answer
22 views

### Practical interpretation of Precision-Recall AUC

I have a classifier with an AUC (PR) of 0.06 which I will use for a practical interpretation. My test set consists of three months of data with a total of 2,200,000 observations of which 0.03 are ...
0answers
15 views

### compute AUC and statistical testing of stratified data

I have a dataset where samples are stratified in groups. That is, there are N samples per examination, and M examinations per subject. I would like to account for this when estimating the ...
1answer
45 views

### Area between the ROC curve and the Random Guessing Line

How close is my classifier to random guessing? I need to quantify the inability of a binary classifier to obtain better results than random guessing in a single number evaluation metric. The random ...
0answers
60 views

### Does a low Area Under Curve (AUC) for ROC imply worthless classifier? [closed]

I am trying to do a binary classification (0 and 1), and in this case, the end goal is to maximise the True Positives (i.e. maximum no. of 1s should be predicted correctly, even if it gives rise to ...
1answer
37 views

### Correct conditional expectation via logistic regression but terrible AUC

Suppose you have a binary random variable $Y$, and several other random variables $X_1,...,X_p$. Your goal is to "predict $Y$ using $X_1,...,X_p$." So, you go ahead and fit logistic regression, which ...
1answer
46 views

### AUC of single model vs AUC of separate models on same data

I have created two separate binary classifiers that predict the same kind of label using 2 separate datasets. The data is in the same format. They both have a AUC of 0.94 and 0.95 I have then created ...
0answers
14 views

### need auc scoring with gridsearch in keras

I have unbalance dataset , i need to implement auc scoring in keras with gridsearch cv to find the best score. But in keras classifier auc and other metrics cannot be used directly , can anyone help ...
0answers
37 views

### Accuracy score or AUC extracted from Gradient Boosting Classifier of scikit-learn? [duplicate]

I'm working on developing a predictive model for a binary classification problem related to biomedical applications (need a really high and promising accuracy). I'm training on my training dataset and ...
1answer
62 views

### High AUC but low R squared in a random forest classifier

I have been looking for an answer on this website and on Google but I can't seem to find a clear explanation anywhere. The problem is the following. I built a Random Forest model (using Python's ...
0answers
22 views

### AUC depends on random splitting

I have around 1500 data samples, 10-15 features and I am training Logistic Regression for binary classification. Data is not "very clean" and I makes some imputation. I split data into training and ...
1answer
34 views

### Stacking AUC vs. average over folds

I have a two class prediction problem where in one class I have 70% of the samples and in the other class 30% of the samples, so class imbalance. I'm conducting 10-fold cross-validation. To calcualte ...
0answers
8 views

### Testing difference of cumulative/integrated quantities of time series?

I am new to time series data analysis. I wonder how would one test the statistical significance of the difference between integrated quantities (area under curve) from different time-series curves? (...
0answers
27 views

### Comparing AUC between two subsets of test-fold

I suspect that the out-of-sample AUC of my model depends on the number of events in the out-of-sample/testing fold (I am modelling a binary variable, 0/1). In order to test this hypothesis, I want to ...
0answers
35 views

### Aggregating ROC AUC values of several Logistic Regression Models

I have a dataset that consists of six different segments. I have calculated a Logit Regression Model for each of those segments (binary response variable, 30.000 observations in total, 63 variables ...
1answer
39 views

### Area Under the Curve - Variable and Log Transformed Variable

I have a situation where I am fitting two simple logistic regression models - one model with the variable of interest included as the only predictor, and the other model with the log of the variable ...
0answers
14 views

### What is a reliable way to obtain an optimism-correct AUC with confidence limits?

I have seen that Frank Harrell's rms package does not offer a CI for Somers Dxy (and subsequently the c-statistic/AUROC). I am trying to look at a method with good performance and a plan to ...
1answer
35 views

### Is it correctly understood that ROC/AUC cannot be calculated without flexible criterion value?

I have a proprietary predictor that simply gives me a binary output. Let's say that it is detecting faulty units. In a set where 27 units are faulty and 76 units are working the predictor correctly ...
0answers
21 views

### Evaluating concordance between multiple AUC statistics measured on the same task

I am evaluating a series of features within large genomic regions. Each feature has $n$ AUC values associated with its capacity to discriminate control from the test samples: extreme values of AUC ...
0answers
13 views

### Why is the AUC when plotted against the wilcoxon rank sum p-value, result in a plot that is not linear?

I have a binary response variable with many continuous predictor variables (about 5000). I first computed the AUC for each predictor variable, then computed the p-value associated with the AUC using ...
1answer
24 views

### Minimising CV standard deviation to increase accuracy

Does minimising the standard deviation of CV folds have any correlation with model accuracy as a theme? I've noticed that changing the order of rows in a training data can change the AUC for each CV ...
2answers
50 views

### Why does pROC roc work with non-probability predictions?

With the pROC package, I can do this: true <- c(1, 1, 1, 0) predicted <- c(0.5, 0.1, 0.6, 0.1) roc(true, predicted) which gives as expected: ...
1answer
47 views

### High AUC and Accuracy but weird output in confusion matrix

I am working on image classification problem to determine gender given a face. The dataset is located here gender face dataset on kaggle (link to my notebook). The class distribution is as follows. <...
1answer
37 views

### Why there is decrease in AUC values with increase in number of iteration in Held out validation?

I have a dataset with 600 rows and 4000 columns for which I am trying to do held-out cross-validation with 10 and 100 iterations. At first, the dataset is split into 80%:20% training and “held-out” ...
0answers
48 views

### Why is XGBoost prediction proba so concentrated within specific range? (unbalanced class)

I am pretty to new to Machine Learning. I am training on some past Kaggle competitions including the Santander Customer Satisfaction Challenge (https://www.kaggle.com/c/santander-customer-satisfaction)...
1answer
117 views

### Why is cross_val_score substantially lower than .score or roc_auc_score?

I have a trained model, a GradientBoostingClassifier. My dataset is 60 thousand something rows of data that I've split into 66/33 train/test sets. Scoring the model via the ...
0answers
29 views

### Sample size for AUC based on mean and SD of raw data

I'm trying to estimate the sample size required to achieve 80% power at a 5% significance level for a superiority study comparing AUC from bioavailability data. My challenge is the only previous ...
0answers
61 views

### Computing Standard Error of AUC over different cv folds, and assessing statistical significance

I have a dataset in a healthcare setting for which the task is to predict a binary outcome, I have done this using the Support Vector Machine algorithm in a 5-fold cross validation setup. The ...
0answers
10 views

### Which performance metric to use for stratified data? [duplicate]

I'm trying to classify a data into 3 classes (supervised), one of which is heavily underrepresented in the data set. In order to combat this imbalance, I decided to stratify the data. Now I want to ...
1answer
241 views

### How to distinguish overfitting and underfitting from the ROC AUC curve?

For model selection, one of the metric is (AUC Area Under Curve) which tell us how the models are performing and based on AUC value we can choose the best model. But how to distinguish whether a ...
0answers
17 views

### What is a Rank in AUC in a classification method?

I have read a post on AUC for which the link is given below What does AUC stand for and what is it? I am putting a new post as the above is rather old. The person gave a few points related to Rank ...
0answers
25 views

### ada model- variables overall importance

I have the object ada from a model I didn't train to predict a binary result (I don't have the training set). Ada package was used. And the result are 200 binary trees. I would like to have a ...
1answer
46 views

### K-NN optimal value of 'K'

How can I find the optimal value of 'K' in K-NN using the cross_val_score function, with scoring metric as auc_score? Do I need ...
1answer
38 views

### AUC ROC when one class consists of smaller subclasses

This question is different from Binary classification when one class consists of multiple subclasses I have two classes that I want to distinguish by a supervised learning classifier such as a random ...
1answer
313 views

### Does AUC/ROC curve return a p-value?

When reading this article, I noticed that the legend in Figure 3 gives a p-value for each AUC (Area Under the Curve) from the ROC (Receiver Operator Characteristic) curves. It says: The area under ...
2answers
54 views

### Appropriate way to get Cross Validated AUC

I was thinking about cross-validation and how it is the most appropriate way to do it... Let's take the case of binary logistic regression where the goal is to calculate the AUC. Make the partition ...
0answers
67 views

### interprete and calculate Area Under the Curve of classifier in Matlab using perfcurve?

I hope that somebody could help me with the following question. I would like to calculate the Area Under the Curve (AUC) of a classifier (Linear Discrimiant Analyses). To do this, I am using the ...
0answers
57 views

### What is the integral of the False Positive Rate over the False Positive Rate, compared to the AUC?

In machine learning the Area Under the Receiver Operating Characteristic Curve ($AUC$) can be illustrated in a plot of the True Positive Rate ($TPR$) against the False Positive Rate ($FPR$). Formally, ...
0answers
39 views

### How to calculate the standard error of ratio of two AUC

Please can anyone help me with this! I hope my explanation is going to be simple: This is the data I have: I only have access to the aggregated data as below. the first column is the time points a ...
1answer
191 views

### How do I calculate AUC with leave-one-out CV

In a binary response setting (data matrix D with N rows) I have performed LOOCV and obtained a final lambda*. The average CV error for this lambda* is also, as I understand it, an unbiased estimator ...
0answers
42 views

### High accuracy for single classes & low accuracy for multiclass classification

I was trying to do a multiclass-classification, where each sample belongs to one of the four classes. Now that I have a probability vector $(p_1^i,p_2^i,p_3^i,p_4^i)$ for each sample $i$ as my ...
2answers
210 views

### Can I trust my random forest model with low sample size and unequal class distribution?

I have a general question regarding model evaluation for random forest with low sample size and unequal class distribution. I am doing some explorative modeling by using 400 features to classify ...
0answers
16 views

### AUC/R^2 score strictly lower on partitioned dataset than total dataset

Suppose I've fit a logistic regression to my data and I calculate the AUC. I then partition my dataset according to one of the explanatory variables (say those rows where the variable value is greater/...
0answers
45 views

### hierarchical bootstrapping and calculation of variance (in a Random-Effects ROC Analysis) in R

I would like to calculate the variance of the AUC of readers (for each reader and averaged results) giving a score(1-5) to ...
1answer
240 views

### Reason for higher AUC from a test set than a training set using a random forest

I made a 70:30 split of the data to build a random forest model for binary classification. Although the prevalence of $Y=1$ was about 25% in both training and test sets, the two sets became imbalanced ...
1answer
33 views

### Why are ROC curves and AUC values not always relevant?

So I have read in posts and in literature (Frank Harrel - Regression Modeling Strategies fx) that depending on what you do, ROC curves and AUC values are not always relevant, but often written in ...
1answer
35 views

### ROC curve: models have different FPR ranges

I've created ROC curves by calculating the TPR and FPR at various thresholds. The FPR range differs between models, so I'm wondering if AUC is still a valid way to compare the curves. A curve will ...
1answer
123 views

### How do iso accuracy line related to ROC curve

I have read many articles about ROC curve. Some specified a method to calculate the accuracy of a classifier using iso accuracy lines in a convex hull. Just like in this articles : http://mlwiki.org/...
2answers
237 views

### Why AUC is not a good performance metric for a classification model?

After understanding the benefits of AUC I was stumbled to know that in some scenarios it might not be a good performance metric for evaluating a classification model. The below are the 2 scenarios: ...
1answer
49 views

### how to interpret the behavior of a ROC curve very close to one

annex the ROC curve of a logit model, which has a very high AUC, how can I interpret the behavior of this curve, I doubt the logit model?