Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [auc]

AUC stands for the Area Under the Curve and usually refers to the area under the receiver operator characteristic (ROC) curve.

2
votes
1answer
42 views

What is metrics.roc_curve and metrics.auc measuring when I'm comparing binary data with probability estimates?

I was working on a challenge, and I was excited because the metric.auc for my predicted values compared to my test values was very high. This was for a binary selection process. However, when I ...
0
votes
1answer
22 views

Practical interpretation of Precision-Recall AUC

I have a classifier with an AUC (PR) of 0.06 which I will use for a practical interpretation. My test set consists of three months of data with a total of 2,200,000 observations of which 0.03 are ...
0
votes
0answers
15 views

compute AUC and statistical testing of stratified data

I have a dataset where samples are stratified in groups. That is, there are N samples per examination, and M examinations per subject. I would like to account for this when estimating the ...
2
votes
1answer
45 views

Area between the ROC curve and the Random Guessing Line

How close is my classifier to random guessing? I need to quantify the inability of a binary classifier to obtain better results than random guessing in a single number evaluation metric. The random ...
1
vote
0answers
60 views

Does a low Area Under Curve (AUC) for ROC imply worthless classifier? [closed]

I am trying to do a binary classification (0 and 1), and in this case, the end goal is to maximise the True Positives (i.e. maximum no. of 1s should be predicted correctly, even if it gives rise to ...
0
votes
1answer
37 views

Correct conditional expectation via logistic regression but terrible AUC

Suppose you have a binary random variable $Y$, and several other random variables $X_1,...,X_p$. Your goal is to "predict $Y$ using $X_1,...,X_p$." So, you go ahead and fit logistic regression, which ...
0
votes
1answer
46 views

AUC of single model vs AUC of separate models on same data

I have created two separate binary classifiers that predict the same kind of label using 2 separate datasets. The data is in the same format. They both have a AUC of 0.94 and 0.95 I have then created ...
-1
votes
0answers
14 views

need auc scoring with gridsearch in keras

I have unbalance dataset , i need to implement auc scoring in keras with gridsearch cv to find the best score. But in keras classifier auc and other metrics cannot be used directly , can anyone help ...
0
votes
0answers
37 views

Accuracy score or AUC extracted from Gradient Boosting Classifier of scikit-learn? [duplicate]

I'm working on developing a predictive model for a binary classification problem related to biomedical applications (need a really high and promising accuracy). I'm training on my training dataset and ...
0
votes
1answer
62 views

High AUC but low R squared in a random forest classifier

I have been looking for an answer on this website and on Google but I can't seem to find a clear explanation anywhere. The problem is the following. I built a Random Forest model (using Python's ...
1
vote
0answers
22 views

AUC depends on random splitting

I have around 1500 data samples, 10-15 features and I am training Logistic Regression for binary classification. Data is not "very clean" and I makes some imputation. I split data into training and ...
0
votes
1answer
34 views

Stacking AUC vs. average over folds

I have a two class prediction problem where in one class I have 70% of the samples and in the other class 30% of the samples, so class imbalance. I'm conducting 10-fold cross-validation. To calcualte ...
0
votes
0answers
8 views

Testing difference of cumulative/integrated quantities of time series?

I am new to time series data analysis. I wonder how would one test the statistical significance of the difference between integrated quantities (area under curve) from different time-series curves? (...
1
vote
0answers
27 views

Comparing AUC between two subsets of test-fold

I suspect that the out-of-sample AUC of my model depends on the number of events in the out-of-sample/testing fold (I am modelling a binary variable, 0/1). In order to test this hypothesis, I want to ...
1
vote
0answers
35 views

Aggregating ROC AUC values of several Logistic Regression Models

I have a dataset that consists of six different segments. I have calculated a Logit Regression Model for each of those segments (binary response variable, 30.000 observations in total, 63 variables ...
2
votes
1answer
39 views

Area Under the Curve - Variable and Log Transformed Variable

I have a situation where I am fitting two simple logistic regression models - one model with the variable of interest included as the only predictor, and the other model with the log of the variable ...
0
votes
0answers
14 views

What is a reliable way to obtain an optimism-correct AUC with confidence limits?

I have seen that Frank Harrell's rms package does not offer a CI for Somers Dxy (and subsequently the c-statistic/AUROC). I am trying to look at a method with good performance and a plan to ...
0
votes
1answer
35 views

Is it correctly understood that ROC/AUC cannot be calculated without flexible criterion value?

I have a proprietary predictor that simply gives me a binary output. Let's say that it is detecting faulty units. In a set where 27 units are faulty and 76 units are working the predictor correctly ...
0
votes
0answers
21 views

Evaluating concordance between multiple AUC statistics measured on the same task

I am evaluating a series of features within large genomic regions. Each feature has $n$ AUC values associated with its capacity to discriminate control from the test samples: extreme values of AUC ...
0
votes
0answers
13 views

Why is the AUC when plotted against the wilcoxon rank sum p-value, result in a plot that is not linear?

I have a binary response variable with many continuous predictor variables (about 5000). I first computed the AUC for each predictor variable, then computed the p-value associated with the AUC using ...
0
votes
1answer
24 views

Minimising CV standard deviation to increase accuracy

Does minimising the standard deviation of CV folds have any correlation with model accuracy as a theme? I've noticed that changing the order of rows in a training data can change the AUC for each CV ...
2
votes
2answers
50 views

Why does pROC roc work with non-probability predictions?

With the pROC package, I can do this: true <- c(1, 1, 1, 0) predicted <- c(0.5, 0.1, 0.6, 0.1) roc(true, predicted) which gives as expected: ...
0
votes
1answer
47 views

High AUC and Accuracy but weird output in confusion matrix

I am working on image classification problem to determine gender given a face. The dataset is located here gender face dataset on kaggle (link to my notebook). The class distribution is as follows. <...
0
votes
1answer
37 views

Why there is decrease in AUC values with increase in number of iteration in Held out validation?

I have a dataset with 600 rows and 4000 columns for which I am trying to do held-out cross-validation with 10 and 100 iterations. At first, the dataset is split into 80%:20% training and “held-out” ...
0
votes
0answers
48 views

Why is XGBoost prediction proba so concentrated within specific range? (unbalanced class)

I am pretty to new to Machine Learning. I am training on some past Kaggle competitions including the Santander Customer Satisfaction Challenge (https://www.kaggle.com/c/santander-customer-satisfaction)...
1
vote
1answer
117 views

Why is cross_val_score substantially lower than .score or roc_auc_score?

I have a trained model, a GradientBoostingClassifier. My dataset is 60 thousand something rows of data that I've split into 66/33 train/test sets. Scoring the model via the ...
1
vote
0answers
29 views

Sample size for AUC based on mean and SD of raw data

I'm trying to estimate the sample size required to achieve 80% power at a 5% significance level for a superiority study comparing AUC from bioavailability data. My challenge is the only previous ...
0
votes
0answers
61 views

Computing Standard Error of AUC over different cv folds, and assessing statistical significance

I have a dataset in a healthcare setting for which the task is to predict a binary outcome, I have done this using the Support Vector Machine algorithm in a 5-fold cross validation setup. The ...
0
votes
0answers
10 views

Which performance metric to use for stratified data? [duplicate]

I'm trying to classify a data into 3 classes (supervised), one of which is heavily underrepresented in the data set. In order to combat this imbalance, I decided to stratify the data. Now I want to ...
0
votes
1answer
241 views

How to distinguish overfitting and underfitting from the ROC AUC curve?

For model selection, one of the metric is (AUC Area Under Curve) which tell us how the models are performing and based on AUC value we can choose the best model. But how to distinguish whether a ...
0
votes
0answers
17 views

What is a Rank in AUC in a classification method?

I have read a post on AUC for which the link is given below What does AUC stand for and what is it? I am putting a new post as the above is rather old. The person gave a few points related to Rank ...
0
votes
0answers
25 views

ada model- variables overall importance

I have the object ada from a model I didn't train to predict a binary result (I don't have the training set). Ada package was used. And the result are 200 binary trees. I would like to have a ...
2
votes
1answer
46 views

K-NN optimal value of 'K'

How can I find the optimal value of 'K' in K-NN using the cross_val_score function, with scoring metric as auc_score? Do I need ...
1
vote
1answer
38 views

AUC ROC when one class consists of smaller subclasses

This question is different from Binary classification when one class consists of multiple subclasses I have two classes that I want to distinguish by a supervised learning classifier such as a random ...
2
votes
1answer
313 views

Does AUC/ROC curve return a p-value?

When reading this article, I noticed that the legend in Figure 3 gives a p-value for each AUC (Area Under the Curve) from the ROC (Receiver Operator Characteristic) curves. It says: The area under ...
2
votes
2answers
54 views

Appropriate way to get Cross Validated AUC

I was thinking about cross-validation and how it is the most appropriate way to do it... Let's take the case of binary logistic regression where the goal is to calculate the AUC. Make the partition ...
0
votes
0answers
67 views

interprete and calculate Area Under the Curve of classifier in Matlab using perfcurve?

I hope that somebody could help me with the following question. I would like to calculate the Area Under the Curve (AUC) of a classifier (Linear Discrimiant Analyses). To do this, I am using the ...
1
vote
0answers
57 views

What is the integral of the False Positive Rate over the False Positive Rate, compared to the AUC?

In machine learning the Area Under the Receiver Operating Characteristic Curve ($AUC$) can be illustrated in a plot of the True Positive Rate ($TPR$) against the False Positive Rate ($FPR$). Formally, ...
2
votes
0answers
39 views

How to calculate the standard error of ratio of two AUC

Please can anyone help me with this! I hope my explanation is going to be simple: This is the data I have: I only have access to the aggregated data as below. the first column is the time points a ...
1
vote
1answer
191 views

How do I calculate AUC with leave-one-out CV

In a binary response setting (data matrix D with N rows) I have performed LOOCV and obtained a final lambda*. The average CV error for this lambda* is also, as I understand it, an unbiased estimator ...
0
votes
0answers
42 views

High accuracy for single classes & low accuracy for multiclass classification

I was trying to do a multiclass-classification, where each sample belongs to one of the four classes. Now that I have a probability vector $(p_1^i,p_2^i,p_3^i,p_4^i)$ for each sample $i$ as my ...
2
votes
2answers
210 views

Can I trust my random forest model with low sample size and unequal class distribution?

I have a general question regarding model evaluation for random forest with low sample size and unequal class distribution. I am doing some explorative modeling by using 400 features to classify ...
0
votes
0answers
16 views

AUC/R^2 score strictly lower on partitioned dataset than total dataset

Suppose I've fit a logistic regression to my data and I calculate the AUC. I then partition my dataset according to one of the explanatory variables (say those rows where the variable value is greater/...
0
votes
0answers
45 views

hierarchical bootstrapping and calculation of variance (in a Random-Effects ROC Analysis) in R

I would like to calculate the variance of the AUC of readers (for each reader and averaged results) giving a score(1-5) to ...
2
votes
1answer
240 views

Reason for higher AUC from a test set than a training set using a random forest

I made a 70:30 split of the data to build a random forest model for binary classification. Although the prevalence of $Y=1$ was about 25% in both training and test sets, the two sets became imbalanced ...
1
vote
1answer
33 views

Why are ROC curves and AUC values not always relevant?

So I have read in posts and in literature (Frank Harrel - Regression Modeling Strategies fx) that depending on what you do, ROC curves and AUC values are not always relevant, but often written in ...
1
vote
1answer
35 views

ROC curve: models have different FPR ranges

I've created ROC curves by calculating the TPR and FPR at various thresholds. The FPR range differs between models, so I'm wondering if AUC is still a valid way to compare the curves. A curve will ...
1
vote
1answer
123 views

How do iso accuracy line related to ROC curve

I have read many articles about ROC curve. Some specified a method to calculate the accuracy of a classifier using iso accuracy lines in a convex hull. Just like in this articles : http://mlwiki.org/...
2
votes
2answers
237 views

Why AUC is not a good performance metric for a classification model?

After understanding the benefits of AUC I was stumbled to know that in some scenarios it might not be a good performance metric for evaluating a classification model. The below are the 2 scenarios: ...
2
votes
1answer
49 views

how to interpret the behavior of a ROC curve very close to one

annex the ROC curve of a logit model, which has a very high AUC, how can I interpret the behavior of this curve, I doubt the logit model?