Questions tagged [auc]
AUC stands for the Area Under the Curve and usually refers to the area under the receiver operator characteristic (ROC) curve.
473
questions
0
votes
0answers
7 views
Comparing AUC of a binomial model to Cox for a specific time point
I have a model for 2 year mortality which was derived using binomial regression. Now I want to create a time-to-event model with Cox and I want to compare the performance to the already available ...
0
votes
0answers
5 views
Perfect recall but moderate precision due to imbalance?
I have a patient dataset on which I trained a RF classifier to predict whether a patient ends up in the hospital or not. Nevertheless, this dependent variable is imbalanced (66% of the patients ended ...
0
votes
0answers
21 views
When would AUC fail in comparing models? [closed]
It is possible that a classifier might have 1 threshold where there is highest possible true positive rate and least possible false positive rate (and lets say that is what the application requires), ...
0
votes
0answers
7 views
Convert RMSE, MSE, AUC, AUC PR or log loss to accuracy
I'm using h2o automl system and it returns all the models evaluation scores in MSE, RMSE, area under curve(AUC), AUC PR (dunno what this is) and log loss. I need this in accuracy so I can compare it ...
1
vote
2answers
50 views
Showing that $P(X_1>X_2) = \int_{0}^1 P(X_1>X_2 | X_2=x) f_{X_2}(x) dx$
I am going through this post in trying to prove the probabilistic interpretation of the AUC for a ROC Curve (for a classifier):
The AUC for a ROC curve is the the probability of the classifier ...
0
votes
0answers
13 views
P-value for precision recall curve significance
I am computing the precision-recall curve for my ML model with 2 classes. I want to have a p-value that compare the observed area under the precision recall curve (AUPR_obs) and the area of such a ...
0
votes
1answer
18 views
Sklearn Average_Precision_Score vs. AUC
Can someone explain in an intuitive way the difference between Average_Precision_Score and AUC?
I read the documentation and understand that they are calculated slightly differently. But what is the ...
0
votes
0answers
8 views
Is KS statstic (in classification) from ROC Curve same as K-S from cum Gain curve or PR Curve?
This is related to classification problems (specifically binary classification problems covered in scoring) where curves referenced below are used. I understand each of these curves for a classifier ...
0
votes
1answer
23 views
How is the cost weight function $w_G(c)$ implicitly used by the AUC calculated?
In this article by David Hand an implicit function of the classification cost ratio is calculated for a specific dataset, resulting in a discrete distribution:
This is defined as
$$ w_G(c) = \pi_0 ...
0
votes
0answers
19 views
Why did KNN Perform poorly with Error Rate but good with AUC
I have a set of error rates and AUC values for classification methods, Logistic Regression, QDA, LDA, and KNN. It looks like:
...
0
votes
0answers
15 views
Is the PR AUC invariant under label flip?
The ROC-AUC curve is invariant under a flip of the labels. I don't know if it's a famous result, so I will give the proof below. My question is if the PR-AUC curve also has this property. I have not ...
0
votes
0answers
40 views
0
votes
0answers
7 views
What is the right way to take a PR AUC average of a set of binary classifiers?
I have a multilabel classification problem. I am choosing to treat it as a set of independent binary classifiers. Each label has its corresponding skewness which can range from 1% to 20%. Suppose for ...
0
votes
0answers
9 views
How to evaluate differences in standard accuracy (PCC), Kappa, and AUC scores between random forest models?
I am using two sets of variables (life-history traits and human factors) to predict extinction Risk (Threatened/Non-Threatened) in turtles. In my first model (RF1 below) I only include life-history ...
1
vote
0answers
38 views
Precision-Recall Curve and Area under Precision-Recall Curve (AUC)
I created model (logistic regression) and now trying to create Precision-Recall plot and calculate area under Precision-Recall Plot. I'd like to note that this model is defective:
...
1
vote
1answer
19 views
Concordance estimate survival data in long format
I would like to use concordance to have an idea of my models discrimination power. Because I have time dependent covariates, the time to event data is in a "long" format, i.e. one row per ...
0
votes
0answers
8 views
AUC plot from a MLSeq::classify object
I have generated a classify object using the MLSeq::classify function. I wonder how I can visualise this using a ROC or AUC curve with sensitivity and specificity on the axis.
...
1
vote
1answer
24 views
Will the AUC of the full logistic regression model be equal to or higher than the AUC of models with less features?
I want a model to predict the probability of an event occurring. I would like to use logistic regression for this. An external condition that needs to be satisfied, however, to allow me to use such a ...
0
votes
0answers
18 views
Comparing AUCs: Same Control, different Test Group - paired or unpaired? pROC
I want to compare two AUCs using a bootstraping method from pROC package (roc.test).
I have done a classification using Logistic Regression:
1. Cancer (high risk) vs. Control = AUC 1
2. Cancer (low ...
1
vote
1answer
149 views
How to generate a ROC curve from ground truth and predictions (in R package pROC)?
I've trained several machine learning models (GLM, SVM, random forest) in R to perform binary classification (predicting the presence of gallstones) and plotted ROC curves/computed AUC using the pROC ...
0
votes
0answers
11 views
What should I do when overfitting appears on AUC and not AUPR
I am training a classifier with imbalanced classes (pos/neg = 0.0006). When training the algorithm (xgboost in this case) I noticed that it shows an overfitting when we look at the AUC but the other ...
0
votes
0answers
18 views
AUC for crossvalidation
I have a medical research scenario where I am trying to predict disease progression. I need to produce a model to integrate into clinical decision support (and evaluate further). In addition to ...
3
votes
1answer
31 views
ROC Curve for unbounded scores
Say I have a classifier that assigns a score to an image based on whether it has a cat in it. The higher the score, the more likely there's a cat in it. But for this classifier, the value of the score ...
1
vote
0answers
23 views
How to decide which features are important in this binary classification task?
Consider a binary classification problem, where the dataset is highly imbalanced, with only around 20% positive labels and 80% negative labels.
Feature A has higher AuROC when considering all the data,...
1
vote
0answers
24 views
ROC and AUC for clustering algorithms [closed]
I am working on some clustering algorithms like DBSCAN and local outlier factor. Now i want to know how can I make ROC and AUC curves from clustering results. Do anyone know how can i make RO and AUC ...
0
votes
0answers
17 views
Binary probability scoring: Intuition on why a method might perform better in terms of Brier, log loss but worse in terms of Area under ROC/PR curve?
I'm trying to compare two methods. I have surface knowledge about these scorers, so I've noticed that scorers in which method A performs better are both proper scoring rule, while B performs better in ...
0
votes
0answers
17 views
Understanding the DeLong vs t-test for comparing AUROCs
DeLong's paper was recommended to me for a statistical test to compare cross-validation AUCs $c_1$ and $c_2$ from two different classifiers on the same dataset. My understanding is that it is used to ...
0
votes
1answer
23 views
How to select best models if the ROC AUC score changes drastically at each separate run?
Below are two plots for ROC curves with their AUC mentioned in the legend brackets. How do I shortlist the best models if the scores differ at each run?
Should I rather calculate the ROC AUC only from ...
2
votes
0answers
33 views
How can an ROC AUC and its 95%CI be estimated from repeated crossvalidation?
I'm using repeated 100 times 10-fold cross-validation to provide internal validation of a logistic regression model. What is the best way to obtain the ROC AUC and it's 95% CI from this data. If I ...
0
votes
0answers
20 views
Can AUC increase while both sensitivity and specificity decrease?
Assume I am given two classifiers - A and B. Is it possible that the specificity and sensitivity obtained from using B will be higher than A, but the AUC of A will be higher than the AUC of B?
Will ...
0
votes
0answers
28 views
ROC Curves for Regression Output
I am working on a broad machine learning-based problem, which can be approached in several different ways. Essentially, my training values are floats between 0.0 and 1.0, and I have approached this in ...
1
vote
1answer
49 views
Is it correct to evaluate individual drivers with the AUC value?
I have a discussion with my supervisor about the usage of AUC to determine, basically, the importance of three different drivers consisting of multiple variables each. He claims I can look into the ...
0
votes
0answers
22 views
Feature selection for optimising LOOCV AUC
at the moment I am dealing with the following problem: I have a binary classification problem with low sample size (12 and 36 instances in the two classes) and large number of features (200).
My ...
10
votes
4answers
1k views
Determine how good an AUC is (Area under the Curve of ROC)
I'm currently working on a project involving using different sets of data as a predictor to predict the outcome of out-sample data. I use AUC (Area under the Curve of ROC) to compare the performances ...
1
vote
0answers
59 views
R Studio: Calculate area under the curve with respect to ground
I would like to calculate an area under the curve with respect to ground (AUCg) according to this formula in this paper (Pruessner, Kirschbaum, Meinlschmid & Hellhammer, 2003). I have three ...
1
vote
0answers
166 views
Building AUC/ROC curve without probabilities, only with actual/predicted labels
If we dont have access to model and have just actual and predicted labels without probabilities, is it still be possible to plot AUC/ROC curve.
For example can we have the curve from the following ...
0
votes
1answer
38 views
Can I draw the ROC curve from ROC_AUC Score?
The question may look rediculous. The problem in my hand is a multi-class (5 class labels) classification problem and I coded it a year back. I am having the resulsts such as G-mean, F-score, and ...
0
votes
0answers
101 views
Is the pooled AUC calculation for imputated data in (psfmi package) mivalext_lr() correct?
I have an imputated data with several nonmissing and not-imputated variables. However, I realised when I use mivalext_lr() to obtain pooled AUC and 95% CI of my ...
0
votes
1answer
15 views
All of my AU-PRC values are the same, is there something wrong with my code or models?
I have been doing some training of basic models for a certain binary outcome, and most of the training has been on optimizing the AUC. But when I plot the precision recall curves, I get essentially ...
1
vote
0answers
25 views
One-sided test for AUC (area under the curve for ROC)
I am trying to determine the power of testing the AUC of a diagnostic test with continuous score against a one-sided alternative. I was thinking I could just use the confidence intervals produced by <...
0
votes
0answers
13 views
Validation of flexible parametric survival models
Is there any dedicated R packages available for estimating AUC for flexible parametric survival models ("stpm2")?
0
votes
0answers
22 views
Generate synthetic data given AUC for multivariate X
I need to simulate data for a fixed AUC and for multivariate X. I came across the link below which explains for the univariate x.
Generate synthetic data given AUC.
Has anyone had any ideas or ...
2
votes
1answer
47 views
How can I rank preferences without ranking records in a ML model?
In a binary classification problem about the purchase of a product I use AUC to evaluate the performance of the model.
Due to some restrictions I can't assign to each record of my data set any metric ...
0
votes
0answers
13 views
One AUC for each folder or Average AUC of all folders in Cross Validation?
Suppose I am running a leave-one-folder-out cross-validation (binary). The sample size of each folder is unbalanced.
E.g.,
folder 1: X1 = [x11, x12, x13], y1 = [0,1,1]
folder 2: X2 = [x21, x22], y2 = ...
0
votes
0answers
22 views
How to calculate the confidence interval of an AUC from its p-value?
Is there a way to calculate the confidence interval of an area under the curve (AUC) from its p-value? Let's say a study reports an AUC of 0.78 and a p-value of p = 0.001. Is there a way to calculate ...
0
votes
0answers
34 views
What's the connection between Kendall tau and AUC ROC? [duplicate]
What is the relationship between Kendall tau and the area under the receiver operating characteristic curve (AUC ROC)?
0
votes
0answers
43 views
Improve classification performance with H2O AutoML
I have to solve a simple binary classification problem using H2O AutoML. I would like yo optimize a custom loss function since the classification problem is inside a clear business framework and the ...
2
votes
0answers
83 views
Is the AUC an incoherent measure of classifier performance?
I'm learning about performance measures for binary classifiers. Reading about the AUC-ROC score I came across the article Measuring classifier performance: a coherent alternative to the area under the ...
2
votes
1answer
158 views
Is my AUC too good to be true?
I've been going in circles for months...I want to generate a list of thresholds from my training data so I can see all the thresholds at every sensitivity/1-specificity of a model. You can do this ...
0
votes
1answer
102 views
Does test AUC of 0.98 mean overfitting if we have highly imbalanced dataset (0.5% minority class)?
I built a Random Forest model to classify imbalanced data (0.5% for minority class.) I used grid search to hyper tune parameters. I got the test AUC of almost 0.99. The test AUC is completely out of ...
