Questions tagged [auc]
AUC stands for the Area Under the Curve and usually refers to the area under the receiver operator characteristic (ROC) curve.
443
questions
1
vote
0answers
8 views
R Studio: Calculate area under the curve with respect to ground
I would like to calculate an area under the curve with respect to ground (AUCg) according to this formula in this paper (Pruessner, Kirschbaum, Meinlschmid & Hellhammer, 2003). I have three ...
1
vote
0answers
36 views
Building AUC/ROC curve without probabilities, only with actual/predicted labels
If we dont have access to model and have just actual and predicted labels without probabilities, is it still be possible to plot AUC/ROC curve.
For example can we have the curve from the following ...
0
votes
1answer
34 views
Can I draw the ROC curve from ROC_AUC Score?
The question may look rediculous. The problem in my hand is a multi-class (5 class labels) classification problem and I coded it a year back. I am having the resulsts such as G-mean, F-score, and ...
0
votes
0answers
21 views
Is the pooled AUC calculation for imputated data in (psfmi package) mivalext_lr() correct?
I have an imputated data with several nonmissing and not-imputated variables. However, I realised when I use mivalext_lr() to obtain pooled AUC and 95% CI of my ...
0
votes
0answers
38 views
confused AUC ROC score [migrated]
I am working on binary classification problem, I try to evaluate the performance of some classification algorithms (LR,Decission Tree , Random forest ...). I am using a 10 fold cross-validation ...
0
votes
1answer
11 views
All of my AU-PRC values are the same, is there something wrong with my code or models?
I have been doing some training of basic models for a certain binary outcome, and most of the training has been on optimizing the AUC. But when I plot the precision recall curves, I get essentially ...
0
votes
0answers
13 views
One-sided test for AUC (area under the curve for ROC)
I am trying to determine the power of testing the AUC of a diagnostic test with continuous score against a one-sided alternative. I was thinking I could just use the confidence intervals produced by <...
0
votes
0answers
9 views
Validation of flexible parametric survival models
Is there any dedicated R packages available for estimating AUC for flexible parametric survival models ("stpm2")?
0
votes
0answers
22 views
Generate synthetic data given AUC for multivariate X
I need to simulate data for a fixed AUC and for multivariate X. I came across the link below which explains for the univariate x.
Generate synthetic data given AUC.
Has anyone had any ideas or ...
2
votes
1answer
45 views
How can I rank preferences without ranking records in a ML model?
In a binary classification problem about the purchase of a product I use AUC to evaluate the performance of the model.
Due to some restrictions I can't assign to each record of my data set any metric ...
0
votes
0answers
13 views
One AUC for each folder or Average AUC of all folders in Cross Validation?
Suppose I am running a leave-one-folder-out cross-validation (binary). The sample size of each folder is unbalanced.
E.g.,
folder 1: X1 = [x11, x12, x13], y1 = [0,1,1]
folder 2: X2 = [x21, x22], y2 = ...
0
votes
0answers
19 views
How to calculate the confidence interval of an AUC from its p-value?
Is there a way to calculate the confidence interval of an area under the curve (AUC) from its p-value? Let's say a study reports an AUC of 0.78 and a p-value of p = 0.001. Is there a way to calculate ...
0
votes
0answers
13 views
What's the connection between Kendall tau and AUC ROC? [duplicate]
What is the relationship between Kendall tau and the area under the receiver operating characteristic curve (AUC ROC)?
0
votes
0answers
16 views
Improve classification performance with H2O AutoML
I have to solve a simple binary classification problem using H2O AutoML. I would like yo optimize a custom loss function since the classification problem is inside a clear business framework and the ...
1
vote
0answers
40 views
Is the AUC an incoherent measure of classifier performance?
I'm learning about performance measures for binary classifiers. Reading about the AUC-ROC score I came across the article Measuring classifier performance: a coherent alternative to the area under the ...
2
votes
1answer
98 views
Is my AUC too good to be true?
I've been going in circles for months...I want to generate a list of thresholds from my training data so I can see all the thresholds at every sensitivity/1-specificity of a model. You can do this ...
0
votes
1answer
64 views
Does test AUC of 0.98 mean overfitting if we have highly imbalanced dataset (0.5% minority class)?
I built a Random Forest model to classify imbalanced data (0.5% for minority class.) I used grid search to hyper tune parameters. I got the test AUC of almost 0.99. The test AUC is completely out of ...
3
votes
2answers
37 views
Do repeated measures affect point estimates of predicted probabilities or AUC from logistic regression?
I understand the presence of repeated measures affects standard errors and hence inferences using p-values, confidence intervals, etc.
However, do repeated measures affect the point estimates of ...
0
votes
1answer
29 views
Cross Validation Random Forest AUC very high
first time poster so hopefully this makes sense and please let me know of any important information I may leave out.
I'm trying to identify my cross-validated AUCs for basic GLM, elastic net, and ...
1
vote
1answer
47 views
Logarithmic loss vs Brier score vs AUC score
I have a dataset with two classes of elements. I also have two methods which assign (complementary) probabilities to each element in the dataset of belonging to either class.
Given that I work with ...
1
vote
1answer
31 views
H2o Python - How do I determine the threshold for AUC
I am new to H2o and I having trouble with AUC and Confusion Matrix
I have a classification model using H2o in Python for which the AUC = 71%
But the accuracy based on confusion Matrix is only 61%. I ...
1
vote
0answers
24 views
Models have high accuracy but very low AUC PR curve
I'm doing balanced (70%-30%) binary classification (yes/no)
I'm trying to combine caret train objects with PRROC's pr.curve function. I'm using a confusion matrix to determine which class is labelled ...
1
vote
1answer
42 views
When is a dataset “too imbalanced” for AUC ROC and PR is preferred?
I’ve read that precision-recall (PR) curves are preferred over AUC-ROC curves when a dataset is imbalanced as there’s more of a focus on the model’s performance in correctly identifying the minority/...
0
votes
0answers
14 views
What is a good PR-AUC and should I undersample time series for rare event detection? [duplicate]
I have a binary classifier for a highly imbalanced multivariate time series.
I use an LSTM Network to predict the next time step and use the prediction error to decide whether a data point is an ...
1
vote
0answers
26 views
Q: AUC of each subgroup is smaller than overall AUC
I have a validation data set of 29242 patients, with known labels/health outcomes and predictions that were generated by some model. 28626 patients are negative and 616 are positive
The overall AUC is ...
0
votes
0answers
14 views
Can a classifier's ROC AUC be compared to a person's accuracy at correctly ranking a positive, negative pair?
I've read that ROC AUC is equivalent to the probability of correctly ranking a positive, negative pair. If a human were to attempt to correctly rank a bunch of positive, negative pairs, could their ...
0
votes
1answer
43 views
Understanding AUC curve [closed]
This post is hidden. You deleted this post 12 hours ago.
enter image description here
In this image, i have a series of threshold as follows 0.5, 0.8, 0.85, 0.95. now in this diagram, if i choose the ...
1
vote
0answers
7 views
How to draw Receiver Operating Characteristics (ROC) of 34 video sequences using MatLab
I am working on video processing using MatLab R2017a. I have datasets containing 34 video sequences. I have to calculate the AUC of this datasets. As I know AUC is dependent upon the ROC. And ROC is a ...
0
votes
0answers
23 views
How can logistic regression maximize AUC? [duplicate]
$AUC$ is the area under the receiver operating characteristic curve. It's said that a loss function set to $1-AUC$ can be minimized by maximizing $AUC$. How does this tie into logit (logistic) ...
0
votes
1answer
66 views
Which performance metrics for highly imbalanced multiclass dataset?
I have a dataset with 5 classes. About 98% of the dataset belong to class 5. Classes 1-4 share equally about 2% of the dataset. However, it is highly important, that classes 1-4 are correctly ...
1
vote
0answers
23 views
Generating confidence interval for difference between 2 AUCs
I am trying to get a confidence interval for the difference between 2 AUCs for paired ROC curves. I've been reading the paper pROC: an open-source package for R and S+ to analyze and compare ROC ...
1
vote
0answers
26 views
Relationship between AUC of ROC curve and AUC of PR (precision-recall) curve
I know that both the ROC curve and the PR curve can be used to evaluate the performance of a binary classification prediction model, and PR curve is preferred in the case of imbalanced class ...
0
votes
0answers
38 views
Improving multiclass classification accuracy for LSTM network
I am working on time series multiclass classification task.
My data consists of 5 features (3 categorical and 2 numerical features) and 3 classes target value. Here are the histograms of the class ...
0
votes
0answers
16 views
Evaluation metric for classification + regression, as in weighted True by another objective? AUC + MSE?
I'm using purchase data to build a cross-selling T/F response model for the banking industry, to score customers based on their likelihood to acquire a loan. On the other hand, I'm also building a ...
0
votes
0answers
13 views
What scoring metric is optimal when performing hyperparameter optimization with a multiclass target variable?
I'm trying to find the optimal hyperparameter for different algorithms where the target variable has 3 classes. I was wondering if maximizing the average AUC over the 3 classes, which I'm currently ...
1
vote
0answers
31 views
How to interpret an AUC value of 0.55
I have a classifier with an AUC value of 0.55. I understand that an AUC value of 0.5 is essentially a random classifier and not predictive at all. So I am wondering if an AUC of 0.55 is ...
0
votes
1answer
25 views
Why is an ROC curve TPR (Y) against FPR (X)?
I am trying to thoroughly understand the ROC Curve and I was wondering why is an ROC Curve always (seemingly) TPR against FPR? I have had discussions with others about this matter and I cannot think ...
0
votes
1answer
36 views
How to compare two or more mean AUC
I have different machine learning models and for each of them after a 10-fold cross validation I obtained a mean AUC (+-std). Now, How can I check if there is a statistically significant difference (...
0
votes
0answers
8 views
How to understand the mathematics of ROC/AUC in Wikipedia [duplicate]
Could someone help explain what happened in the red box? I'm struggling to figure out where the indicator function comes from in this calculation. Thank you~ wiki link
1
vote
0answers
161 views
R: Caret package: AUC (“ROC”) standard error from cross validation
This seems like a basic questions, so I'm likely missing the big picture here...
I would like to know the AUC of each fold of a cross-validation performed in Caret's train function, in order to ...
2
votes
1answer
33 views
Adding predictors in ROC curves and how does it affect AUC?
I have a general question about ROC curves and how adding predictors affect AUC values.
Let's say I have a model that contains only predictor A and produces an AUC of 0.6.
I then add into the model a ...
0
votes
1answer
42 views
Calculating AUC on test set of random forest model in R
I have built a random forest model within a k-fold cv, and have had no problem calculating the AUC on the training set, but after trying a few methods of calculating AUC I keep getting errors to the ...
1
vote
1answer
29 views
calculate Specificity and sensitivity from AUC
can someone please tell me how do you calculate Specificity and sensitivity from AUC.
thank you.
2
votes
0answers
72 views
ROC Convex Hull for Model Selection
The area under the convex hull of a roc curve is by construction always "better" than its area under curve. Some curves might see more of an increase in reported auc than others. Is it a viable ...
0
votes
0answers
19 views
Feature Selection by individual AUC
I am creating a model for classification and I have several ways to get subset of features but I was wondering if the following is reasonable:
Use the train set to calculate LOOCV or LPOCV AUC values ...
1
vote
1answer
47 views
ROC Curve AUC for Hypothesis Testing Sensitivity (Power) vs Specificity ($1-\alpha$)
Let's set up a hypothesis test of $H_0: \theta=\theta_0$ versus $H_1: \theta\ne\theta_0$, and let's say that I have two techniques to assess this (say equal-variance t-test versus unequal-variance ...
1
vote
0answers
109 views
The usage of TOC curves?
In machine learning literature ROC curves are common performance measures. Quite recently published and not as popular (50+ citatations) are TOC curves as proposed by Pontius Jr, Robert Gilmore and Si,...
0
votes
1answer
43 views
Feature Selection - Overfit?
I have a dataset of 100 patients and 1500 features. I split 80 train 20 test first and then use the train set to get the best hyperparameters / best feature subset doing the following:
I randomly ...
2
votes
0answers
47 views
What is the appropriate use of the DeLong Test when comparing AUC of two different models tested on the same dataset?
In this scenario there are three datasets. Each dataset has 1000 images of an apple and 1000 images of a banana. Dataset A is used to train machine learning model A to classify whether the image is of ...
0
votes
1answer
42 views
Sampling distribution of AUC in multivariate normal distribution
I need to derive the sampling distribution of the above equation, which is sample mean of the AUC from what I see. The first picture describes the distribution of the data, which I understand is ...
