Questions tagged [auc]

AUC stands for the Area Under the Curve and usually refers to the area under the receiver operator characteristic (ROC) curve.

Filter by
Sorted by
Tagged with
3
votes
1answer
353 views

How is iso accuracy line related to ROC curve

I have read many articles about ROC curve. Some specified a method to calculate the accuracy of a classifier using iso accuracy lines in a convex hull. Some article examples: http://mlwiki.org/index....
8
votes
3answers
12k views

AUC and class imbalance in training/test dataset

I just start to learn the Area under the ROC curve (AUC). I am told that AUC is not reflected by data imbalance. I think it means that AUC is insensitive to imbalance in test data, rather than ...
1
vote
0answers
20 views

Models have high accuracy but very low AUC PR curve

I'm doing balanced (70%-30%) binary classification (yes/no) I'm trying to combine caret train objects with PRROC's pr.curve function. I'm using a confusion matrix to determine which class is labelled ...
0
votes
1answer
21 views

When is a dataset “too imbalanced” for AUC ROC and PR is preferred?

I’ve read that precision-recall (PR) curves are preferred over AUC-ROC curves when a dataset is imbalanced as there’s more of a focus on the model’s performance in correctly identifying the minority/...
0
votes
0answers
12 views

What is a good PR-AUC and should I undersample time series for rare event detection? [duplicate]

I have a binary classifier for a highly imbalanced multivariate time series. I use an LSTM Network to predict the next time step and use the prediction error to decide whether a data point is an ...
1
vote
0answers
9 views

Q: AUC of each subgroup is smaller than overall AUC

I have a validation data set of 29242 patients, with known labels/health outcomes and predictions that were generated by some model. 28626 patients are negative and 616 are positive The overall AUC is ...
0
votes
0answers
13 views

Can a classifier's ROC AUC be compared to a person's accuracy at correctly ranking a positive, negative pair?

I've read that ROC AUC is equivalent to the probability of correctly ranking a positive, negative pair. If a human were to attempt to correctly rank a bunch of positive, negative pairs, could their ...
0
votes
1answer
39 views

Understanding AUC curve [closed]

This post is hidden. You deleted this post 12 hours ago. enter image description here In this image, i have a series of threshold as follows 0.5, 0.8, 0.85, 0.95. now in this diagram, if i choose the ...
13
votes
1answer
331 views

Does a logistic regression maximizing likelihood necessarily also maximize AUC over linear models?

Given a data set with binary outcomes $y\in\{0,1\}^n$ and some predictor matrix $X\in\mathbb{R}^{n\times p}$, the standard logistic regression model estimates coefficients $\beta_{MLE}$ which maximize ...
1
vote
0answers
7 views

How to draw Receiver Operating Characteristics (ROC) of 34 video sequences using MatLab

I am working on video processing using MatLab R2017a. I have datasets containing 34 video sequences. I have to calculate the AUC of this datasets. As I know AUC is dependent upon the ROC. And ROC is a ...
0
votes
0answers
22 views

How can logistic regression maximize AUC? [duplicate]

$AUC$ is the area under the receiver operating characteristic curve. It's said that a loss function set to $1-AUC$ can be minimized by maximizing $AUC$. How does this tie into logit (logistic) ...
0
votes
1answer
33 views

Which performance metrics for highly imbalanced multiclass dataset?

I have a dataset with 5 classes. About 98% of the dataset belong to class 5. Classes 1-4 share equally about 2% of the dataset. However, it is highly important, that classes 1-4 are correctly ...
14
votes
2answers
7k views

Evaluate Random Forest: OOB vs CV

When we assess the quality of a Random Forest, for example using AUC, is it more appropriate to compute these quantities over the Out of Bag Samples or over the hold out set of cross validation? I ...
1
vote
0answers
22 views

Generating confidence interval for difference between 2 AUCs

I am trying to get a confidence interval for the difference between 2 AUCs for paired ROC curves. I've been reading the paper pROC: an open-source package for R and S+ to analyze and compare ROC ...
1
vote
0answers
24 views

Relationship between AUC of ROC curve and AUC of PR (precision-recall) curve

I know that both the ROC curve and the PR curve can be used to evaluate the performance of a binary classification prediction model, and PR curve is preferred in the case of imbalanced class ...
0
votes
0answers
21 views

Improving multiclass classification accuracy for LSTM network

I am working on time series multiclass classification task. My data consists of 5 features (3 categorical and 2 numerical features) and 3 classes target value. Here are the histograms of the class ...
0
votes
0answers
14 views

Evaluation metric for classification + regression, as in weighted True by another objective? AUC + MSE?

I'm using purchase data to build a cross-selling T/F response model for the banking industry, to score customers based on their likelihood to acquire a loan. On the other hand, I'm also building a ...
244
votes
6answers
376k views

What does AUC stand for and what is it?

Searched high and low and have not been able to find out what AUC, as in related to prediction, stands for or means.
0
votes
0answers
11 views

What scoring metric is optimal when performing hyperparameter optimization with a multiclass target variable?

I'm trying to find the optimal hyperparameter for different algorithms where the target variable has 3 classes. I was wondering if maximizing the average AUC over the 3 classes, which I'm currently ...
1
vote
0answers
29 views

How to interpret an AUC value of 0.55

I have a classifier with an AUC value of 0.55. I understand that an AUC value of 0.5 is essentially a random classifier and not predictive at all. So I am wondering if an AUC of 0.55 is ...
4
votes
1answer
902 views

How to plot ROC for knn (and potentially kernel spectral regression)

I understand how to plot ROC for logistic classifier (like varies the probability cutoff). For KNN, how can I find the ROC? Also, what about kernel spectral regression?
0
votes
1answer
22 views

Why is an ROC curve TPR (Y) against FPR (X)?

I am trying to thoroughly understand the ROC Curve and I was wondering why is an ROC Curve always (seemingly) TPR against FPR? I have had discussions with others about this matter and I cannot think ...
0
votes
0answers
7 views

How to understand the mathematics of ROC/AUC in Wikipedia [duplicate]

Could someone help explain what happened in the red box? I'm struggling to figure out where the indicator function comes from in this calculation. Thank you~ wiki link
0
votes
1answer
28 views

How to compare two or more mean AUC

I have different machine learning models and for each of them after a 10-fold cross validation I obtained a mean AUC (+-std). Now, How can I check if there is a statistically significant difference (...
3
votes
1answer
133 views

AUC variance for estimators with randomness (e.g. Random Forest)

I know that we can calculate the standard error for the AUC for all estimators, assuming that the conditional density is fixed. What I'd like to do, however, is additionally account for the randomness ...
1
vote
0answers
72 views

R: Caret package: AUC (“ROC”) standard error from cross validation

This seems like a basic questions, so I'm likely missing the big picture here... I would like to know the AUC of each fold of a cross-validation performed in Caret's train function, in order to ...
5
votes
2answers
11k views

How to improve F1 score with skewed classes?

I've a dataset of roughly 40K samples, with 39.6K samples belonging to the target class 0 and 400 to class 1. I've tried several classification algorithms, without too much fine tuning, just to get a ...
2
votes
1answer
31 views

Adding predictors in ROC curves and how does it affect AUC?

I have a general question about ROC curves and how adding predictors affect AUC values. Let's say I have a model that contains only predictor A and produces an AUC of 0.6. I then add into the model a ...
2
votes
0answers
65 views

ROC Convex Hull for Model Selection

The area under the convex hull of a roc curve is by construction always "better" than its area under curve. Some curves might see more of an increase in reported auc than others. Is it a viable ...
0
votes
1answer
23 views

Calculating AUC on test set of random forest model in R

I have built a random forest model within a k-fold cv, and have had no problem calculating the AUC on the training set, but after trying a few methods of calculating AUC I keep getting errors to the ...
1
vote
1answer
19 views

calculate Specificity and sensitivity from AUC

can someone please tell me how do you calculate Specificity and sensitivity from AUC. thank you.
0
votes
0answers
15 views

Feature Selection by individual AUC

I am creating a model for classification and I have several ways to get subset of features but I was wondering if the following is reasonable: Use the train set to calculate LOOCV or LPOCV AUC values ...
1
vote
1answer
309 views

Sensitivity/Specificity compared to Area Under Curve as measure of Screen Accuracy

In relation to the disorder I'm studying Screen A is reported as having a sensitivity of 90% and a specificity of 89%. Screen B is reported as having a AUC of .79 with no other data provided. Could ...
27
votes
3answers
28k views

ROC curve for discrete classifiers like SVM: Why do we still call it a “curve”?, Isn't it just a “point”?

In the discussion : how to generate a roc curve for binary classification, I think that the confusion was that a "binary classifier" (which is any classifier that separates 2 classes) was for Yang ...
1
vote
0answers
101 views

The usage of TOC curves?

In machine learning literature ROC curves are common performance measures. Quite recently published and not as popular (50+ citatations) are TOC curves as proposed by Pontius Jr, Robert Gilmore and Si,...
0
votes
0answers
23 views

ROC Curve AUC for Hypothesis Testing Sensitivity (Power) vs Specificity ($1-\alpha$)

Let's set up a hypothesis test of $H_0: \theta=\theta_0$ versus $H_1: \theta\ne\theta_0$, and let's say that I have two techniques to assess this (say equal-variance t-test versus unequal-variance ...
1
vote
0answers
34 views

Precision and recall for SVM from Confusion matrix is different from Precision-Recall graph

Coming from Stackoverflow- So, I am creating a SVM model for a highly imbalanced data set and trying to create to calculate F, Pression and recall from the confusion matrix of the model. Confusion ...
2
votes
1answer
744 views

Improve the precision of random forest for count data

I am trying to create a classification model that predicts whether a customer will enquire for a financial product based on some 250 independent variables. 98% of the variables are count variables and ...
0
votes
1answer
34 views

Feature Selection - Overfit?

I have a dataset of 100 patients and 1500 features. I split 80 train 20 test first and then use the train set to get the best hyperparameters / best feature subset doing the following: I randomly ...
2
votes
0answers
44 views

What is the appropriate use of the DeLong Test when comparing AUC of two different models tested on the same dataset?

In this scenario there are three datasets. Each dataset has 1000 images of an apple and 1000 images of a banana. Dataset A is used to train machine learning model A to classify whether the image is of ...
0
votes
1answer
37 views

Sampling distribution of AUC in multivariate normal distribution

I need to derive the sampling distribution of the above equation, which is sample mean of the AUC from what I see. The first picture describes the distribution of the data, which I understand is ...
2
votes
0answers
27 views

Odd results of AUC and McFadden R2 on insure tech article

I was reading this article entitled: Usage-Based Vehicle Insurance: Driving Style Factors of Accident Probability and Severity (Korishchenko et all., 2019) [1], and watching the results section, I saw ...
0
votes
0answers
15 views

Is it normal to have a high AUC in train test

I have Random Forest classification model which is already tunned using k-folds cross validation, when I train the model in the train set, the auc gives me 0.97, in test set is 0.75. Is it normal to ...
16
votes
3answers
14k views

Area under the ROC curve or area under the PR curve for imbalanced data?

I have some doubts about which performance measure to use, area under the ROC curve (TPR as a function of FPR) or area under the precision-recall curve (precision as a function of recall). My data is ...
0
votes
0answers
32 views

Why 1 out of 5 AUC cross validation score is very low?

I am using xgboost model for binary classification problem. I am using 5 fold cross-validation (stratified as class imbalance) which results into the following. ...
0
votes
0answers
29 views

AUC's standard deviation for a 10 x 10-cross validation test

In the following thesis http://arno.uvt.nl/show.cgi?fid=147278 the user compute the AUC standard deviation as measure of robustness. Let's say I have run a repeated (10) 10-cross validation ...
0
votes
0answers
18 views

Connection between prob output LogisticReg/SVM and ROC

I have the following ROC generated using LPOCV and Logistic regression or SVM (l2 norm). Now, let's say I have a test set containing 10 patients and I get that the probabilities of those patients to ...
0
votes
0answers
20 views

How to calculate the standard error of the area under the curve (AUC) of prognostic studies with 95% C.I

I have to do a meta-analysis of the area under the curve (AUC) of prognostic models. The AUC and 95% confidence intervals have been provided. I need the standard error of the AUC to proceed. How do I ...
1
vote
1answer
34 views

Imbalanced dataset - Majority positive class

My dataset consists of 150 patients where 50 are controls/healthy (negative) and 100 are sick (positive). If I want my model to have high sensitivity at high specificity (left side of the ROC), in ...
1
vote
2answers
338 views

How to generate ROC Plot for semi-supervised algorithm?

By having a data-set 1000 (900 unlabeled, 100 labeled) record data-set for binary classification, I want to apply a semi supervised algorithm. The problem is that I don't know how to get values for ...

1
2 3 4 5
9