Questions tagged [auc]

AUC stands for the Area Under the Curve and usually refers to the area under the receiver operator characteristic (ROC) curve.

Filter by
Sorted by
Tagged with
1 vote
0 answers
12 views

RecSys model performance stalling at 47% AUC and F1-Score. Is the problem due to ratio of users to items in my dataset?

I'm having trouble with making my validation metrics go down for the binary_crossentropy and go up for the F1-score and AUC. I've tried tuning my hyper parameters such as the number of latent features ...
Mig Rivera Cueva's user avatar
2 votes
1 answer
42 views

Delong's test for comparing the significance difference of two AUC

I have done two prediction model in R as an example as below: ...
user358238's user avatar
0 votes
0 answers
15 views

De Longs test on ROC curves?

I have two different ROC curves for Model A and B. I wanted to calculate the DeLongs test to identify the statisical significance of the difference between the model. model_A_tpr = [0, 0.2, 0.4, 0.6, ...
a_parida's user avatar
  • 101
-1 votes
1 answer
37 views

difference between c-statistics and AUC for prediction performance

I have two types of outcome from a big dataset and I performed prediction models and checked the prediction performance using 10-fold cross-validation, so: for binary outcome: lasso logistic ...
user358238's user avatar
2 votes
3 answers
90 views

cutoff and auc and changing cutoff

can you tell me if this is ok? While the AUC (i.e. AUC of 0.6) we got is acceptable since it's bigger than 0.5, we may need to re-evaluate at our cutoff selections again. Because we can select cutoffs ...
Shawn Kim's user avatar
0 votes
0 answers
23 views

P-value for AUC of logistic regression model vs AUC=0.50, how?

In R studio, I am running a logistic regression model with a binary variable vs binary outcome. I get a AUC and a AUC confidence interval. Now my supervisor wants a p-value to show which auc are ...
DerekJones5's user avatar
0 votes
0 answers
14 views

C-statistic (or AUC) for fractional logistic regression (i.e. continuous regression)

I have proportional data to which I have fit a logistic regression (i.e. fractional logistic regression). The statistician in our group wants me to provide a c-statistic for the regression. My ...
Brandon's user avatar
  • 326
14 votes
3 answers
243 views

Area under the ROC curve when there is imbalance: is there a problem, and if not, why does this rumor exist?

Cross Validated has a rather thorough debunking of class imbalance being an inherent problem that must be fixed in order to do quality predictive modeling of categorical outcomes [1, 2]. However, ...
Dave's user avatar
  • 55.2k
0 votes
0 answers
6 views

Use of area under the curve to compare treatments with count ordinal data over time

I am working on questionnaire data, looking into bimonthly responses where "improvement" and "worsening" for different health parameters is recorded as counts or frequencies for a ...
Colbie's user avatar
  • 1
1 vote
0 answers
40 views

C-statistic vs AUC [closed]

I am analysing diagnostic accuracy. I have a dataset with a ground truth and 3 predictors. Ground truth = binary (0/1) Predictor 1-2 = binary (0/1) Predictor 3 = continuous (0-100) I have 50,000 ...
user12541161's user avatar
0 votes
0 answers
17 views

Question regarding deriving the AUC value

I have a dataset (auc_data) which includes the response variable (y) and the predicted probabilities (y_hat). I have calculated the AUC value based on this data. You can see that there are 3 rows in ...
student_R123's user avatar
0 votes
0 answers
46 views

Power/sample size estimation when comparing two AUCs (area under the curve)?

Are there any R functions (or other free software) for calculating power/sample size needed to compare 2 AUCs (area under the curve)? Specifically, suppose you want to fit 2 different models to a ...
arbetj's user avatar
  • 1
1 vote
1 answer
64 views

Why use average_precision_score from sklearn? [duplicate]

I have precision and recall values and want to measure an estimator performance: ...
Ars ML's user avatar
  • 31
0 votes
0 answers
9 views

Can correlating features change the AUROC of a model trained on randomly shuffled labels?

I did a binary classification with a random forest on my dataset, where I removed all correlating features and figured I'd try out a sort of "negative control" and shuffle all labels ...
MWeb's user avatar
  • 1
0 votes
0 answers
8 views

Mean AUC vs Mean Average Precision?

Both AUC and Average Precision (AP) are commonly used to evaluate the quality of a ranked list of relevant / irrelevant documents. When evaluating a search engine, we want to evaluate the quality over ...
usual me's user avatar
  • 1,157
1 vote
1 answer
120 views

Confused about AUC of 0 from a logistic regression classifier with a non-informative predictor

I am using logistic regression as a classifier to predict the variable Group (old/young) from a variable called Sentence_length. When I plot density functions of Sentence_length by Group, it is clear ...
Galit's user avatar
  • 177
0 votes
0 answers
12 views

Compare AUC and LDA results

I have a dataset with multiple variables (~20) that I could use in order to diagnose an illness. In my datset, I know if the individuals are sick or not. I intend to create a model using ...
Dimitri's user avatar
  • 41
0 votes
1 answer
16 views

Interpretation of ROC curve curving early

I am running a binary LASSO logistic regression using glmnet. The initial data I work with is raster spatial data. When I create an ROC (AUC ~ 0.72) curve based on the test data, the resulting curve ...
user avatar
2 votes
1 answer
76 views

ROC-AUC in GEE models?

I am wondering if it is possible to make ROC-AUC curves for GEE models? I found few papers who did that and it wasn't clear for me. I thought it was impossible given how they are marginal models. ...
Youknowme's user avatar
0 votes
0 answers
20 views

AUC for Multi-Label Classification using SVM

I am tackling a multi-label classification problem and I want to choose a SVM model maximising the AUC. I am not sure if AUC can be used in this case and if yes it is sufficient just to change the ...
data_miner_frbg's user avatar
1 vote
0 answers
34 views

Combination of AUC for different subset of samples

Assuming I have ROC-AUC for multiple subsets of samples, is there a way to produce a "weighted" average of AUC or something that will represent/approximate the AUC for the complete ...
Gideon Kogan's user avatar
0 votes
0 answers
16 views

Consistent very low (<0.5) AUC preformance of pipeline, cannot explain why

I have a 2-step ML pipeline (classification, binary, balanced dataset, about 300 samples and tens of thousands of features): step one I train 7 algorithms (XGB, SVM etc.) and step two I train a RFC on ...
SebDL's user avatar
  • 211
1 vote
0 answers
97 views

Interpretation of area under the precision-recall curve

The area under the receiver-operator characteristic curve has a interpretation of how well the predictions of two categories are separated. This post gives the area under the precision-recall curve as ...
Dave's user avatar
  • 55.2k
8 votes
6 answers
2k views

Where in the ROC curve does it tell you what the threshold is?

In my understanding, the ROC curve plots the True positive rate and the False positive rate. However, I've also read in other places that the ROC curve helps determine where the threshold for ...
Katsu's user avatar
  • 704
1 vote
0 answers
50 views

Using the Bootstrap in conjunction with Cross Validation for Confidence Intervals

I am trying to estimate confidence bounds for prediction metrics of a binary-outcome logistic regression model. In particular, the AUC and the Brier score. I've looked at many other posts on this ...
Valerie Ramos's user avatar
1 vote
1 answer
43 views

Surprising disparity between Confusion matrix values and AUC?

I'm working on getting a read out of a Logistic regression classification model (setup in Python via Scikit-learn's LogisticRegression() wrapped in a OneVsRestClassifier()). I got the confusion matrix ...
AnimNeuroGrad's user avatar
2 votes
1 answer
23 views

How to find AUC from Binary Classification Decision Tree?

Decision Tree I have found Misclassification rates for all the leaf nodes. samples = 3635 + 1101 = 4736, class = Cash, misclassification rate = 1101 / 4736 = 0.232. samples = 47436 + 44556 = 91992, ...
Aman Rangapur's user avatar
1 vote
0 answers
19 views

Calculating ROC AUC for a fair coin by definition. Where's the mistake?

Here's the formula for a continuous case (taken from this great answer): \begin{align} \text{AUC} &= \int_{-\infty}^{+\infty} \big( 1-F_1(\tau) \big) f_0(\tau) d\tau \end{align} Since a coin ...
forveg's user avatar
  • 11
0 votes
0 answers
170 views

AUC values of training and cross-validation are lower than AUC values of test set

I am training a Full model (logistic regression) and a few different models (LASSO, Elastic net, CART, random forest) to predict a certain clinical outcome. I split my data into training and test sets ...
SAN's user avatar
  • 71
0 votes
0 answers
256 views

xgboost algorithm is giving excellent auc score on train, validation and test datasets (0.94) but giving worst public score (0.5) after submission

I am a beginner in ML modelling. I am working on Santander Customer Satisfaction Prediction competition and the evaluation metric is AUC. The dataset has 370 features and the target variable is TARGET ...
Sownik Turaga's user avatar
1 vote
1 answer
215 views

Comparing the AUC of two models by using a combination of nested cross-validation and bootstrapping

Main question: I have an imbalanced binary labeled dataset (6% positive labels) and two different methods of training a predictive model for binary classification (e.g. Tree Model vs. Neural Network), ...
Valo's user avatar
  • 33
0 votes
0 answers
32 views

What does it mean if optimal classification threshold found on ROC curve is really small?

I've trained a simple NN to perform binary classification with goal of maximizing area under ROC curve. Right now AUC is around 0.85. Out of curiosity, I checked which thresholds are best in terms of ...
Brzoskwinia's user avatar
2 votes
1 answer
74 views

Can I calculate ROC AUC from Mann Whitney U tests when I am comparing unequal sample sizes?

I know I am able to calculate Mann Whitney U tests when comparing 2 samples unequal inside but I am wondering if I am able to carry this same principle when calculating ROC AUC via the formula: AUC = ...
jamberhee's user avatar
3 votes
1 answer
192 views

ROC AUC has $0.5$ as random performance. Does PR AUC have a similar notion?

In considering ROC AUC, there is a sense in which $0.5$ is the performance of a random model. Conveniently, this is true, no matter the data or the prior probability of class membership; the ROC AUC ...
Dave's user avatar
  • 55.2k
0 votes
0 answers
25 views

ROC analysis with a fuzzy reference standard with estimates of its reliability

I‘m conducting ROC analyses in order to assess the diagnostic accuracy (AUC, sensitivity and specificity for certain cut-offs) of multiple index tests (ordinal scaled measures). The goal is to compare ...
soelderer's user avatar
3 votes
1 answer
133 views

Showing the difference between two models with similar AUC-ROC curves

I have a plot of ROC curves for about 5 models. The curves are overlapping, as shown in the attached figure. Is there a way to still call out the differences between these models in a research paper ...
cidi30bg's user avatar
0 votes
0 answers
27 views

The correlation between ROC and AUC curve

For multi-classification. The ROC curve not completely covers the upper-left corner of the plot, why the AUC is equal to 1. The result of the testing set is completely classified with an accuracy ...
user381137's user avatar
0 votes
0 answers
37 views

Correctly evaluating unsupervised learning model

I am trying to compare various unsupervised machine learning models to detect anomalous water consumption in each user's house. Now I have 10 datasets (minutely data, no anomalous points) that have no ...
Captain Jacky's user avatar
0 votes
0 answers
72 views

ROC with bootstraping

I have a data with 2 variables: diagnosis- yes/no Score- numeric variable from 0-10. I need to do ROC analysis for this data and to find the best cut off values. The problem is the data is too small ...
Inbar Lavie's user avatar
4 votes
2 answers
209 views

ROC/AUC Curve for False Negatives (Type 2 Errors)?

I keep seeing this curve and I understand this basically tells you how well the model is doing in terms of predicting True positives vs. False positives. I was wondering if there is a version of this ...
Katsu's user avatar
  • 704
0 votes
0 answers
16 views

AUC ROC validity merge experiments

Following situation: I want to discuss my results. I repeated an experiment for 3 times for a binary classifier for validity. Now I want to draw a ROC-AUC curve. What I do not want to do: I do not ...
Stani Petrov's user avatar
0 votes
0 answers
32 views

ROC AUC interpretation [duplicate]

When reading about AUC/ROC there is often this notion that comes up that a perfect score is 1.0. I have two problems with that: There can be models that are 100% accurate but do not have an AUC of 1....
Stani Petrov's user avatar
2 votes
1 answer
60 views

AUC - different interpretations

When browsing through literature about ROC - AUC, there seems to be a disparity. While some plot TPR and FPR, e.g. from Wikipedia: "The ROC curve is created by plotting the true positive rate (...
Stani Petrov's user avatar
0 votes
0 answers
25 views

Why does different bagging fractions lead to the same result?

I am using R to subset a few data samples with a random seed beforehand. However, by trying different values of bagging fraction, the results somehow are the same. The lgbm model (lgbm.mod) is ...
Phoebe's user avatar
  • 41
0 votes
0 answers
15 views

Reading and Interpreting two models performance from this ROC

This question is for educational goal. I trained a KNN with a public diabetes dataset If it was for developing a new diabetes test which model would you chose? Why? How should interpret the point ...
Andrea Ciufo's user avatar
0 votes
0 answers
60 views

Can someone explain Weighted ROC curves? and how do I chose the weights?

Thanks to this blog what is AUC, I got a clear understanding of AUC and how its calculated. However I fail to understand how are weighted AUC calculated? I have read somewhere that "weighted ROC ...
Kaushik J's user avatar
  • 101
3 votes
1 answer
50 views

Built three logit models, one with oversampling, undersampling, and one without sampling. My AUC is the same for all three. Why is that? [duplicate]

I used the ROSE package in R to balance a dataset. I wasn't sure which would yield better results so I split my data into training and test sets (75/25) then over and undersampled my sample before ...
snerbs's user avatar
  • 33
2 votes
1 answer
94 views

How to interpret the overlap of AUC-CI(95%) of two logistic regression models

Here is my question. I am comparing two logistic regression models which use the same data and endpoint but different variables. My goal is to see if the model improves based on the performance using ...
Keith Sanders's user avatar
0 votes
0 answers
19 views

Stratifying the performance of a classifier

I have trained a classifier and evaluated my classifier's performance on the testing set by Area Under the Precision-recall curve. My testing set comes from 2000 different categories, and my ...
poorya mirzavand's user avatar
0 votes
1 answer
362 views

How to get an AUC plot for Cox Regression?

This is a figure for survival analysis on a dataset. The article is found at this link I am self teaching survival analysis. Up until recently, I believed that AUC and ROC were only for classification....
Antonio's user avatar
  • 403

1
2 3 4 5
13