Questions tagged [auc]
AUC stands for the Area Under the Curve and usually refers to the area under the receiver operator characteristic (ROC) curve.
613
questions
1
vote
0
answers
12
views
RecSys model performance stalling at 47% AUC and F1-Score. Is the problem due to ratio of users to items in my dataset?
I'm having trouble with making my validation metrics go down for the binary_crossentropy and go up for the F1-score and AUC. I've tried tuning my hyper parameters such as the number of latent features ...
2
votes
1
answer
42
views
Delong's test for comparing the significance difference of two AUC
I have done two prediction model in R as an example as below:
...
- 129
0
votes
0
answers
15
views
De Longs test on ROC curves?
I have two different ROC curves for Model A and B. I wanted to calculate the DeLongs test to identify the statisical significance of the difference between the model.
model_A_tpr = [0, 0.2, 0.4, 0.6, ...
- 101
-1
votes
1
answer
37
views
difference between c-statistics and AUC for prediction performance
I have two types of outcome from a big dataset and I performed prediction models and checked the prediction performance using 10-fold cross-validation, so:
for binary outcome: lasso logistic ...
- 129
2
votes
3
answers
90
views
cutoff and auc and changing cutoff
can you tell me if this is ok?
While the AUC (i.e. AUC of 0.6) we got is acceptable since it's bigger than 0.5, we may need to re-evaluate at our cutoff selections again. Because we can select cutoffs ...
- 21
0
votes
0
answers
23
views
P-value for AUC of logistic regression model vs AUC=0.50, how?
In R studio, I am running a logistic regression model with a binary variable vs binary outcome. I get a AUC and a AUC confidence interval. Now my supervisor wants a p-value to show which auc are ...
0
votes
0
answers
14
views
C-statistic (or AUC) for fractional logistic regression (i.e. continuous regression)
I have proportional data to which I have fit a logistic regression (i.e. fractional logistic regression). The statistician in our group wants me to provide a c-statistic for the regression.
My ...
- 326
14
votes
3
answers
243
views
Area under the ROC curve when there is imbalance: is there a problem, and if not, why does this rumor exist?
Cross Validated has a rather thorough debunking of class imbalance being an inherent problem that must be fixed in order to do quality predictive modeling of categorical outcomes [1, 2]. However, ...
- 55.2k
0
votes
0
answers
6
views
Use of area under the curve to compare treatments with count ordinal data over time
I am working on questionnaire data, looking into bimonthly responses where "improvement" and "worsening" for different health parameters is recorded as counts or frequencies for a ...
- 1
1
vote
0
answers
40
views
C-statistic vs AUC [closed]
I am analysing diagnostic accuracy. I have a dataset with a ground truth and 3 predictors.
Ground truth = binary (0/1)
Predictor 1-2 = binary (0/1)
Predictor 3 = continuous (0-100)
I have 50,000 ...
- 79
0
votes
0
answers
17
views
Question regarding deriving the AUC value
I have a dataset (auc_data) which includes the response variable (y) and the predicted probabilities (y_hat). I have calculated the AUC value based on this data.
You can see that there are 3 rows in ...
- 869
0
votes
0
answers
46
views
Power/sample size estimation when comparing two AUCs (area under the curve)?
Are there any R functions (or other free software) for calculating power/sample size needed to compare 2 AUCs (area under the curve)?
Specifically, suppose you want to fit 2 different models to a ...
- 1
1
vote
1
answer
64
views
Why use average_precision_score from sklearn? [duplicate]
I have precision and recall values and want to measure an estimator performance:
...
- 31
0
votes
0
answers
9
views
Can correlating features change the AUROC of a model trained on randomly shuffled labels?
I did a binary classification with a random forest on my dataset, where I removed all correlating features and figured I'd try out a sort of "negative control" and shuffle all labels ...
- 1
0
votes
0
answers
8
views
Mean AUC vs Mean Average Precision?
Both AUC and Average Precision (AP) are commonly used to evaluate the quality of a ranked list of relevant / irrelevant documents. When evaluating a search engine, we want to evaluate the quality over ...
- 1,157
1
vote
1
answer
120
views
Confused about AUC of 0 from a logistic regression classifier with a non-informative predictor
I am using logistic regression as a classifier to predict the variable Group (old/young) from a variable called Sentence_length. When I plot density functions of Sentence_length by Group, it is clear ...
- 177
0
votes
0
answers
12
views
Compare AUC and LDA results
I have a dataset with multiple variables (~20) that I could use in order to diagnose an illness. In my datset, I know if the individuals are sick or not.
I intend to create a model using ...
- 41
0
votes
1
answer
16
views
Interpretation of ROC curve curving early
I am running a binary LASSO logistic regression using glmnet. The initial data I work with is raster spatial data. When I create an ROC (AUC ~ 0.72) curve based on the test data, the resulting curve ...
Jason Connelly
2
votes
1
answer
76
views
ROC-AUC in GEE models?
I am wondering if it is possible to make ROC-AUC curves for GEE models? I found few papers who did that and it wasn't clear for me. I thought it was impossible given how they are marginal models. ...
- 31
0
votes
0
answers
20
views
AUC for Multi-Label Classification using SVM
I am tackling a multi-label classification problem and I want to choose a SVM model maximising the AUC. I am not sure if AUC can be used in this case and if yes it is sufficient just to change the ...
1
vote
0
answers
34
views
Combination of AUC for different subset of samples
Assuming I have ROC-AUC for multiple subsets of samples, is there a way to produce a "weighted" average of AUC or something that will represent/approximate the AUC for the complete ...
- 360
0
votes
0
answers
16
views
Consistent very low (<0.5) AUC preformance of pipeline, cannot explain why
I have a 2-step ML pipeline (classification, binary, balanced dataset, about 300 samples and tens of thousands of features): step one I train 7 algorithms (XGB, SVM etc.) and step two I train a RFC on ...
- 211
1
vote
0
answers
97
views
Interpretation of area under the precision-recall curve
The area under the receiver-operator characteristic curve has a interpretation of how well the predictions of two categories are separated.
This post gives the area under the precision-recall curve as ...
- 55.2k
8
votes
6
answers
2k
views
Where in the ROC curve does it tell you what the threshold is?
In my understanding, the ROC curve plots the True positive rate and the False positive rate.
However, I've also read in other places that the ROC curve helps determine where the threshold for ...
- 704
1
vote
0
answers
50
views
Using the Bootstrap in conjunction with Cross Validation for Confidence Intervals
I am trying to estimate confidence bounds for prediction metrics of a binary-outcome logistic regression model. In particular, the AUC and the Brier score.
I've looked at many other posts on this ...
1
vote
1
answer
43
views
Surprising disparity between Confusion matrix values and AUC?
I'm working on getting a read out of a Logistic regression classification model (setup in Python via Scikit-learn's LogisticRegression() wrapped in a OneVsRestClassifier()). I got the confusion matrix ...
2
votes
1
answer
23
views
How to find AUC from Binary Classification Decision Tree?
Decision Tree
I have found Misclassification rates for all the leaf nodes.
samples = 3635 + 1101 = 4736, class = Cash, misclassification rate = 1101 / 4736 = 0.232.
samples = 47436 + 44556 = 91992, ...
1
vote
0
answers
19
views
Calculating ROC AUC for a fair coin by definition. Where's the mistake?
Here's the formula for a continuous case (taken from this great answer):
\begin{align}
\text{AUC}
&= \int_{-\infty}^{+\infty} \big( 1-F_1(\tau) \big) f_0(\tau) d\tau
\end{align}
Since a coin ...
- 11
0
votes
0
answers
170
views
AUC values of training and cross-validation are lower than AUC values of test set
I am training a Full model (logistic regression) and a few different models (LASSO, Elastic net, CART, random forest) to predict a certain clinical outcome.
I split my data into training and test sets ...
- 71
0
votes
0
answers
256
views
xgboost algorithm is giving excellent auc score on train, validation and test datasets (0.94) but giving worst public score (0.5) after submission
I am a beginner in ML modelling. I am working on Santander Customer Satisfaction Prediction competition and the evaluation metric is AUC. The dataset has 370 features and the target variable is TARGET ...
1
vote
1
answer
215
views
Comparing the AUC of two models by using a combination of nested cross-validation and bootstrapping
Main question:
I have an imbalanced binary labeled dataset (6% positive labels) and two different methods of training a predictive model for binary classification (e.g. Tree Model vs. Neural Network), ...
- 33
0
votes
0
answers
32
views
What does it mean if optimal classification threshold found on ROC curve is really small?
I've trained a simple NN to perform binary classification with goal of maximizing area under ROC curve. Right now AUC is around 0.85. Out of curiosity, I checked which thresholds are best in terms of ...
- 211
2
votes
1
answer
74
views
Can I calculate ROC AUC from Mann Whitney U tests when I am comparing unequal sample sizes?
I know I am able to calculate Mann Whitney U tests when comparing 2 samples unequal inside but I am wondering if I am able to carry this same principle when calculating ROC AUC via the formula:
AUC = ...
- 21
3
votes
1
answer
192
views
ROC AUC has $0.5$ as random performance. Does PR AUC have a similar notion?
In considering ROC AUC, there is a sense in which $0.5$ is the performance of a random model. Conveniently, this is true, no matter the data or the prior probability of class membership; the ROC AUC ...
- 55.2k
0
votes
0
answers
25
views
ROC analysis with a fuzzy reference standard with estimates of its reliability
I‘m conducting ROC analyses in order to assess the diagnostic accuracy (AUC, sensitivity and specificity for certain cut-offs) of multiple index tests (ordinal scaled measures). The goal is to compare ...
3
votes
1
answer
133
views
Showing the difference between two models with similar AUC-ROC curves
I have a plot of ROC curves for about 5 models. The curves are overlapping, as shown in the attached figure.
Is there a way to still call out the differences between these models in a research paper ...
- 33
0
votes
0
answers
27
views
The correlation between ROC and AUC curve
For multi-classification. The ROC curve not completely covers the upper-left corner of the plot, why the AUC is equal to 1. The result of the testing set is completely classified with an accuracy ...
0
votes
0
answers
37
views
Correctly evaluating unsupervised learning model
I am trying to compare various unsupervised machine learning models to detect anomalous water consumption in each user's house. Now I have 10 datasets (minutely data, no anomalous points) that have no ...
- 101
0
votes
0
answers
72
views
ROC with bootstraping
I have a data with 2 variables: diagnosis- yes/no
Score- numeric variable from 0-10.
I need to do ROC analysis for this data and to find the best cut off values.
The problem is the data is too small ...
4
votes
2
answers
209
views
ROC/AUC Curve for False Negatives (Type 2 Errors)?
I keep seeing this curve and I understand this basically tells you how well the model is doing in terms of predicting True positives vs. False positives.
I was wondering if there is a version of this ...
- 704
0
votes
0
answers
16
views
AUC ROC validity merge experiments
Following situation:
I want to discuss my results. I repeated an experiment for 3 times for a binary classifier for validity. Now I want to draw a ROC-AUC curve.
What I do not want to do: I do not ...
- 33
0
votes
0
answers
32
views
ROC AUC interpretation [duplicate]
When reading about AUC/ROC there is often this notion that comes up that a perfect score is 1.0. I have two problems with that:
There can be models that are 100% accurate but do not have an AUC of 1....
- 33
2
votes
1
answer
60
views
AUC - different interpretations
When browsing through literature about ROC - AUC, there seems to be a disparity.
While some plot TPR and FPR, e.g. from Wikipedia: "The ROC curve is created by plotting the true positive rate (...
- 33
0
votes
0
answers
25
views
Why does different bagging fractions lead to the same result?
I am using R to subset a few data samples with a random seed beforehand. However, by trying different values of bagging fraction, the results somehow are the same. The lgbm model (lgbm.mod) is ...
- 41
0
votes
0
answers
15
views
Reading and Interpreting two models performance from this ROC
This question is for educational goal.
I trained a KNN with a public diabetes dataset
If it was for developing a new diabetes test which model would you chose? Why?
How should interpret the point ...
- 111
0
votes
0
answers
60
views
Can someone explain Weighted ROC curves? and how do I chose the weights?
Thanks to this blog what is AUC, I got a clear understanding of AUC and how its calculated. However I fail to understand how are weighted AUC calculated?
I have read somewhere that "weighted ROC ...
- 101
3
votes
1
answer
50
views
Built three logit models, one with oversampling, undersampling, and one without sampling. My AUC is the same for all three. Why is that? [duplicate]
I used the ROSE package in R to balance a dataset. I wasn't sure which would yield better results so I split my data into training and test sets (75/25) then over and undersampled my sample before ...
- 33
2
votes
1
answer
94
views
How to interpret the overlap of AUC-CI(95%) of two logistic regression models
Here is my question. I am comparing two logistic regression models which use the same data and endpoint but different variables. My goal is to see if the model improves based on the performance using ...
0
votes
0
answers
19
views
Stratifying the performance of a classifier
I have trained a classifier and evaluated my classifier's performance on the testing set by Area Under the Precision-recall curve.
My testing set comes from 2000 different categories, and my ...
0
votes
1
answer
362
views
How to get an AUC plot for Cox Regression?
This is a figure for survival analysis on a dataset. The article is found at this link
I am self teaching survival analysis. Up until recently, I believed that AUC and ROC were only for classification....
- 403