Questions tagged [auc]
AUC stands for the Area Under the Curve and usually refers to the area under the receiver operator characteristic (ROC) curve.
0
votes
1answer
21 views
High AUC and Accuracy but weird output in confusion matrix
I am working on image classification problem to determine gender given a face. The dataset is located here gender face dataset on kaggle (link to my notebook). The class distribution is as follows.
<...
0
votes
1answer
26 views
Why there is decrease in AUC values with increase in number of iteration in Held out validation?
I have a dataset with 600 rows and 4000 columns for which I am trying to do held-out cross-validation with 10 and 100 iterations.
At first, the dataset is split into 80%:20% training and “held-out” ...
0
votes
0answers
12 views
Why is XGBoost prediction proba so concentrated within specific range? (unbalanced class)
I am pretty to new to Machine Learning. I am training on some past Kaggle competitions including the Santander Customer Satisfaction Challenge (https://www.kaggle.com/c/santander-customer-satisfaction)...
1
vote
1answer
22 views
Why is cross_val_score substantially lower than .score or roc_auc_score?
I have a trained model, a GradientBoostingClassifier. My dataset is 60 thousand something rows of data that I've split into 66/33 train/test sets. Scoring the model via the ...
1
vote
0answers
14 views
Sample size for AUC based on mean and SD of raw data
I'm trying to estimate the sample size required to achieve 80% power at a 5% significance level for a superiority study comparing AUC from bioavailability data.
My challenge is the only previous ...
0
votes
0answers
17 views
Computing Standard Error of AUC over different cv folds, and assessing statistical significance
I have a dataset in a healthcare setting for which the task is to predict a binary outcome, I have done this using the Support Vector Machine algorithm in a 5-fold cross validation setup. The ...
0
votes
0answers
8 views
Which performance metric to use for stratified data? [duplicate]
I'm trying to classify a data into 3 classes (supervised), one of which is heavily underrepresented in the data set. In order to combat this imbalance, I decided to stratify the data. Now I want to ...
0
votes
1answer
52 views
How to distinguish overfitting and underfitting from the ROC AUC curve?
For model selection, one of the metric is (AUC Area Under Curve) which tell us how the models are performing and based on AUC value we can choose the best model.
But how to distinguish whether a ...
0
votes
0answers
16 views
What is a Rank in AUC in a classification method?
I have read a post on AUC for which the link is given below
What does AUC stand for and what is it?
I am putting a new post as the above is rather old.
The person gave a few points related to Rank ...
0
votes
0answers
20 views
ada model- variables overall importance
I have the object ada from a model I didn't train to predict a binary result (I don't have the training set). Ada package was used. And the result are 200 binary trees.
I would like to have a ...
2
votes
1answer
35 views
K-NN optimal value of 'K'
How can I find the optimal value of 'K' in K-NN using the cross_val_score function, with scoring metric as auc_score?
Do I need ...
1
vote
1answer
23 views
AUC ROC when one class consists of smaller subclasses
This question is different from Binary classification when one class consists of multiple subclasses
I have two classes that I want to distinguish by a supervised learning classifier such as a random ...
2
votes
1answer
81 views
Does AUC/ROC curve return a p-value?
When reading this article, I noticed that the legend in Figure 3 gives a p-value for each AUC (Area Under the Curve) from the ROC (Receiver Operator Characteristic) curves. It says:
The area under ...
2
votes
2answers
47 views
Appropriate way to get Cross Validated AUC
I was thinking about cross-validation and how it is the most appropriate way to do it...
Let's take the case of binary logistic regression where the goal is to calculate the AUC.
Make the partition ...
0
votes
0answers
20 views
interprete and calculate Area Under the Curve of classifier in Matlab using perfcurve?
I hope that somebody could help me with the following question. I would like to calculate the Area Under the Curve (AUC) of a classifier (Linear Discrimiant Analyses). To do this, I am using the ...
1
vote
0answers
37 views
What is the integral of the False Positive Rate over the False Positive Rate, compared to the AUC?
In machine learning the Area Under the Receiver Operating Characteristic Curve ($AUC$) can be illustrated in a plot of the True Positive Rate ($TPR$) against the False Positive Rate ($FPR$). Formally, ...
2
votes
0answers
23 views
How to calculate the standard error of ratio of two AUC
Please can anyone help me with this!
I hope my explanation is going to be simple:
This is the data I have:
I only have access to the aggregated data as below. the first column is the time points a ...
1
vote
1answer
94 views
How do I calculate AUC with leave-one-out CV
In a binary response setting (data matrix D with N rows) I have performed LOOCV and obtained a final lambda*. The average CV error for this lambda* is also, as I understand it, an unbiased estimator ...
0
votes
0answers
12 views
High accuracy for single classes & low accuracy for multiclass classification
I was trying to do a multiclass-classification, where each sample belongs to one of the four classes. Now that I have a probability vector $(p_1^i,p_2^i,p_3^i,p_4^i)$ for each sample $i$ as my ...
2
votes
2answers
64 views
Can I trust my random forest model with low sample size and unequal class distribution?
I have a general question regarding model evaluation for random forest with low sample size and unequal class distribution.
I am doing some explorative modeling by using 400 features to classify ...
0
votes
0answers
15 views
AUC/R^2 score strictly lower on partitioned dataset than total dataset
Suppose I've fit a logistic regression to my data and I calculate the AUC. I then partition my dataset according to one of the explanatory variables (say those rows where the variable value is greater/...
0
votes
0answers
41 views
hierarchical bootstrapping and calculation of variance (in a Random-Effects ROC Analysis) in R
I would like to calculate the variance of the AUC of readers (for each reader and averaged results) giving a score(1-5) to ...
2
votes
1answer
77 views
Reason for higher AUC from a test set than a training set using a random forest
I made a 70:30 split of the data to build a random forest model for binary classification. Although the prevalence of $Y=1$ was about 25% in both training and test sets, the two sets became imbalanced ...
1
vote
1answer
28 views
Why are ROC curves and AUC values not always relevant?
So I have read in posts and in literature (Frank Harrel - Regression Modeling
Strategies fx) that depending on what you do, ROC curves and AUC values are not always relevant, but often written in ...
1
vote
1answer
27 views
ROC curve: models have different FPR ranges
I've created ROC curves by calculating the TPR and FPR at various thresholds. The FPR range differs between models, so I'm wondering if AUC is still a valid way to compare the curves. A curve will ...
1
vote
1answer
70 views
How do iso accuracy line related to ROC curve
I have read many articles about ROC curve. Some specified a method to calculate the accuracy of a classifier using iso accuracy lines in a convex hull.
Just like in this articles :
http://mlwiki.org/...
2
votes
2answers
117 views
Why AUC is not a good performance metric for a classification model?
After understanding the benefits of AUC
I was stumbled to know that in some scenarios it might not be a good performance metric for evaluating a classification model. The below are the 2 scenarios:
...
2
votes
1answer
37 views
how to interpret the behavior of a ROC curve very close to one
annex the ROC curve of a logit model, which has a very high AUC, how can I interpret the behavior of this curve, I doubt the logit model?
0
votes
0answers
45 views
AUC and accuracy interpretation
An accuracy $80\%$ of a model that predicts binary outcomes is interpreted as:
Given a sample whose outcomes we want to predict, 80% of the prediction will be correct.
What does an AUC of $80\%$...
2
votes
1answer
66 views
Why are the trivial points included when calculating AUC?
I'm aware of some of the issues associated with using AUC for model comparison (see for example the articles referenced on Wikipedia: here, here, or here). But so far I have found nothing on an issue ...
0
votes
2answers
881 views
What is the formula to calculate the area under the ROC curve from a contingency table?
For example, if my table is:
...
2
votes
1answer
172 views
How to Interpret AUROC score?
My model has an AUROC value of 0.7, and I have a 75:25 class (75% negative, 25% positive) imbalance. From my understanding, AUROC is calculated by using different thresholds for considering the ...
0
votes
0answers
30 views
Insignificant t test or MW U test yet high ROC AUC and vice versa
In SPSS, when I conduct a Student's t test or Mann-Whitney U test on (lots of) variables when comparing between 2 groups, some differences are denoted significant, and others aren't.
When I conduct ...
1
vote
2answers
124 views
Do you need to calculate sample size to evaluate a new diagnostic test?
I am writing a grant application which will be evaluating a new diagnostic test. The test will predict whether a patient with lung fibrosis will remain stable or progress. I am using an existing ...
0
votes
2answers
42 views
Performance evaluation
I'd like to test the performance of a penalized regression. I did three separate regressions for each response variable (one numerical, one binomial and one multinomial). I was checking this link, and ...
3
votes
0answers
32 views
Confidence intervals of AUC obtained by merging/pooling predictions from different test sets
I have one question regarding the CIs of the AUROC calculated merging/pooling the predictions coming from different test sets.
In one analysis, we use a sort of nested cross-validation approach, ...
1
vote
0answers
32 views
How to make really bad results from a machine learning model better by reversing predictions
I trained a classification model on some data with two classes and have really low accuracy. I have a false-positive rate of 86 % for both classes I am trying to predict. I was wondering if I could ...
0
votes
0answers
14 views
improve model roc_auc score
From the GridSearchcv on a random forest classifier, the best parameters is giving me an auc_roc score of 0.80. But when i train a new random forest model with the best parameters i am getting an ...
0
votes
0answers
21 views
which recall value to plot for same precision in PR curve?
Suppose, after sorting the true labels by the corresponding classifier scores, we obtain the following:
$$[False, True, False, True, True, True, False, False],$$
which leads to the following points ...
1
vote
0answers
24 views
How to determine the optimal threshold for my classification problem using fpr ,tpr values for each classification record?
The output of my prediction using classification algorithm is in dataframe that contains TPR and FPR value for each prediction. Suppose I have 100 records for prediction then in that cade my data ...
0
votes
0answers
22 views
Directly optimizing over AUC surrogate
This is in relation to this paper
I am looking for ways to optimize Recall @ fixed Precision ($R@P$) for a machine learning problem and i didnt want to use accuracy as a proxy for $R@P$. Upon ...
0
votes
0answers
30 views
How to prove that a surrogate cost function is lower bound to original cost function?
This is very specific to a research paper that have been reading recently:
It is about constructing cost functions that are more correlated to non-decomposable (cannot be broken down to a summation ...
0
votes
0answers
41 views
Density curve in R - AUC bigger than 1
correct me if I'm wrong but I was expecting the area under the curve should be 1 for a probability density function.
Can anybody explain why it's not always the case when using the ...
1
vote
0answers
206 views
Micro- or macro-averaged AUC for highly imbalanced data?
I have a classification problem with 3 classes. With random forest classifier I'm getting the following confusion matrix:
The micro-averaged AUC is 0.76 and the macro-averaged AUC is 0.55.
On the ...
1
vote
0answers
89 views
How to interpret ROC curve with confusion matrix and F1 score?
I have implemented a random forest classifier to do a binary classification in highly imbalanced class. As the performance measurements, ROC and the f1 score was considered. However, the ROC curve ...
1
vote
1answer
55 views
Is it possible for a model to have higher sensitivity/specificity but lower accuracy and AUC?
In the evaluation of classification models, I've found one model to have a higher accuracy and c-statistic (AUC) as compared to a second model. However, the second model has higher sensitivity, ...
1
vote
0answers
34 views
How to derive the the AUROC from the Bayes Minimum Risk (Hand 2009)?
The area-under-the-receiver-operating-characteristic-curve (AUROC) can be derived from the Bayes Minimum Risk. The derivation requires the assumption that the exact costs are unknown but follow a ...
0
votes
0answers
81 views
AUC and Accuracy baseline
I implemented the different classification algorithms like Bayesian network, Decision tree or Naive Bayes, on my data to predict the right class (binary classes). By considering confusion matrix, I ...
1
vote
0answers
55 views
Poor P-R curve for binary classifier trained on balanced data, with imbalanced test data
I have a very imbalanced dataset (9:1), for which I have performed under-sampling and achieved a balanced training set (~130k samples total post balancing).
I am performing classification using ...
1
vote
3answers
124 views
When is an AUC score misleadingly high?
I have an algorithm which gives an AUC (area under the receiver operating curve) of 0.94.
I mean, this is amazing, but... probably too amazing, considering the difficulty of the task I am working on. ...
