Questions tagged [auc]
AUC stands for the Area Under the Curve and usually refers to the area under the receiver operator characteristic (ROC) curve.
2
votes
1answer
42 views
What is metrics.roc_curve and metrics.auc measuring when I'm comparing binary data with probability estimates?
I was working on a challenge, and I was excited because the metric.auc for my predicted values compared to my test values was very high. This was for a binary selection process.
However, when I ...
0
votes
1answer
22 views
Practical interpretation of Precision-Recall AUC
I have a classifier with an AUC (PR) of 0.06 which I will use for a practical interpretation.
My test set consists of three months of data with a total of 2,200,000 observations of which 0.03 are ...
0
votes
0answers
15 views
compute AUC and statistical testing of stratified data
I have a dataset where samples are stratified in groups. That is, there are N samples per examination, and M examinations per subject.
I would like to account for this when estimating the ...
2
votes
1answer
45 views
Area between the ROC curve and the Random Guessing Line
How close is my classifier to random guessing?
I need to quantify the inability of a binary classifier to obtain better results than random guessing in a single number evaluation metric.
The random ...
1
vote
0answers
60 views
Does a low Area Under Curve (AUC) for ROC imply worthless classifier? [closed]
I am trying to do a binary classification (0 and 1), and in this case, the end goal is to maximise the True Positives (i.e. maximum no. of 1s should be predicted correctly, even if it gives rise to ...
0
votes
1answer
37 views
Correct conditional expectation via logistic regression but terrible AUC
Suppose you have a binary random variable $Y$, and several other random variables $X_1,...,X_p$. Your goal is to "predict $Y$ using $X_1,...,X_p$." So, you go ahead and fit logistic regression, which ...
0
votes
1answer
46 views
AUC of single model vs AUC of separate models on same data
I have created two separate binary classifiers that predict the same kind of label using 2 separate datasets. The data is in the same format. They both have a AUC of 0.94 and 0.95
I have then created ...
-1
votes
0answers
14 views
need auc scoring with gridsearch in keras
I have unbalance dataset , i need to implement auc scoring in keras with gridsearch cv to find the best score. But in keras classifier auc and other metrics cannot be used directly , can anyone help ...
0
votes
0answers
37 views
Accuracy score or AUC extracted from Gradient Boosting Classifier of scikit-learn? [duplicate]
I'm working on developing a predictive model for a binary classification problem related to biomedical applications (need a really high and promising accuracy). I'm training on my training dataset and ...
0
votes
1answer
62 views
High AUC but low R squared in a random forest classifier
I have been looking for an answer on this website and on Google but I can't seem to find a clear explanation anywhere.
The problem is the following. I built a Random Forest model (using Python's ...
1
vote
0answers
22 views
AUC depends on random splitting
I have around 1500 data samples, 10-15 features and I am training Logistic Regression for binary classification. Data is not "very clean" and I makes some imputation.
I split data into training and ...
0
votes
1answer
34 views
Stacking AUC vs. average over folds
I have a two class prediction problem where in one class I have 70% of the samples and in the other class 30% of the samples, so class imbalance. I'm conducting 10-fold cross-validation. To calcualte ...
0
votes
0answers
8 views
Testing difference of cumulative/integrated quantities of time series?
I am new to time series data analysis. I wonder how would one test the statistical significance of the difference between integrated quantities (area under curve) from different time-series curves? (...
1
vote
0answers
27 views
Comparing AUC between two subsets of test-fold
I suspect that the out-of-sample AUC of my model depends on the number of events in the out-of-sample/testing fold (I am modelling a binary variable, 0/1).
In order to test this hypothesis, I want to ...
1
vote
0answers
35 views
Aggregating ROC AUC values of several Logistic Regression Models
I have a dataset that consists of six different segments. I have calculated a Logit Regression Model for each of those segments (binary response variable, 30.000 observations in total, 63 variables ...
2
votes
1answer
39 views
Area Under the Curve - Variable and Log Transformed Variable
I have a situation where I am fitting two simple logistic regression models - one model with the variable of interest included as the only predictor, and the other model with the log of the variable ...
0
votes
0answers
14 views
What is a reliable way to obtain an optimism-correct AUC with confidence limits?
I have seen that Frank Harrell's rms package does not offer a CI for Somers Dxy (and subsequently the c-statistic/AUROC).
I am trying to look at a method with good performance and a plan to ...
0
votes
1answer
35 views
Is it correctly understood that ROC/AUC cannot be calculated without flexible criterion value?
I have a proprietary predictor that simply gives me a binary output. Let's say that it is detecting faulty units. In a set where 27 units are faulty and 76 units are working the predictor correctly ...
0
votes
0answers
21 views
Evaluating concordance between multiple AUC statistics measured on the same task
I am evaluating a series of features within large genomic regions.
Each feature has $n$ AUC values associated with its capacity to discriminate control from the test samples: extreme values of AUC ...
0
votes
0answers
13 views
Why is the AUC when plotted against the wilcoxon rank sum p-value, result in a plot that is not linear?
I have a binary response variable with many continuous predictor variables (about 5000). I first computed the AUC for each predictor variable, then computed the p-value associated with the AUC using ...
0
votes
1answer
24 views
Minimising CV standard deviation to increase accuracy
Does minimising the standard deviation of CV folds have any correlation with model accuracy as a theme?
I've noticed that changing the order of rows in a training data can change the AUC for each CV ...
2
votes
2answers
50 views
Why does pROC roc work with non-probability predictions?
With the pROC package, I can do this:
true <- c(1, 1, 1, 0)
predicted <- c(0.5, 0.1, 0.6, 0.1)
roc(true, predicted)
which gives as expected:
...
0
votes
1answer
47 views
High AUC and Accuracy but weird output in confusion matrix
I am working on image classification problem to determine gender given a face. The dataset is located here gender face dataset on kaggle (link to my notebook). The class distribution is as follows.
<...
0
votes
1answer
37 views
Why there is decrease in AUC values with increase in number of iteration in Held out validation?
I have a dataset with 600 rows and 4000 columns for which I am trying to do held-out cross-validation with 10 and 100 iterations.
At first, the dataset is split into 80%:20% training and “held-out” ...
0
votes
0answers
48 views
Why is XGBoost prediction proba so concentrated within specific range? (unbalanced class)
I am pretty to new to Machine Learning. I am training on some past Kaggle competitions including the Santander Customer Satisfaction Challenge (https://www.kaggle.com/c/santander-customer-satisfaction)...
1
vote
1answer
117 views
Why is cross_val_score substantially lower than .score or roc_auc_score?
I have a trained model, a GradientBoostingClassifier. My dataset is 60 thousand something rows of data that I've split into 66/33 train/test sets. Scoring the model via the ...
1
vote
0answers
29 views
Sample size for AUC based on mean and SD of raw data
I'm trying to estimate the sample size required to achieve 80% power at a 5% significance level for a superiority study comparing AUC from bioavailability data.
My challenge is the only previous ...
0
votes
0answers
61 views
Computing Standard Error of AUC over different cv folds, and assessing statistical significance
I have a dataset in a healthcare setting for which the task is to predict a binary outcome, I have done this using the Support Vector Machine algorithm in a 5-fold cross validation setup. The ...
0
votes
0answers
10 views
Which performance metric to use for stratified data? [duplicate]
I'm trying to classify a data into 3 classes (supervised), one of which is heavily underrepresented in the data set. In order to combat this imbalance, I decided to stratify the data. Now I want to ...
0
votes
1answer
241 views
How to distinguish overfitting and underfitting from the ROC AUC curve?
For model selection, one of the metric is (AUC Area Under Curve) which tell us how the models are performing and based on AUC value we can choose the best model.
But how to distinguish whether a ...
0
votes
0answers
17 views
What is a Rank in AUC in a classification method?
I have read a post on AUC for which the link is given below
What does AUC stand for and what is it?
I am putting a new post as the above is rather old.
The person gave a few points related to Rank ...
0
votes
0answers
25 views
ada model- variables overall importance
I have the object ada from a model I didn't train to predict a binary result (I don't have the training set). Ada package was used. And the result are 200 binary trees.
I would like to have a ...
2
votes
1answer
46 views
K-NN optimal value of 'K'
How can I find the optimal value of 'K' in K-NN using the cross_val_score function, with scoring metric as auc_score?
Do I need ...
1
vote
1answer
38 views
AUC ROC when one class consists of smaller subclasses
This question is different from Binary classification when one class consists of multiple subclasses
I have two classes that I want to distinguish by a supervised learning classifier such as a random ...
2
votes
1answer
313 views
Does AUC/ROC curve return a p-value?
When reading this article, I noticed that the legend in Figure 3 gives a p-value for each AUC (Area Under the Curve) from the ROC (Receiver Operator Characteristic) curves. It says:
The area under ...
2
votes
2answers
54 views
Appropriate way to get Cross Validated AUC
I was thinking about cross-validation and how it is the most appropriate way to do it...
Let's take the case of binary logistic regression where the goal is to calculate the AUC.
Make the partition ...
0
votes
0answers
67 views
interprete and calculate Area Under the Curve of classifier in Matlab using perfcurve?
I hope that somebody could help me with the following question. I would like to calculate the Area Under the Curve (AUC) of a classifier (Linear Discrimiant Analyses). To do this, I am using the ...
1
vote
0answers
57 views
What is the integral of the False Positive Rate over the False Positive Rate, compared to the AUC?
In machine learning the Area Under the Receiver Operating Characteristic Curve ($AUC$) can be illustrated in a plot of the True Positive Rate ($TPR$) against the False Positive Rate ($FPR$). Formally, ...
2
votes
0answers
39 views
How to calculate the standard error of ratio of two AUC
Please can anyone help me with this!
I hope my explanation is going to be simple:
This is the data I have:
I only have access to the aggregated data as below. the first column is the time points a ...
1
vote
1answer
191 views
How do I calculate AUC with leave-one-out CV
In a binary response setting (data matrix D with N rows) I have performed LOOCV and obtained a final lambda*. The average CV error for this lambda* is also, as I understand it, an unbiased estimator ...
0
votes
0answers
42 views
High accuracy for single classes & low accuracy for multiclass classification
I was trying to do a multiclass-classification, where each sample belongs to one of the four classes. Now that I have a probability vector $(p_1^i,p_2^i,p_3^i,p_4^i)$ for each sample $i$ as my ...
2
votes
2answers
210 views
Can I trust my random forest model with low sample size and unequal class distribution?
I have a general question regarding model evaluation for random forest with low sample size and unequal class distribution.
I am doing some explorative modeling by using 400 features to classify ...
0
votes
0answers
16 views
AUC/R^2 score strictly lower on partitioned dataset than total dataset
Suppose I've fit a logistic regression to my data and I calculate the AUC. I then partition my dataset according to one of the explanatory variables (say those rows where the variable value is greater/...
0
votes
0answers
45 views
hierarchical bootstrapping and calculation of variance (in a Random-Effects ROC Analysis) in R
I would like to calculate the variance of the AUC of readers (for each reader and averaged results) giving a score(1-5) to ...
2
votes
1answer
240 views
Reason for higher AUC from a test set than a training set using a random forest
I made a 70:30 split of the data to build a random forest model for binary classification. Although the prevalence of $Y=1$ was about 25% in both training and test sets, the two sets became imbalanced ...
1
vote
1answer
33 views
Why are ROC curves and AUC values not always relevant?
So I have read in posts and in literature (Frank Harrel - Regression Modeling
Strategies fx) that depending on what you do, ROC curves and AUC values are not always relevant, but often written in ...
1
vote
1answer
35 views
ROC curve: models have different FPR ranges
I've created ROC curves by calculating the TPR and FPR at various thresholds. The FPR range differs between models, so I'm wondering if AUC is still a valid way to compare the curves. A curve will ...
1
vote
1answer
123 views
How do iso accuracy line related to ROC curve
I have read many articles about ROC curve. Some specified a method to calculate the accuracy of a classifier using iso accuracy lines in a convex hull.
Just like in this articles :
http://mlwiki.org/...
2
votes
2answers
237 views
Why AUC is not a good performance metric for a classification model?
After understanding the benefits of AUC
I was stumbled to know that in some scenarios it might not be a good performance metric for evaluating a classification model. The below are the 2 scenarios:
...
2
votes
1answer
49 views
how to interpret the behavior of a ROC curve very close to one
annex the ROC curve of a logit model, which has a very high AUC, how can I interpret the behavior of this curve, I doubt the logit model?
