Questions tagged [threshold]
Used (1) for discrete classification (if an instance's predicted probability exceeds a threshold, classify as TRUE, otherwise FALSE), or (2) for discretizing/binning continuous data. *If you are tempted to use this tag, PLEASE read the tag wiki!*
254
questions
0
votes
0
answers
15
views
What does it mean if optimal classification threshold found on ROC curve is really small?
I've trained a simple NN to perform binary classification with goal of maximizing area under ROC curve. Right now AUC is around 0.85. Out of curiosity, I checked which thresholds are best in terms of ...
- 181
0
votes
0
answers
14
views
Setting threshold with dynamic feedback
I have a dataset with 200,000 obs and the following variables:
score (continuous: 0-100), pred (binary: 0/1).
I want to create a binary variabel: pred2, that acts the following:
a) if score is high, ...
- 19
0
votes
0
answers
11
views
Regression discontinuity design when only one value is below or above threshold
I conduct a RDD with age as continuous running variable. My threshold is 18 (sharp). I observe people between 17 and 80. Thus I ...
- 284
0
votes
0
answers
14
views
Threshold selection for Peaks over threshold
I have a problem with selecting a threshold for a dataset I cannot show here.
In a MRL-plot, a suitable threshold should be linear with the gradient ξ/(1−ξ) and intercept σ_u/(1−ξ) (as of my ...
- 31
0
votes
0
answers
26
views
How does p-value affect FDR
Is it always true that a lower p-value threshold will lead to a lower FDR?
To answer this question I would say that it is not always true, for example in the case when the distribution is not normal ...
- 101
1
vote
0
answers
49
views
How do you do decision threshold tuning when doing k-fold cross validation?
I'm training a binary classifier for disease detection.
Because of my small amount of data (~1000 datapoints, 10% positive, 90% negative), I've realized that doing an 80-20 train-test split produces ...
- 11
2
votes
1
answer
80
views
Threshold Tuning before or after parameter tuning?
My goal is to increase the F1 score of Class 1 by 1-2%.
I achieved this by changing the threshold from 0.5 to X using the precision recall curve when the dataset is imbalanced. I did this after I have ...
0
votes
1
answer
33
views
How does one get from ROC curve to selecting the actual decision threshold of a classification model?
Edit to explain how this is different from the suggested duplicate: Reduce Classification Probability Threshold
My question relates to the same topic, but is thoroughly different, so I'm surprised ...
- 237
0
votes
0
answers
18
views
Is it possible to derive a threshold when using glmer? [duplicate]
Using the glmer function from the package lme4 I want to know whether and if so, how, it´s possible to obtain a specific threshold with the binomial model?
We fitted the model as follows with ...
- 1
3
votes
2
answers
84
views
When is an unbalanced dataset large enough for calculating a decision threshold?
I have a (large i.e. >1M rows) very unbalanced (1% event label, binary classification) dataset with data from various institutions. At the moment, I train an XGBoost model on this data and get good ...
- 31
0
votes
0
answers
16
views
Probability that a linear filtration of a stochastic process exceeds a value for a given number of consecutive samples
I am trying to work out how to calculate the probability that a weighted sum of IID random variables will remain above a given threshold for a particular amount of time ie;
Given a collection of IID ...
- 1
1
vote
0
answers
10
views
How to find thresholds/lower/upper limit for weather factors for species distribution modelling?
What's the best way to find an estimate of weather factors' thresholds/lower/upper limits for a response variable (in my case disease severity) in studies conducted under field conditions? I have ...
- 141
0
votes
0
answers
447
views
Plotting precision-recall curve using plot_precision_recall_curve and precision_recall_curve results in different plots
I am plotting the precision-recall curves for my models which I have built using an imbalanced dataset.
I initially plotted the precision-recall curve for my models using the ...
- 133
0
votes
0
answers
36
views
Threshold linear regression estimation
When it comes to threshold linear regression, in order to estimate it can we simply divide our dataset according to the threshold rule into 2 datasets and then simply estimate 2 equations with OLS? Or ...
- 165
1
vote
1
answer
349
views
Logistic regression - Does a decision threshold of 0.5 ever make sense?
Say I fit a logistic classifier on a supervised dataset with binary labels.
If I select a threshold of decision of 0.5, which assumption am I implicitly making?
Is there any situation where 0.5 makes ...
- 1,137
1
vote
0
answers
19
views
What is the optimal technique for determining statistical thresholds?
Relevant context: epidemiologists define an outbreak according to six defined stages (investigation, recognition, initiation, acceleration, deceleration, and preparation). From a local perspective, it ...
- 91
0
votes
1
answer
23
views
Better in AUC and AUC PR, but lower in the optimal threshold
Suppose we have two models; model A and model B.
Model A outperforms both AUC ROC and AUC PR to model B.
However, when we compare the two models with their optimal threshold values, model B ...
1
vote
1
answer
156
views
What happens if we change the threshold probability value for classifying into different class? [duplicate]
Suppose, I classify something as 1 when predicted probability of that event is greater than 0.5 (referred as threshold, henceforth) and 0 when predicted probability of that event is less than 0.5.
...
- 139
0
votes
0
answers
43
views
Tune threshold in hyperparameter tuning is giving worse results in MLR
First, I tried not tuning the hyperparameters without setting tune.threshold=TRUE.
...
- 101
1
vote
0
answers
29
views
Prediction of 'other' class
I'm training a MLP classifier with a softmax output that outputs 4 classes.
For my particular application I'd like the classifier to output a fifth 'other' class when the input don't belong to any of ...
- 4,319
0
votes
1
answer
48
views
What is the procedure to find the optimal decision threshold in an imbalanced classification problem to maximize F1 score?
What is the procedure to find the optimal decision threshold in an imbalanced classification problem to maximize the F1 score? I'm using an xgboost model.
Your help is highly appreciated.
- 213
0
votes
0
answers
27
views
does the ROC curve of a committee based predictor have any meaning?
would appreciate it if you'd take a moment to read the pipeline I've described below - it relates to how a learner that is based on a committee should be optimized w.r.t the threshold of the ROC curve....
- 115
1
vote
1
answer
36
views
Violation of IID in Peaks over Threshold
I'm using the peaks over threshold method to answer a researchquestion. I'm working with time-series data and the observations are not entirely independent. I know that there is some methods you could ...
- 31
0
votes
0
answers
34
views
Fixing threshold while improving classifier model?
I was playing with a situation where I am forced to fix the threshold of a classifier model. I was wondering if we can fix a cutoff for a mode but add data later on while maintaining the same cutoff? ...
0
votes
0
answers
13
views
Market cointegration
Does anyone here have a code or steps of how to conduct a Threshold vector error correction model? I've been reading a lot of literature but I've only understand the implications of the model. Most of ...
0
votes
0
answers
25
views
Choosing operating point for ~100% recall but max precision on unseen data
I will be training a binary classifier with the goal of getting 100% recall (or, 99.99% recall) when it's in production. The goal is to maximize the precision while keeping the probability of getting ...
- 101
0
votes
0
answers
43
views
How to select a threshold when retraining on the whole dataset?
I am training a Resnet model for a binary classification task, and since my dataset is quite small, I'd like to estimate my performance by doing a 10-fold cross validation, then retrain on the whole ...
1
vote
0
answers
43
views
How to find the optimal coefficients of the two predict_proba output matrices of two different classifiers using regression and maximizing accuracy? [closed]
I am performing classification, where there are six labels and two predict_proba (predicted probabilities) matrices as outputs. These two predict_proba matrices correspond to the outputs of two ...
- 11
1
vote
1
answer
519
views
Threshold optimization with cross validation
I have an imbalanced dataset; 95% negative class and 5% positive class. I split my data into train (80%) and test (20%) sets. I am using 5-fold cross-validation on the train set to determine the ...
- 21
2
votes
1
answer
78
views
Will threshold of logistic regression change accuracy? Any relationship with the incidence of disease? [duplicate]
I am using a logistic regression model to predict breast cancer. I trained and tested the model in a population with a pretty high incidence of breast cancer(since the individuals all went to the ...
- 511
0
votes
0
answers
21
views
Predicting behavioural changes based on absolute and change variables
I hope this is the adequate forum for my question.
My aim is to predict behavioural changes --which are simply measured by a dummy-- following from changes in some characteristics. To do this, I have ...
- 1
0
votes
0
answers
104
views
How should I select the optimal threshold using tcplot?
To find the optimal threshold for my gpd model, I made a threshold selection plot for the scale and the shape parameter. But I find it a bit vague how to select the optimal threshold graphically. Can ...
ameliedc
2
votes
0
answers
57
views
Does threshold on the model probability depend upon the spread in the dataset among positive and negative classes (binary classification)?
I think that the threshold on model probability through which one discern positive (y=0) and negative(y=1) class depends on the spread in the training dataset b/w y=0 and y=1.
This question came when ...
- 31
1
vote
0
answers
560
views
XGBoost Feature Importance Changes with Random Seed
Analysis Goal: Identify features that provide an accurate prediction of a binary outcome and also explain how the features are related to the output
Data: 72 features and 200 instances.
Process: ...
- 11
1
vote
0
answers
437
views
Thresholds and Cutoff Values Confusion
I am currently having confusion on a part in the paper:
Unal, Ilker. “Defining an Optimal Cut-Point Value in ROC Analysis: An Alternative Approach.” Computational and mathematical methods in medicine ...
0
votes
1
answer
45
views
How to determine a cutoff based on a dataframe with stats (TN TP FN FP MCC F1) on thresholds?
I have gotten a dataframe with corresponding stats (TN TP FN FP MCC F1) on different thresholds (~10,000 thresholds). I'm wondering if there is any statistical methods that help determine the best ...
- 103
0
votes
0
answers
68
views
Model ensembling when classifiers work with different classification thresholds
I have a 2-class classification problem at hand and trained three classifiers to tackle this task. In doing so, I determined for each classifier the optimal classification threshold. For example, ...
- 87
2
votes
1
answer
375
views
Normalize different binary prediction probability thresholds
I am trying to build an ensemble of three binary classifiers: A, B and C. Each one generates probabilities for the positive class. My goal is to generate a single probability for each case from the ...
- 1,870
2
votes
0
answers
257
views
OptimalCutPoints for GLM (Logistic Regression) in R | Find the Threshold
I want to find a threshold value so I can classify which observation is classified to be sucess or not. But I am a confused about how to use the package for logistic regression. For example my glm ...
0
votes
1
answer
93
views
Setting the observation likelihood threshold for outlier detection if you know know the percentage of outliers
Let's assume I have a sensor that gives me measurements $z$ and I know that $50\%$ of the measurements I read are outliers (more than 3 standard deviations away from the real measurement distribution)....
- 340
1
vote
1
answer
118
views
Relation between AUROC and threshold
As I understand, AUROC tells us the probability the model will score a randomly chosen positive class higher than a randomly chosen negative class. Meaning that, if AUROC = 0.7, than we expect that ...
- 521
1
vote
0
answers
35
views
When do Efficient Decision Algorithms for 1D Anomaly Detection exist (compared to threshold-tests)?
I'm tasked with investigating whether machine learning algorithms can be used to efficiently identify if a certain type of anomaly is present in the temporal spacing of incoming network packets. ...
- 11
0
votes
2
answers
2k
views
AUC with different threshold
I know AUC is supposed to be independent on the threshold, which means AUC does not change while the threshold changes. However, I'm getting different AUC values while changing the thresholds. I'm ...
- 511
1
vote
0
answers
25
views
Random censoring threshold
I am trying to estimate the following censored model: $y_{it}=\beta X_{it} + \epsilon_{it}$.
I only observe $y_{it}$ if $y_{it}\leq z_{it}$, otherwise $y_{it}=z_{it}$. The trick is that $z_{it}$ ...
- 397
2
votes
1
answer
607
views
Picking a model threshold based on Validation set or Test set
I have developed a machine learning model to predict a quantitative output for medical diagnosis (low bone density). I want to convert the model output to a binary outcome and compare it to the gold-...
- 35
2
votes
0
answers
36
views
Define outliers in correlation with right-skewed data (log-log plot)
I have a dataset of counts of occurrences of variables in different classes. For each class, I have an equivalent control created by shuffling the dataset.
For instance, this could be words from ...
- 85
0
votes
0
answers
312
views
Given a multiclass classifier, calculate one threshold per class to maximize recall under precision constraint
Given a classifier $f$, $N$ possible classes, and an input $x$, $f$ produces a class from $[1,..., N]$ and its matching confidence $[0,...,100]$.
Then I run $f$ on a large set of examples $X$, and I ...
- 53
1
vote
0
answers
627
views
Find the best threshold for logistic regression?
I am working on a customer purchase problem. I have 150 campaigns sent by email (or adds if you prefer), that I denote C0, C1 ......
- 21
0
votes
0
answers
16
views
Adjusting decision thresholds for dataset with seasonality component
I am working with data on an infection which exhibits seasonality over the year - the disease tends to be more prevalent during the rainy seasons compared with dry seasons.
I'm looking at a binary ...
- 13
1
vote
1
answer
243
views
Using statistics to determine a trigger/threshold value between continuous variables
I'm working with a few continuous variables, as below:
...
- 11