All Questions
Tagged with specificity or sensitivity-specificity
70 questions with no upvoted or accepted answers
13
votes
0
answers
259
views
Logistic regression for classification: are there any analytical solutions for the out-of-sample accuracy?
I run a binary logistic regression, with a binary dependent variable and a continuous independent one.
Now I want to evaluate the out-of-sample performance of the classification algorithm so obtained. ...
- 2,433
4
votes
1
answer
2k
views
ROC Curve AUC for Hypothesis Testing Sensitivity (Power) vs Specificity ($1-\alpha$)
(I called this a ROC curve when I posted two years ago, but it's really the empirical CDF of the p-values.)
Let's set up a hypothesis test of $H_0: \theta=\theta_0$ versus $H_1: \theta\ne\theta_0$, ...
- 67.1k
4
votes
0
answers
145
views
What is the correct definition of Selectivity : TPR or TNR? Since contradictory literature exists
The reason of my question is because contradictory articles exist. Talking about the confusion matrix's performance metrics, we can find:
English wikipedia pages mention that ...
- 661
3
votes
0
answers
42
views
How to do a sensitivity analysis for a Bayesian Model? Prior or Posterior?
I'm developing a mathematical (SIR-style) model for cholera transmission in the Bengal Delta which is fit to data using MCMC.
The purpose of the model is more explanatory rather than predictive, and I ...
3
votes
0
answers
176
views
Assessing correlated predictions
Let's assume we have a prediction algorithm (if it helps, imagine it's using some boosted tree method) that does daily predictions for whether some event will happen to a unit (e.g. a machine that ...
- 35.2k
3
votes
0
answers
91
views
Diagnostic device hypothesis testing for superiority to a fixed value
I am designing a study to demonstrate that a diagnostic medical device is more sensitive than a fixed performance goal of 70%. Currently I am attempting to determine the appropriate sample size. My ...
- 569
3
votes
1
answer
388
views
The effect of oversampling on the positive predictive value
I need to calculate the positive predictive value for a validation set for a rare event. The problem is that the validation set was oversampled for the rare event. The event occurs in 5 percent of the ...
- 31
2
votes
1
answer
118
views
Binary classification metrics - Combining sensitivity and specificity?
The harmonic mean between precision and recall (F1 score) is a common metric to evaluate binary classification. It is useful because it strikes a balance between precision (FP) and recall (FN).
For ...
- 1,257
2
votes
0
answers
487
views
When should one look at sensitivity vs. specificity instead of precision vs. recall?
The precision vs. recall tradeoff is the most common tradeoff evaluated while developing models, but sensitivity vs. specificity addresses a similar issue. When should one of these pairs of metrics be ...
- 145
2
votes
0
answers
276
views
Prevalence threshold calculations error
I see on this page the formula for calculation of prevalence threshold:
Given also are following values on that page:
...
- 10.2k
2
votes
0
answers
2k
views
what to do when true positive rate is undefined?
I have a binary classification set with about 900 samples. I first use kfold cross validation and AUC of the ROC curve to determine which classifiers perform best. Once I've done that, I use a leave-...
- 133
2
votes
0
answers
273
views
Calculating and reporting accuracy, sensitivity and specificity with multiple raters
I'm conducting a study in which I'm comparing 15 radiologists evaluations on a dichotomous variable to a gold standard, in a sample with 58 patients.
Obviously I've calculated overall accuracy, ...
- 81
2
votes
1
answer
41
views
Is this a sensitivity?
There are test A and test B.
I want to show a ratio of test A positive among all test positive(test A positive or test B positive).
Can I call the ratio as sensitivity?
Or is it safe to call as ratio?...
2
votes
0
answers
2k
views
How to improve the sensitivity of minority class on imbalanced datasets
I am working on a classifier which stratifies a population of samples into different classes.
The class distribution (ground truth) is imbalanced, and the prevalence of each class is:
$$\begin{...
- 1,257
1
vote
0
answers
22
views
Hypergeometric approximation of herd sensitivity and its non-linear nature
I have recently been exploring the use of the equation described in Cannon 2001 paper "Sense and Sensitivity" on herd sensitivity (love the name).
I have been looking to test a hypothetical ...
- 63
1
vote
0
answers
17
views
Interpreting sobol indices
I am trying to test the sensitivity of my model to a change in the parameters std and translation_factor. I get the following results :
Sobol First Order Indices:
translation_factor: -1....
- 11
1
vote
1
answer
23
views
Is there an equivalent for Yates' correction for a confusion matrix-derived metrics?
Given the following table of predictions vs. actual states:
...
- 1,291
1
vote
0
answers
15
views
Best way to sum up and visualize the relation between results of 2 tests measuring the same outcome with 3 results
I have a dataset of different individuals who underwent 2 test that measure the same outcome with 3 possible results
not reduced
mildly reduced
highly reduced
I know that the distribution of the ...
- 11
1
vote
0
answers
48
views
Sensitivity and specificity for multiple raters
I am conducting a study where 3 raters assessed performance of two similar test in regards to classify a lesion as malignant vs. benign. I wish to calculate specificity and sensitivity of each test ...
- 11
1
vote
2
answers
89
views
Individual Measurement in Overlapping Distributions
I have a situation where I have measurements taken from samples of healthy and pathological populations. The pathological populations tend to have much higher measurements, but the distributions are ...
- 1,486
1
vote
1
answer
34
views
How to show that simple random sample sensitivity is unbiased for population sensitivity
In diagnostic testing, sensitivity $S$ is the probability that the test gives a positive result given that you have the condition being tested. From a simple random sample of people who take the test, ...
1
vote
0
answers
47
views
Multiple observations AUC, Sensitivity and Specificity
I am trying to find a CIs for the AUC, sensitivity and specificity. In my scenario, each observation consistent of a subject and an area examined, that is it is obvious that areas examined for the ...
- 1,455
1
vote
0
answers
36
views
Estimating the mean and variance for a set of probabilities (bounded by 0 and 1) based on Image Segmentation results
Data
My data is from a set of images wherein I am computing the Sensitivity of an image segmentation algorithm. Sensitivity is computed as:
$$Sensitivity=\frac{TP_{pixels}}{TP_{pixels}+FN_{pixels}}$$
...
- 569
1
vote
0
answers
367
views
Why is the optimal cutoff for AUC different from that of specificity in this simulation?
I am working on a binary classification problem on an imbalanced data where the majority class is about 90% 'no' and the minority class is about 10% 'yes' of the total data.
Iteration 1: I randomly ...
- 548
1
vote
0
answers
92
views
Sensitivity, Accuracy, AUROC, Gini
I got following chart:
The algorithms have been applied to a dataset where an outcome is pretty rare, it happens 10% of the times (binary, 0- 90%, 1-10%). It is the response whether a client is going ...
- 111
1
vote
0
answers
16
views
Tuning the Logit model based on sensitivity
I am working on a logistic model to predict whether or not an employee is likely to take a leave day during a given three month period. The model only correctly classified the “non-event"
...
- 11
1
vote
0
answers
80
views
Take into account Sensitivity and Specificity when calculating Estimates/CI's
I'm trying to thing about how to calculate confidence intervals in the presence of known measurement error for some test protocol.
Let's assume we're doing a sample survey in which we're performing ...
- 41
1
vote
0
answers
31
views
Population selection for sensitivity and specificity
For medical physical assessment sensitivity and specificity is commonly used to assess the clinical relevance of the test. I'm specifically referring to manual medicine here, with orthopedic tests of ...
- 123
1
vote
0
answers
21
views
OneR and area under the ROC curve
In the book Data Mining, author Ian Witten says that it does not make sense to calculate the area under the ROC curve when applying OneR because it finds only one point (sensitivity, specificity) ...
- 79
1
vote
0
answers
19
views
Sensitivity of a diagnostic test
Diagnostic tests A1 and A2 are used to detect the presence or absence of a disease. The results of A1 and A2 are a priori independent of the presence or absence of the disease.
A1 and A2 have ...
- 11
1
vote
0
answers
134
views
Deriving a loss function properly accounting for different error type costs
Consider a classification setting like a medical test: Not finding an existing health issue might be much worse (by a factor of 50) than assuming an issue when there is none.
I.e. a setting in which ...
- 161
1
vote
0
answers
23
views
Sensitivity and specificity on cases rather than individual days
I have data collected from sensors to detect disease in several individuals. Each individual has a known disease status (labelled Actual in the table below) for ...
- 11
1
vote
0
answers
584
views
Why are specificity and sensitivity (recall) inversely related?
I remember learning this about medical tests when I studied biochemistry and now I met with this rule again by learning about binary classification and PR-curves. If we check the equations for both ...
- 161
1
vote
0
answers
29
views
Compare cut-offs with respect to sensitivity etc
I have a score and a dichotomous criterion for n observations. Using a ROC-Analysis, I found some optimal cut-off, say c. Now, literature (e.g., author of scale with which the score was determined) ...
1
vote
0
answers
1k
views
Calculating sensitivity and specificity from survival data
I have a diagnostic test performed on 100 participants at baseline. I then follow up these participants for variable periods of time and have data regarding survival.
I have used a Cox regression ...
- 253
1
vote
0
answers
88
views
How to make specificity a stable evaluation metric?
ROC (including AUC) metrics are widely used for binary classification problems. AUC is usually selected to evaluate the model.
However, some tasks may require high specificity with fixed sensitivity. ...
- 439
1
vote
0
answers
622
views
Calculating a pooled estimate
I am writing a systematic review on certain imaging studies and I was hoping to add a meta-analysis of the sensitivities of the various modalities reported on. My goal was to put together a forest ...
- 11
1
vote
0
answers
77
views
Confidence interpretation from false negative rate
For my research I'm using a tool called seeSUMO, which predicts sumoylation sites in a protein based on sequence features. When it reports your results, it gives you a level of confidence for each ...
- 11
0
votes
2
answers
51
views
Is it possible that false-positive rate decreases with increasing prevalence?
I am interested in the effect of prevalence on prediction performance. Chouldechova (2016) states that:
[w]hen using a test-fair [recidivism prediction instrument] in
populations where recidivism ...
- 113
0
votes
0
answers
22
views
Se, Sp, NPV and PPV questions for repeated measures data
I have a dataset that contains multiple test results (expressed as %) per participant, at various time points post kidney transplant. The dataset also contains the rejection group the participant ...
- 173
0
votes
0
answers
27
views
Meta-analysis of ROC areas with missing confidence interval
I am performing a meta-analysis of a diagnostic score going from 0 to 10 based on 30 reports. For each report I have data on number of false/true positive/negative patients for one and in some cases ...
- 301
0
votes
0
answers
43
views
How to measure Statistical Significance for calculated Sensitivity, Specificity, Precision, Accuracy and f1 values?
I am trying to understand how to calculate one or more measures of statistical significance to display alongside metrics I've calculated from my data.
Abbreviations I am using in the rest of this post:...
- 1
0
votes
0
answers
42
views
How to properly compare a single diagnostic test in two subpopulations
Suppose I have a diagnostic test with a binary response. I have performed this test on some population.
Now, I split this population in two subgroups (e.g. male/female), and would like to check if ...
- 115
0
votes
0
answers
21
views
How can we apply sensitivity analysis in comparing two ARIMA models, both measuring intervention effect?
I have a dataset of reporting rates collected over an 8-month period obtained from the baseline (4 months) and during intervention (4 months). The goal is to determine if the intervention has a ...
0
votes
0
answers
7
views
How to compute the minimum sensitivity and specificity a classifier needs to estimate a proportion to within some percent error?
I am building a binary classifier. Once classified, the proportion of positive events will be calculated. The predicted proportion is required to be within 10% of the true proportion. In practice, ...
- 1
0
votes
0
answers
63
views
What is the relationship between bias-variance and sensitivity-specificity for novelty detection?
An over or under-parameterized binary classification model (- vs +) tends to over or under-fit (bias-variance tradeoff). This leads to errors during prediction on unseen data. Depending on if ...
- 53
0
votes
0
answers
225
views
Comparing Sensitivity, Specificity, PPV and NPV of one test across two subgroups
I am seeking to compare the diagnostic performance of a test within a population with two defined subgroups.
I have the raw data & computed data, but am unsure what the best test is to compare the ...
- 1
0
votes
0
answers
72
views
How to apply Bayes' theorem on a contingency in which the sampled distribution differs drastically from the entire population?
Summary: How can I calculate my probability of being truly positive based on a contingency table that has a completely different distribution than the total population?
I was recently confronted with ...
- 1
0
votes
0
answers
95
views
Automatically obtain optimum cutoff point for accuracy, sensitivity and specificity
Below is a code snippet to plot the accuracy, sensitivity and specificity. I can manually eyeball and see the cutoff point at approximately around 0.3. I was wondering if there anyway to print out ...
0
votes
0
answers
24
views
Sensitivity to settings vs statistical sensitivity
I am confused between two different properties of a model. Both are called sensitivity. First, sensitivity is another word for true positive rate. Second, the output of a model changes in response to ...
- 97