# Is sensitivity or specificity a function of prevalence?

Standard teaching says that sensitivity and specificity are properties of the test and are independent of prevalence. But isn't this just an assumption?

Harrison's principles of internal medicine 19th ed says

It has long been asserted that sensitivity and specificity are prevalence-independent parameters of test accuracy, and many texts still make this statement. This statistically useful assumption, however, is clinically simplistic. ...test sensitivity will likely be higher in hospitalised patients, and test specificity higher in outpatients.

(Prevalence is typically higher in inpatients than in outpatients)

Is there a mathematical or an approximate graphical relation between these parameters?

Even this link calls it an 'simplification'. Why?

Edit: I know how sensitivity is defined. There is no term of prevalence involved, as mentioned in the answers. I myself have maintained that these are properties of the test unaffected by the population used, in until I came across this statement, hence the question. But I assume, this confusion is arising not due to definition but practical calculation of these values. Specificity and sensitivity are calculated using 2x2 tables, does the prevalence of the reference population here matter? Is that what they are referring to? If it does, what's the function?

Although @Tim♦'s and @gung♦'s answers pretty much cover everything, I'll try to synthesize them both into a single one and provide further clarifications.

The context of the quoted lines might mostly refer to clinical tests in form of a certain Threshold, as is most common. Imagine a disease $D$, and everything apart from $D$ including the healthy state referred to as $D^c$. We, for our test, would want to find some proxy measurement which allows us to get a good prediction for $D$.(1) The reason we do not get absolute specificity/sensitivity is that the values of our proxy quantity do not perfectly correlate with the disease state but only generally associate with it, and hence, in individual measurements, we might have a chance of that quantity crossing our threshold for $D^c$ individuals and vice versa. For the sake of clarity, let's assume a Gaussian Model for variability.

Let us say we are using $x$ as the proxy quantity. If $x$ has been chosen nicely, then $E[x_D]$ must be higher than $E[x_{Dc}]$ ($E$ is the expected value operator). Now the problem arises when we realize that $D$ is a composite situation (so is $D^c$), actually made of 3 grades of severity $D_1$, $D_2$, $D_3$, each with a progressively increasing expected value for $x$. For a single individual, selected either from $D$ category or from the $D^c$ category, the probabilities of the 'test' coming positive or not will depend on the threshold value we choose. Let us say we choose $x_T$ based on studying a truly random sample having both $D$ and $D^c$ individuals. Our $x_T$ will cause some false positives and negatives. If we select a $D$ person randomly, the probability governing his/her $x$ value if given by the green graph, and that of a randomly chosen $D_c$ person by the red graph.

The actual numbers obtained will depend on the actual numbers of $D$ and $D^c$ individuals but the resultant specificity and sensitivity will not. Let $F()$ be a cumulative probability function. Then, for the prevalence of $p$ of the disease $D$, here's a 2x2 table as would be expected of the general case, when we try to actually see how our test performs in the combined population.

$$(D,+) = p(1-F_D(x_T))$$ $$(Dc,-) = (1-p)(1-F_{Dc}(x_T))$$ $$(D,-) = p(F_D(x_T))$$ $$(Dc,+) = (1-p)*F_{Dc}(x_T)$$

The actual numbers are $p$ dependent, but sensitivity and specificity are $p$ independent. But, both of these are dependent on $F_D$ and $F_{Dc}$. Hence, all the factors which affect these, will definitely change these metrics. If we were for example, working in the ICU, our $F_D$ would be instead be replaced by $F_{D3}$, and if we were talking about outpatients, replaced by $F_{D1}$. It is a separate matter that in the hospital, the prevalence is also different, but it is not the different prevalence which is causing the sensitivities and specifities to differ, but the different distribution, since the model on which the threshold was defined was not applicable to the population appearing as outpatients, or inpatients. You can go ahead and break down $D^c$ in multiple subpopulations, becasue the inpatient subpart of $D^c$ shall also have a raised $x$ due to other reasons (since most proxies are also 'elevated' in other serious conditions). Breaking of the $D$ population into subpopulation explains the change in sensitivity, while that of the $D^c$ population explains the change in specificity (by corresponding changes in $F_D$ and $F_{Dc}$).This is what the composite $D$ graph actually comprises of. Each of the colors will actually have their own $F$, and hence, as long as this differes from the $F$ on which the original sensitivity and specificity were calculated, these metrics will change.

# Example

Assume a population of 11550 with 10000 Dc, 500,750,300 D1,D2,D3 respectively. The commented out portion is the code used for the above graphs.

set.seed(12345)
dc<-rnorm(10000,mean = 9, sd = 3)
d1<-rnorm(500,mean = 15,sd=2)
d2<-rnorm(750,mean=17,sd=2)
d3<-rnorm(300,mean=20,sd=2)
d<-cbind(c(d1,d2,d3),c(rep('1',500),rep('2',750),rep('3',300)))
library(ggplot2)
#ggplot(data.frame(dc))+geom_density(aes(x=dc),alpha=0.5,fill='green')+geom_density(data=data.frame(c(d1,d2,d3)),aes(x=c(d1,d2,d3)),alpha=0.5, fill='red')+geom_vline(xintercept = 13.5,color='black',size=2)+scale_x_continuous(name='Values for x',breaks=c(mean(dc),mean(as.numeric(d[,1])),13.5),labels=c('x_dc','x_d','x_T'))

#ggplot(data.frame(d))+geom_density(aes(x=as.numeric(d[,1]),..count..,fill=d[,2]),position='stack',alpha=0.5)+xlab('x-values')


We can easily compute the x-means for the various populations, including Dc, D1, D2, D3 and the composite D.

mean(dc)
mean(d1)
mean(d2)
mean(d3)
mean(as.numeric(d[,1]))

> mean(dc) [1] 8.997931
> mean(d1) [1] 14.95559
> mean(d2) [1] 17.01523
> mean(d3) [1] 19.76903
> mean(as.numeric(d[,1])) [1] 16.88382


To get a 2x2 table for our original Test case, we first set a threshold, based on the data (which in a real case would be set after running the test as @gung shows). Anyway, assuming a threshold of 13.5, we get the following sensitivity and specificity when computed on the entire population.

sdc<-sample(dc,0.1*length(dc))
sdcomposite<-sample(c(d1,d2,d3),0.1*length(c(d1,d2,d3)))
threshold<-13.5
truepositive<-sum(sdcomposite>13.5)
truenegative<-sum(sdc<=13.5)
falsepositive<-sum(sdc>13.5)
falsenegative<-sum(sdcomposite<=13.5)
print(c(truepositive,truenegative,falsepositive,falsenegative))
sensitivity<-truepositive/length(sdcomposite)
specificity<-truenegative/length(sdc)
print(c(sensitivity,specificity))

> print(c(truepositive,truenegative,falsepositive,falsenegative)) [1]139 928  72  16
> print(c(sensitivity,specificity)) [1] 0.8967742 0.9280000


Let us assume we are working with the outpatients, and we get diseased patients only from the D1 proportion, or we are working in the ICU where we only get D3. (for a more general case, we need to split the Dc component too) How do our sensitivity and specificity change? By changing the prevalence (i.e. by changing the relative proportion of patients belonging to either case, we do not change the specificity and sensitivity at all. It just so happens that this prevalence also changes with changing distribution)

sdc<-sample(dc,0.1*length(dc))
sd1<-sample(d1,0.1*length(d1))
truepositive<-sum(sd1>13.5)
truenegative<-sum(sdc<=13.5)
falsepositive<-sum(sdc>13.5)
falsenegative<-sum(sd1<=13.5)
print(c(truepositive,truenegative,falsepositive,falsenegative))
sensitivity1<-truepositive/length(sd1)
specificity1<-truenegative/length(sdc)
print(c(sensitivity1,specificity1))
sdc<-sample(dc,0.1*length(dc))
sd3<-sample(d3,0.1*length(d3))
truepositive<-sum(sd3>13.5)
truenegative<-sum(sdc<=13.5)
falsepositive<-sum(sdc>13.5)
falsenegative<-sum(sd3<=13.5)
print(c(truepositive,truenegative,falsepositive,falsenegative))
sensitivity3<-truepositive/length(sd3)
specificity3<-truenegative/length(sdc)
print(c(sensitivity3,specificity3))

> print(c(truepositive,truenegative,falsepositive,falsenegative)) [1]  38 931  69  12
> print(c(sensitivity1,specificity1)) [1] 0.760 0.931
> print(c(truepositive,truenegative,falsepositive,falsenegative)) [1]  30 944  56   0
> print(c(sensitivity3,specificity3)) [1] 1.000 0.944


To summarise, a plot to show the change of sensitivity (specificity would follow a similar trend had we also composed the Dc population from subpopulations) with varying mean x for the population, here’s a graph

df<-data.frame(V1=c(sensitivity,sensitivity1,sensitivity3),V2=c(mean(c(d1,d2,d3)),mean(d1),mean(d3)))
ggplot(df)+geom_point(aes(x=V2,y=V1),size=2)+geom_line(aes(x=V2,y=V1))


1. If it is not proxy, then we would technically have a 100% specificity and sensitivity. Say for example we define $D$ as having a particular objectively defined pathological picture on say Liver Biopsy, then the Liver Biopsy test will become the gold standard and our sensitivity would be measured against itself and hence yield a 100%

First, it's worth recognizing that you cannot typically change sensitivity independently of specificity, and vice versa. This is the point of a ROC curve. Given the nature of the data generating process, and your specific data and model, you will always be stuck with some tradeoff between sensitivity and specificity. You would of course prefer to have 100% sensitivity and 100% specificity at the same time, but typically you can't. You can get better sensitivity, but at the expense of worse specificity, or better specificity, but at the expense of worst sensitivity. The ROC curve shows you the set of tradeoffs you are forced to choose between. (A couple of notes: 1. you can sometimes appear to gain on one dimension without loosing anything on the other because there is a gap in your dataset, but this is mostly illusory; 2. technically a ROC curve is sensitivity as a function of 1-specificity, plotting sensitivity vs specificity itself would be a reflected ROC curve.)

At any rate, how could the apparent sensitivity and specificity change with prevalence? This is an issue where it helps to simulate and play with some data to see how this can work out in practice. Let's imagine that a model is fit to a fairly large dataset that has a particular prevalence, and a threshold is set on the x-axis1. Later, the performance of this test is computed with samples that have substantially different prevalences (and thus different x-values). The result is that the same model, using the same threshold will perform differently when applied to datasets with differing prevalences.

library(caret)  # we'll use these packages
library(binom)
# we'll use this function to convert log odds to probabilities
lo2p = function(lo){ exp(lo)/(1+exp(lo)) }

##### training dataset for original model
set.seed(734)                     # these make the examples exactly reproducible
Nt = 1000
xt = rnorm(Nt, mean=5, sd=1)      # this is the distribution of X
lo = -1.386 + .308*xt             # this is the data generating process
pt = lo2p(lo)
yt = rbinom(Nt, size=1, prob=pt)
mt = glm(yt~xt, family=binomial)
summary(mt)
# ...
# Coefficients:
#             Estimate Std. Error z value Pr(>|z|)
# (Intercept) -1.16736    0.32794  -3.560 0.000371 ***
# xt           0.24980    0.06429   3.886 0.000102 ***
# ...
#     Null deviance: 1384.5  on 999  degrees of freedom
# Residual deviance: 1369.1  on 998  degrees of freedom
# AIC: 1373.1

## determine threshold
# prob(Y) = 50%, where log odds = 0, so:
-coef(mt)[1]/coef(mt)[2]  # 4.673159
threshold = 4.7  # a simple round number
classt    = ifelse(xt>threshold, 1, 0)
tabt      = table(classt, yt)[2:1,2:1]

confusionMatrix(tabt)
#       yt
# classt   1   0
#      1 346 279
#      0 175 200
#
#                Accuracy : 0.546
#                     ...
#             Sensitivity : 0.6641
#             Specificity : 0.4175
#          Pos Pred Value : 0.5536
#          Neg Pred Value : 0.5333
#              Prevalence : 0.5210

##### high prevalence dataset from hospital
set.seed(4528)
Nh = 500
xh = rnorm(Nh, mean=6, sd=1)  # a different distribution of X
lo = -1.386 + .308*xh         # but the same data generating process
ph = lo2p(lo)
yh = rbinom(Nh, size=1, prob=ph)
classh = ifelse(xh>threshold, 1, 0)  # the same threshold is used
tabh   = table(classh, yh)[2:1,2:1]

confusionMatrix(tabh)
#       yh
# classh   1   0
#      1 284 163
#      0  20  33
#
#                Accuracy : 0.634
#                     ...
#             Sensitivity : 0.9342
#             Specificity : 0.1684
#          Pos Pred Value : 0.6353
#          Neg Pred Value : 0.6226
#              Prevalence : 0.6080

##### low prevalence dataset from outpatients
set.seed(1027)
Nl = 500
xl = rnorm(Nl, mean=3, sd=1)
lo = -1.386 + .308*xl
pl = lo2p(lo)
yl = rbinom(Nl, size=1, prob=pl)
classl = ifelse(xl>threshold, 1, 0)
tabl   = table(classl, yl)[2:1,2:1]

confusionMatrix(tabl)
#       yl
# classl   1   0
#      1   9  14
#      0 190 287
#
#                Accuracy : 0.592
#                     ...
#             Sensitivity : 0.04523
#             Specificity : 0.95349
#          Pos Pred Value : 0.39130
#          Neg Pred Value : 0.60168
#              Prevalence : 0.39800

##### sensitivities
binom.confint(346, 521, method="e")
#   method   x   n      mean     lower    upper
# 1  exact 346 521 0.6641075 0.6217484 0.704592
binom.confint(284, 304, method="e")
#   method   x   n      mean   lower     upper
# 1  exact 284 304 0.9342105 0.90022 0.9593543
binom.confint(  9, 199, method="e")
#   method x   n       mean      lower      upper
# 1  exact 9 199 0.04522613 0.02088589 0.08411464

##### specificities
binom.confint(200, 479, method="e")
#   method   x   n      mean     lower     upper
# 1  exact 200 479 0.4175365 0.3729575 0.4631398
binom.confint( 33, 196, method="e")
#   method  x   n      mean     lower     upper
# 1  exact 33 196 0.1683673 0.1188206 0.2282441
binom.confint(287, 301, method="e")
#   method   x   n      mean     lower     upper
# 1  exact 287 301 0.9534884 0.9231921 0.9743417


Here are the sensitivities and specificities as a function of prevalences, with exact 95% confidence intervals:

So what's going on here? Consider that a prototypical logistic regression might look something like the figure below. Note that all of the 'action' is taking place in the interval [4, 6] on the x-axis. Data below that will have very low prevalence, and the model will show poor discrimination and sensitivity. Data above that interval will have very high prevalence, but the model will again not discriminate well and will have poor specificity.

To help understand how this could happen, consider the testing of Alanine transaminase to determine if the patient's liver is failing2. The idea is that the liver normally uses ALT, but that if the liver has stopped functioning ALT will be dumped into the blood stream. So if the level of ALT in a patient's bloodstream is above some threshold, that implies the liver is failing. If you draw a sample with a high prevalence of liver failure, you will be drawing a sample with high levels of ALT in the blood. Thus, you will have more patients above the threshold. Not everyone with high blood levels of ALT will have liver failure—for some patients there will be some other cause. But those with liver failure should be being caught. This leads to higher sensitivity. Likewise, not all patients with normal levels of ALT have healthy livers, but a sample with low prevalence will have lower levels of ALT, and more patients will pass the test. Those whose livers aren't failing, but who have normal levels of ALT will be missed. This leads to lower sensitivity, but higher specificity.

More generally, the whole idea of a medical test is that something or other is a correlate of a disease state that you might like to have direct measures of, but can't. Getting a measure of the correlate gives you insight into the disease state. A (potential) test where this isn't true would be of no value and wouldn't be used. Thus in practice, higher prevalence samples should have a distribution of the correlate with more abnormal values leading to higher sensitivity, and vice versa. (Note that the correlate does not have to be a cause of the disease; in the ALT example, it is an effect, in other examples, both the disease and the correlate could be effects of a common cause, etc.)

1. This is actually quite common in medicine. Consider that cholesterol should be <200, systolic blood pressure should be <140, etc. Those aren't really 'tests' per se, but there are lots of tests that work just like that. For some (perhaps distantly) related discussions on thresholds, it may help to read my answers to Are 0-1 thresholds always equivalent to x-axis thresholds?, and Why are the number of false positives independent of sample size, if we use p-values to compare two independent datasets?
2. Please be aware that I am not a physician, and this example may well be badly botched. Ask an actual physician if you want accurate information about liver function, tests thereof, and related matters.

• Thanks! For showing that it actually does change. But how does it considering @Tim 's answer? Isnt it contradicting? – Polisetty Feb 25 '18 at 8:48
• @Polisetty, Tim states that, "inpatients and outpatients may differ in many aspects, not only in prevalence alone, so some other factors may influence the sensitivity". If the test is a function of some property of patients (say, cholesterol), & the disease is strongly correlated w/ that property as well (which is generally the whole point), then the "other factors" must move in conjunction w/ the prevalence. Thus, when prevalence changes, the other correlates change, & the test has more or less sensitivity w/i that specific group. – gung Feb 25 '18 at 14:36

As already said by others, sensitivity and specificity don't depend on prevalence. Sensitivity is the proportion of true positives among all positives and specificity is proportion of true negatives among all negatives. So if sensitivity is 90%, then the test will be correct for 90% of the cases that are positive. Obviously 90% of something smaller and 90% of something larger is still 90%...

So given the tabular data that you mention,

$$\begin{array}{cc} & \substack{\text{positive} \\ \text{condition}} & \substack{\text{negative} \\ \text{condition}}\\ \substack{\text{positive} \\ \text{test}} & a & c \\ \substack{\text{negative} \\ \text{test}} & b & d \\ \end{array}$$

sensitivity is $\tfrac{a}{a+b+c+d} \,/\, \tfrac{a+b}{a+b+c+d} = \tfrac{a}{a+b}$ (from the definition of conditional probability $p(Y \mid X) = \tfrac{p(Y \cap X)}{p(X)}$) and specifity is $\tfrac{d}{a+b+c+d} \,/\, \tfrac{c+d}{a+b+c+d} = \tfrac{d}{c+d}$. For each of the metrics, you look only at one of the columns at the time, so the prevalence (relative sizes of the columns) does not matter for those metrics. The prevalence does not come into the equations. Also it would be rather strange if the "practical" sensitivity was defined differently then theoretically and led to different conclusions.

But the quote seems also to be saying something else

test sensitivity will likely be higher in hospitalized patients, and test specificity higher in outpatients

so authors say that sensitivity differs in different groups. I guess that inpatients and outpatients may differ in many aspects, not only in prevalence alone, so some other factors may influence the sensitivity. So I agree that they may change between different datasets, that differ in prevalence, but the change will not be a function of the prevalence itself (as shown by @gung in his answer).

On another hand, if I had to guess, maybe the authors are confusing sensitivity with posterior probability. Sensitivity is $p(\text{positive test}\mid\text{condition})$, while the posterior probability is

$$p(\text{condition}\mid\text{positive test}) \propto p(\text{positive test}\mid\text{condition})\times p(\text{condition})$$

and in many cases this is the probability the people are interested in ("how likely is a patient with a positive test result to actually have the disease?") and it depends on the prevalence. Notice that also your link discusses the impact of prevalence on Positive Predictive Value, i.e. the posterior probability, not on sensitivity.

• As I mentioned in one of the previous answers, I'm pretty sure that the authours did not confuse it with the posterior probability, since they explicitly mention that "many texts still make this statement". And I also quote another source, althoguh not as reliable as the Harrison's, that says that it is a safe 'assumption'. All I want to ask is , what is the 'assumption'? – Polisetty Feb 23 '18 at 21:34
• @Polisetty I can't say for the authors, but from the quote, they seem to call the independence on prevalence the "assumption", but this is rather a mathematical fact then assumption. If it didn't hold, it would mean that the probability theory is broken and it isn't. – Tim Feb 23 '18 at 21:44
• Sensitivity and specificity can be considered fixed properties of a diagnostic test. [This is a slight simplification, but it's good enough for our purposes]. - thats what it says – Polisetty Feb 23 '18 at 21:59

See my answer here on true/false positive/negative rates.

Sensitivity is just another name for the true positive rate, and specificity is the same as the true negative rate. Both sensitivity and specificity are conditional probabilities; they condition on the disease status of the patient. Thus the prevalence of the disease (i.e. the a priori probability that a patient has the disease) is irrelevant, since you are assuming a particular disease state.

I can't comment on why the textbook author claims that sensitivity and specificity depend on the clinical context. Are these empirical observations?

• Exactly. Hence the question. The sensitivity of a test depends on the population where used. The assumption that's it is independent is not always true. I'm asking how and why. The book later quotes values too – Polisetty Feb 14 '18 at 20:49
• There could be population-specific factors that affect sensitivity and specificity. But it follows from the mathematical definitions of sensitivity and specificity that prevalence cannot be one of these factors, at least not directly. (By the way, feel free to accept my answer if you're satisfied with my explanation of the mathematical definitions.) – tddevlin Feb 14 '18 at 20:58
• Sorry, i guess it wasn't clear. I wanted to know the relation between sensitivity and prevalence mathematically. I know how they are defined. I guess the relation comes in because of the way they are calculated. Sensitivity is tp/(tp+fn) while prevalence is tp+fn/(tp+fn+fp+tn) – Polisetty Feb 14 '18 at 21:03
• Like I mention in my answer, there is no relationship between sensitivity and prevalence. Prevalence is $P(\text{Disease})$ whereas sensitivity is $P(+|\text{disease})$. You cannot say anything about the value of one based on the value of the other. – tddevlin Feb 14 '18 at 21:43
• Harrison's wouldn't get it wrong. Even this link calls it a simplification. med.uottawa.ca/sim/data/Sensitivity_and_Prevalence_e.htm – Polisetty Feb 14 '18 at 21:49

I cannot, of course, speak to the author's intentions, but here would be my reasoning for that statement:

Consider clinical context as a diagnostic test itself. One with very poor sensitivity and specificity, but a test none the less. If you're in the hospital, you're likely to be sick. If you're not in the hospital, you're not likely to be sick.

From this perspective, the actual diagnostic test you perform is actually the second part of two tests done in serial.

• In your explanation, the priori is changing leading to larger posterior probability. That is true. But how does sensitivity itself change is the question. – Polisetty Feb 14 '18 at 21:09
• @Polisetty What if you call a high posterior a positive test? "Clinical Context is itself a test." I think any arbitrarily decided test can be made to depend upon prevalence in this way, so the "test" must be more specifically defined. I think the statement applies to the usual variety of tests based on a threshold of some proxy measurement. – Satwik Pasani Feb 27 '18 at 8:53

This must be a mistake. I think perhaps the author is trying to suggest that positive and negative predictive value (PPV and NPV) are dependent on the prevalence (as well as sensitivity and specificity). These are often discussed with diagnostic tests and, to a clinician, perhaps more valuable than raw interpretation of sensitivity and specificity.

This graph demonstrates the relationship between the PPV and NPV with prevalence, for a test with 95% sensitivity and 85% specificity.

From Mausner JS, Kramer S: Mausner and Bahn Epidemiology: An Introductory Text. Philadelphia, WB Saunders, 1985, p. 221.

@Satwik, @gung and @Tim have already provided a lot of detail, but I'll try and add a small example of how the case of underlying factors may cause such an effect.

## A Key Principle: Bias

Sensitivity/Specificity and ALL statistical tests share the same caveat: it applies only to repeating the same sampling procedure as before in an unbiased manner.

Hospitals are functioning organisations designed to perform biased sampling, using admissions policies to filter the general population down into those requiring admission and treatment. This is very antithesis of scientific procedure. If you want to know how a test performs in different populations it needs to be tested in different populations.

## The latent effect: Correlation

It is rare (or impossible in the real world if you want to be strict) for a diagnostic to be independent/orthogonal to all other risk factors for a disease, so there is some degree of correlation.

If the screen for admission to hospital is positively correlated with the diagnostic then what you will find is that people who pass the admissions test are favourably predisposed for positive results by the diagnostic, proportional to the correlation. Thus true positives are enriched and false negatives are reduced by amounts proportional to the correlation.

This then makes sensitivity appear larger.

## The explanation of the phenomenon

An observation that sensitivity may be higher in a hospital based context is therefore not unrealistic. In fact if the admissions policy is well thought out and fit for purpose one would expect this to occur.

It is not evidence of a breakdown in the assumption that sensitivity and specificity are prevalence independent, rather it is evidence of a biased sampling based on the hospital admission policy.

Which, given a hospital is there to treat people and not to do scientific experiments, is definitely a good thing.

But it does give scientists a headache.