I learned that sensitivity is not affected by oversampling since it identifies the proportions of correct classification for positive events.
However, I also learned that by doing oversampling of unbalanced data (when the original data has very low percentage of positive cases), we can improve sensitivity (class recall).
The above two statement is contradictory with each other. I wonder if someone can help me sort this out.
The estimation of sensitivity is not dependent on the proportion of positive cases or events in the sample.
Consider the following table:
Disease+ Disease-
Test+ a b
Test- c d
a is the number of indivudals with a certain disease, condition, event or such, and who are classified as positive in the test (true positives). b is the number of individuals who do not have the disease, but who are classified as having the disease (false positives). c is the number of individuals that have the disease, but that are classified as negative (false negatives). d is the number of individuals that do not have the disease, and that are classified as negative (true negatives).
The number of individuals in the sample with the disease is the sum of the true positives and false negatives: a+c. The number of individuals in the sample without the disease is the sum of the false positives and true negatives: b+d.
Sensitivity is the number of true positives divided by the sum of the true positives and false negatives: a/(a+c).
As you can see, sensitivity is only dependent on a and c, so the proportion of positives in the sample ((a+c)/(a+b+c+d)) has no impact on the calculation of sensitivity.
So the first statement in your question is correct - that sensitivity is not affected by oversampling. The second statement is not correct, the estimation of sensitivity does not depend on the proportion of events in your sample. However, the accuracy of the estimated sensitivity depends on the number of observations with the disease or event, so in the estimation of sensitivity, the higher number of observations with the disease or the event, the better.
Update:
I did a small simulation to illustrate that accuracy of the estimate improves as the number of positive cases increase. Suppose we have a condition that appears in 1% of individuals in a population. We have a test with a true but unknown sensitivity of 90%. The goal is to estimate the sensitivity. To our help, we have a gold standard method that can identify all cases. We now sample 1000 randomly selected individuals and apply our test and the gold standard method, and then we calculate the estimated sensitivity of our test. In the simulation below, we do this 100000 times and look at the mean estimated sensitivity as well as the 95% "confidence interval" of the estimate, that is, the range of estimates that occur in 95% of the simulated trials:
set.seed(1) # for reproducibility
n <- rbinom(100000,1000,0.01) # simulating 100000 trials
n <- n[n!=0] # removing trials with no positive cases
p <- rbinom(length(n),n,0.9)/n # simulating estimated sensitivity in each trial
quantile(p, c(0.025, 0.975))
2.5% 97.5%
0.6666667 1.0000000
mean(p)
[1] 0.8999911
So the expected mean is 0.90, which is of course hardly surprising because the true sensitivity is 0.90. The "confidence interval" is wide at 0.67-1.0, so there is a relatively high degree of uncertainty in our estimate.
If we now suppose we oversample some groups that are known to have a higher prevalence of the condition, we might get a proportion of 10% in our sample instead. Let's try that in another simulation:
n <- rbinom(100000,1000,0.1) # simulating 100000 trials
n <- n[n!=0] # removing trials with no positive cases
p <- rbinom(length(n),n,0.9)/n # simulating estimated sensitivity in each trial
quantile(p, c(0.025, 0.975))
2.5% 97.5%
0.8369565 0.9550562
mean(p)
[1] 0.9000495
As we can see, the mean estimated sensitivity is still 0.90, but the "confidence interval" is considerably more narrow at 0.84-0.96. This means that the point estimate of the sensitivity is more likely to be closer to the true sensitivity in this case, simply because we have (on average) 10 times as many positive cases. Also, the confidence intervals of the estimate for a given trial will be much narrower in this case.
