Population proportion inference: what statistical error am I committing here? I am struggling to understand the following paradox.
Suppose we have a large population of people who exhibit different values of a trait, e.g. they are sick with different strains of a virus. Suppose I'm trying to cure them.
Consider 2 scenarios:


*

*I have a vaccine. I survey a subset S of these people and count how many of them my vaccine is able to cure: say, it's 1000 people and 95%. Then I declare, using the classic formula for inference of population proportion and the two sigma rule, that my vaccine will be able to cure something like 0.95 +- 2*sqrt(0.95*0.05/1000) = 94% .. 96% of the general population. So far so good.

*I don't yet have a vaccine. I take a subset S of these people and use them to design a vaccine, declaring success when I am able to cure 95% of them.
Here's the paradox: in case 2, can I still claim that I will be able to cure between 94% and 96% of the general population?
No. For example, the wider population might contain strains of the virus that my vaccine didn't account for. E.g. if the frequency of different strains is distributed by Zipf, then the number of different strains in a population obeys Heaps' law, and even if I was able to cure 100% of my sample, I'm still probably nowhere near 100% of the whole population because it contains a lot of new virus strains.
My question is: Exactly why is population proportion inference not applicable in case 2? Looking at assumptions for it stated in Wikipedia doesn't help much: my sample of patients is large enough, independent, and selected randomly. Is there a name for this class of statistical error?
 A: There are two issues in this paradox. This first is a violation of the assumptions in situation 2, and the second is a general statistical issue in the interpretation of the final vaccine's effectiveness.
Firstly, the sample is no longer a simple random sample - in fact it's no longer a probability sample at all. Imagine if instead of testing a vaccine, I was estimating the number of people who had a blue sticker on their forehead. I selected a sample, stuck blue stickers on their foreheads and then counted how many had blue stickers on their foreheads. Clearly I've intervened - this is no longer a simple random sample from the population.
This is a more extreme example than what you appear to be proposing, but if you design your vaccine to suit the sample you've drawn, then you're doing a similar thing. However, it's easy to get around this: just make a list of all vaccines you're going to test before you select the sample and then test all of them until you get a "hit".
This brings about the second issue: you're not adjusting for multiple comparisons. Although each individual test in this case does satisfy the assumptions, the way you choose the vaccine influences the interpretation of the success of the vaccine. You're only taking the most successful vaccine, but this most successful vaccine is more and more likely (as you test more and more vaccines) to be an overly rosy representation of how that vaccine actually performs across the population. That is, it's more and more likely to be a vaccine test that happens to have a confidence interval that does not capture its true population value.
