Efficiency / Accuracy of Medical Tests for unknown diseases Context : A few days ago, a few friends and I were trying to conduct a little study on the possible distribution of number of asymptomatic people infected by COVID-19 with the help of a slightly modified version of the SIR Model. However, while trying to digest those things, a question struck me suddenly.

Let us consider an unknown disease (e.g. COVID-19) which has started breaking out recently in a community. After a few months, we have a lot of possibly infected people. Now, some researchers claim that they have devised a first-of-its-kind medical test which determines whether a person is infected or not. Suppose, we find a few people who were tested to be infected, but they didn't have any observable symptoms. The researchers claim that those persons are asymptomatic.
But it is possible that a normal person who was was tested to be infected, was actually not infected. Instead, due to some errors in the medical test procedure, some normal people are labelled as infected (and hence, asymptomatic). My question is that in scenarios like this, how can a brand claim that their vaccine is "90% effective", "95%  effective" etc. ? How do the medical companies judge how much effective their vaccine is (Ignore side-effects and other factors) ? It may happen that a normal person is tested to be infected due to an error in the medical test, and in that case, the final accuracy of the medical test is hampered.
In cases like this, does anyone know how can we find out how accurate the test is ? Is there some sort of statistical method which can answer this ?
 A: First of all the tests for vaccines are not solely based on a PCR type of test but also on symptoms (which they separate into 'confirmed COVID-19' and 'confirmed severe COVID-19'). But anyway, since those symptoms are mild and may happen due to many other cold viruses and bacteria, let's assume that the efficiency is solely based on PCR type of tests.
Your idea seems to be the following thought: if vaccine efficiency testing is based on PCR-tests and PCR-tests are unreliable, does this make the vaccine efficiency testing unreliable?
In this answer, which is in short 'No, it is not unreliable', I would like to stress two points:

*

*Indeed, the PCR test might be not very accurate.
If perform a test with a particular person then indeed you can not be very sure whether or not the individual person is sick/infectious/infected or not.
However, on a population level the rise or decrease in positive PCR tests does have a more accurate meaning. The probability of a type I error (claiming an anomaly, patient is sick, while it is not) will decrease a lot when the sample size is very large.
What this means for the vaccine efficiency testing: If you have two groups one treated with placebo and the other treated with vaccine and you find that they greatly differ in the number of positive PCR tests, then this will be very unlikely in the case that the vaccine has not any effect.
(I would say that we can actually reason the other way around and the vaccine proofs that the PCR test works reasonably accurate. Otherwise you can not explain the very unlikely low amount of PCR tests among vaccinated people.)


*A hypothesis test is indeed not only a test for the specific scientific hypothesis, but for the entire test procedure. The statistical model relates to the outcomes (e.g. the hypothesis is that among the PCR verified infections we find the same number of people from the vaccinated group as from the placebo group).
When the hypothesis test fails, then this means we have experienced an unlikely event, an anomaly, that can not be explained given a set of assumptions.
Typically the assumption that is considered to be rejected is the null hypothesis/theory, which is the theory that there is no effect of the vaccine (and we should accept the alternative that the vaccine works).
However, there are multiple assumptions underlying the procedure that generated the data and test-results and possibly those are wrong. It might be that the null hypothesis is correct, but that the anomaly is generated because of our testing procedure or our way of analysis.
I would say that these criticisms are weak arguments against the efficiency of the vaccine (more about that in the following 3rd point).

*

*While the PCR test might not be accurate on an individual level, it is a reasonable probe to test the amount of infections in larger groups. Discussions about the curves of positive cases and how these might relate to changes in testing procedures rather than changes in the crude amount of infections are interesting and useful. However, for the testing of the vaccine, we deal with two groups that have been treated equally and a result that is obviously an anomaly due to the treatment and not due to aspects of the testing. There are many more PCR positive tests among the placebo-treated people than among the vaccine-treated people. This is a very unlikely event that can be considered evidence that the vaccine is efficient in preventing the infection (or at least it prevents positive PCR tests).

*It is clear that an anomaly has been detected. What has caused it is not clear (and the data can not tell us). By assumption we believe that it is the falsification of the null hypothesis (the null hypothesis is that the vaccine is not effective, so the falsification of that means that the vaccine is effective). But maybe the test could be the cause of the anomaly. I would consider this very unlikely. The statistical model is not very far stretched and there are not weird assumptions about the distribution of the final outcome (it is a simple binomial distribution). If you have two groups of equal size, one treated with vaccine and the other with placebo, and among the people with confirmed COVID-19 you only find a few people from the vaccine group, then this is very unlikely and should be considered an anomaly. This is not a vague result that is in some gray area. Statistically speaking there is not some alternative way to look at it that would change the probabilities such that it is not significant anymore. The results are extremely significant.

One third point that must be made, and this is not statistical anymore or is at the border of statistics:


*The numbers only prove that the vaccine works (is efficient) against PCR detections and mild symptoms. I do not think that it would be useful to discuss those numbers and the statistical issues related to it. The numbers are very convincing.
The implications however are open for discussion.

*

*How long does the immunity last?

*What does the immunity mean for cross-immunity (against future coronaviruses or mutations of the current virus) and does it possibly weaken our immune system in that aspect?

*How does the reduction of PCR detections translate to the protection against severe cases and protection among weaker people?

*Is the immune response after vaccination stronger in comparison to the immune response from natural infection?

*If the vaccine works in the tested population during summer, will it also work as effectively in another population during winter (does the test have external validity)?

*Etc.




Some mathematics
If it is indeed true that the PCR tests are not very accurate, then this would actually work against the vaccines and they might be more effective than we think.
Say an infected person has a probability of 99.9% for a positive PCR test and 0.1% for a negative PCR test.
Say an non-infected person has a probability of 0.1% for a positive PCR test 99.9%  for a positive PCR test.
Those figures are actually not very bad, but I believe that there are not many indications that it is worse. If the PCR-tests would be false, then you would not find the small 0.1%-ish percentages of positive PCR-tests among non-infected populations. 
Now if we have 15000 vaccinated people and 15000 placebo people. Let's assume that the virus is 100% effective (nobody get's sick) and that among the placebo people 1% get's sick during the experiment.
Then you will have 15000 non-infected vaccinated people (leading to 15 positive PCR tests), 14850 non-infected placebo people (leading to, rounded up, 15 positive PCR tests) and 150 sick placebo people (leading to, rounded up, 150 positive PCR tests). The result will be 15 positive PCR tests among the vaccinated people and 165 positive PCR tests among the placebo people. The conclusion would be that the vaccine reduces the infections from 165 down to 15, 91% effectivity, while in reality, it is 100% effective.
