What data are used to find the final threshold for a medical diagnostic test?

Question

Suppose I have some blood measurement X whose values correlate with some disease Y (so people with the disease use to have larger values of X). Moreover suppose that the disease is rare, say 1% of a population.

Now I want to develop a new medical diagnosis test.

Should I compute the threshold from say 1000 healthy and 1000 unhealthy people or should the counts correspond to population counts (so for example 100 diseased and 10 000 healthy) because in both cases, the optimal threshold found will be different?

Answer 1

I have three thoughts.

First, shouldn't you be sampling from a population that is similar to the one you will eventually be testing? This is unlikely to be a "normal" population. We generally don't test everyone for everything. We often test because we already suspect the person may have the condition. Yes, there are exceptions. A "regular check up" will generally include some tests, such as blood pressure and body temperature. But most tests are not like that.

Second, diagnosis is often based on multiple tests and observations. A good diagnostician will rely on all the information they have.

Third, while cutoffs and absolute decisions are sometimes necessary, I think this is much less common than we think. The results of testing (for most diagnostic tests) will be ambivalent. They may be very ambivalent, or they may be strong in one direction or the other. And those results will be used to help determine treatment. What is that treatment? Is it potentially harmful? What are its risks? All this should be discussed with the patient.

Therefore, I would emphasize estimating how different results from the test relate to different likelihood of the disease, in combination with other symptoms and traits, and in light of the risks of treatment.

Answer 2

If you oversample the minority class, your model parameters and predictions will be biased. This is usually not a good thing. However, if data collection is expensive, it may be an avenue worth pursuing, especially since you are aware that the optimal threshold will be different.