When we were taught about Bayesian probability, we often saw the following example: in a population, there are 5% of people who has disease X, and among the people who have disease X, the current technology (i.e., a test) can correctly detect this fact 98% of the time. Then we are asking about the probability that a person indeed has disease X given the test states a person has disease X.
In the machine learning class, when talking about the classification problem, we usually have a following setting: assume we have a data set, one column is the diagnostic results, e.g., whether a patient is diagnosed as having disease X; another column is whether this patients indeed has disease X. Then with many attributes of the patients, we build up a machine learning model that will be trained on a training data set, which is say 80% of my data set with BOTH the diagnostic results and the "true label" of the patients, then we try this model on the remaining 20% of the dataset, then we compare the model prediction with the "true label".
Here is my question: in reality like this diagnostic context, how those "true labels" are obtained, especially for some rare disease X, that we don't know much about. I guess the key here is whether a person has disease X or not, needs a operationalizable definition. For example, say we define a person has fever, if his body temperature is about 39 Celsius. Here a person's body temperature is measurable. In this case, there is a "true label" (whether a person has fever or not). And one could invent some diagnostic test that does not need to take a person's body temperature (e.g., heart rate?), but still able to tell something about the person's body temperature.
Here, in general for clinical diagnostic data, I believe usually there will be just a column stating whether a patient is diagnosed as having disease X. There is no column stating whether this patient indeed has disease X. I am wondering how in practice this the validity (or accuracy) of a medical test is done?
Suppose there is a clear medical definition of disease X, say the medical definition could be "if there is $\beta$ amount of X virus per ml of blood, then the patient has X disease". Here, X virus has clear biological definition, and given a tube of blood, there is a true amount of X virus in it (could be zero, but whatever it is, there is a true amount). Now for the current technology, we have a test that can be used to detect the amount of X virus in the sample blood. If it is in the lab, I think the accuracy of the test is able to be measured in this context, which is: the experimenter has fostered a certain amount of X virus and put them into different sample blood. In this case, the experimenter knows the TRUE amount of X virus in each sample blood, then the experiment applies this test to these sample blood and compare the results from the sample blood to the TRUE amount, then calculate the accuracy of this test. Say if this test is very accurate (based on some threshold), then this test is implemented in the hospital.
However, as lab is only a small niche of the reality, now when this test is implemented in the hospital, and it will generate yes/no answer to patients, but how are we able to check whether this test indeed generates the accuracy rate in real life as it has been measured in the lab?
