Combine a probability from a prediction model and the disease prevalence?

Question

I created a prediction model using a multiple logistic regression on a non-representative sample of the target population.

With this model, I can get a probability for a new patient of belonging to class 0 (No Disease) or 1 (Disease) of my outcome variable. The probability in the target population of belonging to class 1 (i.e. the prevalence) is 10%, vs 40% in my sample.

Let's consider the 2 following situations

1. The patient belongs to the target population (Disease prevalence : 10%)
2. The patient belongs to a population where the disease prevalence equals the disease prevalence in the sample used for the model development (40%)

Questions

If I apply the model and get a 50% predicted probability of Disease, would the true probability be identical in these 2 situations (50%)? In other words, do have to take into account the disease prevalence to get the probability that my patient has the disease? And if so, how?

Or do I need to go through a sensitivity/specificity calculation and then (positive/negative) predictive values and therefore use a decision threshold based on the predicted probability?

Answer 1

I can offer a real world answer, but it is anecdotal. From a long life sampling way too many doctors, what doctors actually weigh very heavily is what they are seeing, to the point they fail to diagnose situations that do not fit the population.

To make an example, doctors often see sebaceous (fatty) cysts under the skin, so tend to diagnose all subcutaneous cysts as sebaceous. They are trained to do so. This fits the second situation you describe, where the sample has higher prevalence. Doctors see self selected, not random patients. (As you might guess, my cyst was something else entirely, so the real wrinkly to the problem you state is that neither population has only one disease to contend with)

In terms of infectious disease, again doctors, knowing infectious diseases are infectious, also know the probability of a disease being present is heavily weighted by what is "going around". In flu season they are more likely to diagnose flu, outside flu season they may diagnose cold for the same symptoms (or today Covid-19) The probability is affected by what is in the self selected sample (which is all doctors ever see) and the fact that a disease is infectious or not also affects the probability that a given set of symptoms indicates a given disease. Meaning people pass flu to each other so seeing other patients with flu increases the probability that the next patient has the flu. Seeing a patient with non infectious disease (like cyst) does not affect the probability AS MUCH. But because of self selection doctors are still right more often than wrong when diagnosing similar symptoms as the most common disease which reflect that symptom.

So I think Im saying, yes the probability goes up in the 40% group, and that is the way doctors actually diagnose. I'm also saying doctors never see the 10% group.