This is a question that has been bugging me because I see this practice done in many medical papers. Here's the scenario: you create a prognostic model from a general population of breast cancer patients with sensible exclusion criteria like no previous cancer. Your variables end up being age, HER2 receptor status, and estrogen receptor status. What I see in many papers is the authors validate the model, but then also test it on multiple groups of patients with very specific characteristics. Like testing performance for HER2+ and ER+ patients only, or only on young patients who are Stage III. They then say things like, "the model performed well on ER+/HER2- patients but not HER2+ patients overall."
Something in my gut says that it is not great practice to train a model on a broad patient population then test it on multiple different subsets of patients (e.g. only Stage III patients, or only patients <50 yrs who are Black). I am not a statistician by training so I have no evidence for this. Anyone know if there are bias issues with this practice?