4
$\begingroup$

This is a question that has been bugging me because I see this practice done in many medical papers. Here's the scenario: you create a prognostic model from a general population of breast cancer patients with sensible exclusion criteria like no previous cancer. Your variables end up being age, HER2 receptor status, and estrogen receptor status. What I see in many papers is the authors validate the model, but then also test it on multiple groups of patients with very specific characteristics. Like testing performance for HER2+ and ER+ patients only, or only on young patients who are Stage III. They then say things like, "the model performed well on ER+/HER2- patients but not HER2+ patients overall."

Something in my gut says that it is not great practice to train a model on a broad patient population then test it on multiple different subsets of patients (e.g. only Stage III patients, or only patients <50 yrs who are Black). I am not a statistician by training so I have no evidence for this. Anyone know if there are bias issues with this practice?

$\endgroup$
3
  • $\begingroup$ Can you clarify what you mean by 'prognostic'. I assume that it means response to treatment? It seems that what you are describing are sub-populations for which the model does or does not fit well. This indicates that including co-variates such as age, prior health status, etc. might have improved even the fit of the original model on the larger population. However, on average it did fit well enough. But when you then look at sub-populations the possibility to improve the model becomes more apparent. $\endgroup$ – user12719 Jan 25 '13 at 21:11
  • $\begingroup$ By prognostic I mean, "able to predict a patient's outcome." In my example, it could be the model's ability to predict probability of 5-year overall survival. Yes, the model was trained on a broad population of breast cancer patients, but the researchers then test it on a bunch of sub-populations and state conclusions like "the model performed well with X types of patients but not Y types." My gut tells me you can't draw conclusions like that if you trained the model on a broad population. $\endgroup$ – JJM Jan 25 '13 at 21:26
  • $\begingroup$ If X and Y-types were present in the original model, this means that either the model fit for one of these types was poor and/or one of these types was underrepresented. The model could therefore be improved by including co-variates that distinguish these types, e.g. age or by giving them more relative weight. One would have to exclude the possibility that the difference between X and Y is not due the previous reasons (underrepresentation, difference in age). $\endgroup$ – user12719 Jan 25 '13 at 21:38
1
$\begingroup$

It's not so much that there is a problem with bias. It's that the research sounds inefficient and would accomplish more by including these subpopulations (and the variables that define them) in the modelling. I agree with @user12719 that additional covariates would no doubt be useful to include, and I'd add that statistical interactions might emerge as relevant too. That is to say, not only might the original model benefit from new covariates, and not only might it perform better or worse when considering subgroups defined by those covariates, but certain predictors might loom larger or smaller, or even switch signs, depending on the subgroup under consideration. The method you described seems to gloss over much of this.

$\endgroup$
1
  • $\begingroup$ I should have given a little more detail. These "sub-populations" are from an external validation data set from another institution. No data-splitting of the modeling patients. What I was thinking is if testing the model on multiple patient sub-populations from your external validation set raises problems of multiple comparisons (it performs well/worse on some population due to chance). Something just doesn't quite seem right with training the model on a heterogeneous population and testing it on a small homogeneous sub-population of patients (again, from another institution). $\endgroup$ – JJM Jan 26 '13 at 4:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.