I am not sure how to call an approach to combine a prediction model with a clinical imaging test.

Model A is a prediction model (eg, Random forest algorithm or Logistic regression) using quantitative information of imaging test B that has been developed in cohort X to predict the probability of disease U and has been validated in cohort Y

Imaging test B is an diagnostic test (ie, MRI) which is widely available ant that describes the presence of disease U using nominal probability categories based on visual assessment (1 unlikely 2 unlikely, 3 equivocal, 4 likely, 5 very likely). In clinical practice, the imaging test B is usually thought to be positive for disease U if it is >=3.

To investigate the impact of model A on imaging test B to detect disease U, the model A will be applied to individuals of cohort Y depending on the result of the imaging test B according to two clinical algorithms:

  1. Apply model A to individuals with a positive imaging test B and interpret all individuals with imaging test B negative as negative (ie, probability = 0).
  2. Apply model A to all individuals with imaging test B negative (<3) and interpret individuals with a positive imaging test B as positive (ie, probability = 1).

Performance measure for both clinical algorithm 1 and 2 is AUC that is calculated for the entire cohort Y

My questions:

I am not sure what this setting should be called, is it a validation in a new setting, is it a model update? Is it a new (ensemble) model?

I considered this manual, however, in my opinion the setting above does not fit any of the described settings:

Collins GS, Reitsma JB, Altman DG, Moons KG; TRIPOD Group. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. The TRIPOD Group. Circulation. 2015;131(2):211-219. doi:10.1161/CIRCULATIONAHA.114.014508

Is there a better way to investigate the impact of Model A on imaging test B in detecting disease U?

I considered https://www.fharrell.com/post/addvalue/, however, I think these suggestions are more suited to compare two quantitative models, not a quantitative model with a qualitative test as described above.

https://arxiv.org/abs/1808.07954 mentioned some techniques how to apply models with clinical information sources (e.g. by building a combined logistic regression model). I think this is a good approach, however, if such a combined model would be applied to all patients undergoing the imaging test B it could lead to the situation that imaging test B is positive for disease U (which is in the clinical practice well accepted), however, the probability of the combined model will predict a very low probability of disease U. Moreover, adding the result of imaging test B to a logistic regression model could lead to a almost perfect separation which is not always easy to deal with (How to deal with perfect separation in logistic regression?)

Finally, would it actually be better to develop and validate a Model that was especially developed to predict the probability of the disease U in individuals with a negative imaging test B?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.