0
$\begingroup$

I am not sure how to call an approach to combine a prediction model with a clinical imaging test.

Model A is a prediction model (eg, Random forest algorithm or Logistic regression) using quantitative information of imaging test B that has been developed in cohort X to predict the probability of disease U and has been validated in cohort Y

Imaging test B is an diagnostic test (ie, MRI) which is widely available ant that describes the presence of disease U using nominal probability categories based on visual assessment (1 unlikely 2 unlikely, 3 equivocal, 4 likely, 5 very likely). In clinical practice, the imaging test B is usually thought to be positive for disease U if it is >=3.

To investigate the impact of model A on imaging test B to detect disease U, the model A will be applied to individuals of cohort Y depending on the result of the imaging test B according to two clinical algorithms:

  1. Apply model A to individuals with a positive imaging test B and interpret all individuals with imaging test B negative as negative (ie, probability = 0).
  2. Apply model A to all individuals with imaging test B negative (<3) and interpret individuals with a positive imaging test B as positive (ie, probability = 1).

Performance measure for both clinical algorithm 1 and 2 is AUC that is calculated for the entire cohort Y

My questions:

I am not sure what this setting should be called, is it a validation in a new setting, is it a model update? Is it a new (ensemble) model?

I considered this manual, however, in my opinion the setting above does not fit any of the described settings:

Collins GS, Reitsma JB, Altman DG, Moons KG; TRIPOD Group. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. The TRIPOD Group. Circulation. 2015;131(2):211-219. doi:10.1161/CIRCULATIONAHA.114.014508

Is there a better way to investigate the impact of Model A on imaging test B in detecting disease U?

I considered https://www.fharrell.com/post/addvalue/, however, I think these suggestions are more suited to compare two quantitative models, not a quantitative model with a qualitative test as described above.

https://arxiv.org/abs/1808.07954 mentioned some techniques how to apply models with clinical information sources (e.g. by building a combined logistic regression model). I think this is a good approach, however, if such a combined model would be applied to all patients undergoing the imaging test B it could lead to the situation that imaging test B is positive for disease U (which is in the clinical practice well accepted), however, the probability of the combined model will predict a very low probability of disease U. Moreover, adding the result of imaging test B to a logistic regression model could lead to a almost perfect separation which is not always easy to deal with (How to deal with perfect separation in logistic regression?)

Finally, would it actually be better to develop and validate a Model that was especially developed to predict the probability of the disease U in individuals with a negative imaging test B?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.