How can I update a disease prediction model with new treatment group data while maintaining the original causal relationships?

Question

Context:

• I have a prediction model which predicts the probability of getting a disease.
• This prediction model has been created based on data of patients who did not get any form of treatment.
• I use this model on new patients. Patients which have a probability of higher than X of getting the disease will be treated. Patients with a lower probabilty than X will not be treated. The treatment lowers the probability with Y.

Question:

I want to update my prediction model with the data of the new patients. What is the best way to do that?

---

The problem

You can not simply add the new patients data to the original data and then retrain the model because it will change "the causality" of the model. Since the treatment will interfer with the rest of the variables of the prediction model. I hope the example below will clarify this statement:

Example:

Originally we created a logistic regression model to predict getting lungcancer (yes/no) we used the variables, age, family history of lungcancer(yes/no), gender, currently smoking(yes/no), smokinghistory(yes/no).

We used this model on a new patients. There is a new patient who is a smoker and has a probability of higher than X getting lungcancer and you give him treatment ("lungcancer chance reduction pills") and the patient ends up not getting lungcancer. Now we would like to use the data of this patient to update the original model.

However if we add the data of this patient (smoker) to the original model with as outcome 'not getting lungcancer' the model will be biased with the idea smoking --> not getting longcancer. Which is incorrect.

What is the best way to add new patient data to the model while keeping any 'causal' relationships?

EDIT: To show that updating prediction models with 'treated' patients is a more theoretical methodological problem I shall add an example which I encountered in a business setting:

Originally we created a logistic regression model to predict whether a custumor would stop his mobile phone subscription (yes/no) we used the variables, age, number of send texts, number of calls, years having a subscription, internet usage.

We used this model on a new customers. There is a new customer with a decreasing internet usage (sign of stopping the subscription) and has a probability of higher than X of stopping the subscription. You 'treat' this customer ("call him/her and offer discount") and the customer ends up not stopping the subscription. Now we would like to use the data of this customer to update the original model.

However if we add the data of this customer (low internet usage) to the original model with as outcome 'not stopping his subscription' the model will be biased with the idea low internet usage--> not stopping his subscription. Which is incorrect.

Answer 1

However if we add the data of this patient (smoker) to the original model with as outcome 'not getting lung cancer' the model will be biased with the idea smoking --> not getting lung cancer. Which is incorrect.

I would add a had_treatment boolean in your input variables. That way, the model should understand that "having had treatment" is a major predictor for not developing cancer.

Although the above suggestion is just a way to mitigate the problem that you mention, without suppressing it entirely. If you want to do things really properly, you should refrain from updating your model with new patients, whether you decided to give them treatment or not (if you update the model with patients whom you decided to to give treatment only, this will bias the data in favour of people who have an a-priori lower probability of getting cancer, and you probably don't want that).

So, in a nutshell, one of two ways:

• either include a boolean had_treatment (yes/no) in your input data
• or refrain from adding new patients (including the ones you decided not to give treatment)

One last consideration: when adding new patients data, it's highly likely that this data is biased towards people being at risk. Why would a non-smoking 22 years old with no family antecedent come see you to have a check? Should you decide to include new patients data, you need to counter that bias one way or the other.