# evaluating an intervention against a predictive model

I have the following situation: a large sample of the population, based on which I build a predictive classification model, about a likelihood of a certain event to happen to certain persons in a certain time frame (based on observable characteristics).

My goal is to reduce the likelihood of this event in the population, so I plan an intervention, and I want to evaluate how it is performed.

for reasons beyond my control I cannot run a controlled experiment, and all the population at a certain point in time will go through this intervention. I can of course apply my predictive model and get a prediction of the likelihood of the event for each person, assuming no intervention.

After a certain time frame, I can actually check the occurrence of the event. I think that all I can do is to compare the actual occurrence (with the intervention) against the predicted occurrence (based on no intervention data). But the predictive model is obviously imperfect, and using cross validation on my initial data I can probably assess the extent of this inaccuracy (e.g. how many false positives and false negatives, etc.)

My question is whether there is a way to assess how good is my intervention in reducing the likelihood of the event?

Any idea/reference would be highly appreciated.

Thanks, Amit

• You first need to find out if the intervention works, so maybe partition subjects into quartiles (1,2,3,4) based on initial predicted probability. Then use a different dose (exposure) for each group, or randomize subjects in different quartiles to different exposure groups. If there is only one treatment (exposure), then treat everyone and compare $after - before$ outcome values while adjusting for covariates or confounders in a regression model. If you have repeated measurements after treatment, then use GEE or RPMANOVA. – JoleT Dec 30 '15 at 16:45
• not possible. in the context - all the relevant population will go through the same intervention. no different doses/exposure is possible and same goes for any other potential partition. this is why I'm looking at comparing actual outcomes with predicted outcomes. it seems that this is all i have. – amit Dec 30 '15 at 17:03
• Ok, then go with this: "If there is only one treatment (exposure), then treat everyone and compare $\delta=after−before$ outcome values while adjusting for baseline values of $before$, and covariates or confounders in a regression model. If you have repeated measurements after treatment, then use GEE or RPMANOVA" – JoleT Dec 31 '15 at 23:23
• @LEP This seems to not utilizing my predictive model. I tend to guess that my predictive model is better than just regressing on the covariates. your method seem to relate to the previous population as a "control", and I think I lose some information by doing it. – amit Jan 3 '16 at 8:02
• You might outline a definition of what good means, and then design the study to answer the question. Does good mean that: event rates are lower in this population compared with previous events in a similar population? Does good mean" events in your treated group are lower than an untreated population. You also have two separate issues: how good the prediction is, and how good the treatment is. If the prediction is the only interest, then compare true events with predicted events and find covariates which explain the accuracy of prediction (that is, difference between truth and prediction) – JoleT Jan 3 '16 at 21:44