I have the following situation: a large sample of the population, based on which I build a predictive classification model, about a likelihood of a certain event to happen to certain persons in a certain time frame (based on observable characteristics).
My goal is to reduce the likelihood of this event in the population, so I plan an intervention, and I want to evaluate how it is performed.
for reasons beyond my control I cannot run a controlled experiment, and all the population at a certain point in time will go through this intervention. I can of course apply my predictive model and get a prediction of the likelihood of the event for each person, assuming no intervention.
After a certain time frame, I can actually check the occurrence of the event. I think that all I can do is to compare the actual occurrence (with the intervention) against the predicted occurrence (based on no intervention data). But the predictive model is obviously imperfect, and using cross validation on my initial data I can probably assess the extent of this inaccuracy (e.g. how many false positives and false negatives, etc.)
My question is whether there is a way to assess how good is my intervention in reducing the likelihood of the event?
Any idea/reference would be highly appreciated.