In certain arenas, it's valuable to be able to intervene early on to prevent problems from getting worse, because after a certain point there's not much you can do. Two examples might be public health or education.
Say we have a bunch of information about students -- their demographic info, their past transcripts, their grades in a course as it progressed, the ultimate outcome (A, B, C, D, F) etc. -- and we're interested to know, in some sense, at what point their outcome is "fixed". (I'm actually interested in knowing at what point changes in their behavior or teacher interventions aren't going to change anything, but I don't have data from experiments with interventions done at different points in time or anything like that.)
My current thinking is that I ought to build a series of predictive models based on information that was available on the first day, fifth day, tenth day, etc. Then I would evaluate the accuracy of the probability distributions over the possible grades that each of these models spits out for each student (perhaps with the Brier score) and plot this vs. the date of the information which the model takes into account. Then I could plot on the same graph how the average variance of the probability distributions changes as time goes on.
This would give me an idea of how certain we can be of a student's outcome as time goes on and perhaps reveal a point after which we could guess that interventions would no longer be effective.
Does this seem reasonable? Can you think of better ways to do this?
