I am building a predictive model that forecasts a student's probability of success at the end of a term. I’m specifically interested in whether the student succeeds or fails, where success is usually defined as completing the course and achieving 70% or more points out of the total points possible.
When I deploy the model, the estimation of success probability needs to be updated through time as more information becomes available -- ideally immediately after something occurs, like when a student submits an assignment or gets a grade on one. This updating sounds sort of Bayesian to me, but given my training in educational statistics, that is a little outside my comfort zone.
I have so far been using logistic regression (actually lasso) with a historical data set containing week-based snapshots. This data set has correlated observations, since each student has $TermLength/7$ observations; observations for one student are correlated. I am not specifically modeling the correlation within a particular student’s weekly observations. I believe that I would only need to consider that in an inferential setting since standard errors would be too small. I think--but not sure on this--that the only problem arising from the correlated observations is that I need to be careful when I cross-validate to keep clustered observations in one subset of the data, so that I don’t get artificially low out-of-sample error rates based on making predictions about a person the model has already seen.
I am using R’s glmnet package to do a lasso with a logistic model to generate a probability of success/failure and to automatically pick predictors for a particular course. I have been using the week variable as a factor, interacted with all other predictors. I don't think this differs in general from just estimating individual week-based models except that it gives some idea of how there may be some common model that holds throughout the term that is adjusted via various risk adjustment factors at different weeks.
My main question is this: Is there a better way of updating classification probabilities over time rather than just dividing up the data set into weekly (or other interval-based) snapshots, introducing a time-period factor variable interacted with every other feature, and using cumulative features (cumulative points earned, cumulative days in class, etc)?
My second question is: am I missing something critical here about predictive modeling with correlated observations?
My third question is: how can I generalize this to a real-time updating, given I'm doing weekly snapshots? I'm planning on just plugging in variables for the current weekly interval, but this seems kludgey to me.
FYI, I'm trained in applied educational stats but do have a background in mathematical stats from a long time ago. I can do something more sophisticated if it makes sense but I need it explained in relatively accessible terms.