Updating classification probability in logistic regression through time

Question

I am building a predictive model that forecasts a student's probability of success at the end of a term. I’m specifically interested in whether the student succeeds or fails, where success is usually defined as completing the course and achieving 70% or more points out of the total points possible.

When I deploy the model, the estimation of success probability needs to be updated through time as more information becomes available -- ideally immediately after something occurs, like when a student submits an assignment or gets a grade on one. This updating sounds sort of Bayesian to me, but given my training in educational statistics, that is a little outside my comfort zone.

I have so far been using logistic regression (actually lasso) with a historical data set containing week-based snapshots. This data set has correlated observations, since each student has $TermLength/7$ observations; observations for one student are correlated. I am not specifically modeling the correlation within a particular student’s weekly observations. I believe that I would only need to consider that in an inferential setting since standard errors would be too small. I think--but not sure on this--that the only problem arising from the correlated observations is that I need to be careful when I cross-validate to keep clustered observations in one subset of the data, so that I don’t get artificially low out-of-sample error rates based on making predictions about a person the model has already seen.

I am using R’s glmnet package to do a lasso with a logistic model to generate a probability of success/failure and to automatically pick predictors for a particular course. I have been using the week variable as a factor, interacted with all other predictors. I don't think this differs in general from just estimating individual week-based models except that it gives some idea of how there may be some common model that holds throughout the term that is adjusted via various risk adjustment factors at different weeks.

My main question is this: Is there a better way of updating classification probabilities over time rather than just dividing up the data set into weekly (or other interval-based) snapshots, introducing a time-period factor variable interacted with every other feature, and using cumulative features (cumulative points earned, cumulative days in class, etc)?

My second question is: am I missing something critical here about predictive modeling with correlated observations?

My third question is: how can I generalize this to a real-time updating, given I'm doing weekly snapshots? I'm planning on just plugging in variables for the current weekly interval, but this seems kludgey to me.

FYI, I'm trained in applied educational stats but do have a background in mathematical stats from a long time ago. I can do something more sophisticated if it makes sense but I need it explained in relatively accessible terms.

Answer 1

You can't get there from here. You need to start with a different model. I would keep the weekly snapshots and build a stochastic model around transitions in each student's state variable. Suppose there are 10 weeks, which gives 11 "decision'' points, $t_0, t_1, \ldots, t_n$. The state at $t_i$ is $(Z_i,S_i)$, where $Z_i$ is 1 or 0, according as the student is enrolled or not; and $S_i$ is the score at that point (the sum of test and homework scores to date). Initial values are $(1,0)$. You have two transitions to worry about: $Prob(z_i=0|s_{i-1})$ and the distribution of $S_i$.

The dropout probabilities are not stationary, since you will get a binge of dropouts just before the final drop-without-penalty date. But you can estimate these from past data.You can also estimate the probability of dropping out as a function of current (dismal) performance.

The $S$ scores are a random walk on a binomial outcome (number of correct answers on a test of $n$ items, say). You can probably assume conditional independence -- assume a latent "talent" parameter for each student, and conditional on that value, each new score is independent of current performance. You could test this assumption against your historical data ... do failing students change their study habits and pull off a win? But most students behave true to form ... so a conditionally independent model should work OK.

So basically, a student fails if a $Z$ score transitions to 0, or the $S$ score fails to cross the 70\% pass threshold.

Let's look more closely at the $S$ process. To simplify the model, assume that evaluation involves obtaining 70 points or more from a total of 100 possible points, obtained from 10 test items each week.

At baseline, a student's pass probability is simply the pass rate of the previous class.

At time 1, the student has earned $S_1$ points (or dropped out). He passes if he can earn at least $70-S_1$ points out of 90. this is a binomial problem, which I can easily calculate if I know the student's probability of success. This will no longer be the "class average"; I need to adjust in light of the student's success thus far. I would use a table from past experience for this, but you could do a weighted average of the overall class success rate and the student's personal success. Bayes' Rule should help here.

As a bonus, you can calculate a range of probabilities, which should narrow as the term progresses. In fact, strong students will cross the 70\% mark before the end of term, and their success will be certain at that point. For weak students, failure will also become certain before the end.

RE: question 3. Should you go to continuous time? I wouldn't, because that puts one in the realm of continuous time stochastic processes and the math involved is above my pay grade. Not only that, you are unlikely to get a substantially different outcome.

The best way to upgrade the model I have outlined is not to go to continuous time, but to adjust the transition probabilities on the basis of prior experience. Perhaps weak students fall further behind than an independence model would predict. Incorporating inhomegeneity would improve the model more than going from discrete to continuous time.

Answer 2

When I train predictive models for a similar type of deployment, I make sure my datasets have some sort of Term_End_Date so I can derrive the length of time left until the term ends. This will probably end up being a significant predictor in your model.

Regarding the question of correlated observations, I suppose it matters how big of a repository of data that you have. If possible, I would randomly select 1 observation for each student, stratified on [# of weeks until Term End]. I would also grab from older terms, if possible. If you don't have enough data to do that, maybe you can try a re-sampling method like bootstrap.

I think the most important thing if you have a small dataset is keeping enough data as a holdout to make sure your final model is stable.

I think when you're all done, and you have a scoring formula, it'll be pretty easy to implement. But yes, you should still be plugging in the weekly x variables that you'll need to calculate the score - but this sounds more like a data collection issue and less about model implementation.