I am running Multiple Logistic Regression on a set of around 5000 observations. The series are actually a time series, each observation is a daily average. However, logistic regression treats each day as an independent observation, not knowing what day came after or before.
I separated the last 1/4 of my data (the most recent days) to validate the model. Which means: I apply the coefficients adjusted for the first 3/4 of "calibration/adjustment" data and calculate the GOF metrics (I am using AIC, Area Under ROC curve and pseudo R-squared metrics) two times, calibration and validation periods, being the latter not used to the adjustment of the model parameters.
The calibration/validation of logistic regression models in this manner is something I have not seen very often. I understand that I may lose some information to give the model to adjust coefficients this way, but on the other hand I test it for overfitting or specialization to the given data. I hope to be correct on my trade-off.
The validation period is especially interesting, not only for it containing the most recent observations, but some different patterns have happened within it, so I would be testing if the model that was fit at another period, would capture this different variability. In addition, it looks nice on graphs to plot the fitted model with calibration and validation periods, side by side (below - the light blue bars represent the observed dichotomous variable).
However, the choice of the calibration/validation period is time-dependent (most ancient and recent observations). Therefore, I was suggested to chose the days (observations) to compose these sets in a more random, time-independent way. Would it indeed be a better option in this case? Is the time-dependency of calibration/validation periods a weakness of this method? Should I avoid it either way? Any further thoughts on this?
EDIT: I see I could use k-folded cross validation (here we are :) ). Still, the subsampling could again be made either randomly or time-dependent. The first option seems to be more commonly used, especially when observations are time-independent by their nature (polls, patients...): not the case here.