0
$\begingroup$

I am running Multiple Logistic Regression on a set of around 5000 observations. The series are actually a time series, each observation is a daily average. However, logistic regression treats each day as an independent observation, not knowing what day came after or before.

I separated the last 1/4 of my data (the most recent days) to validate the model. Which means: I apply the coefficients adjusted for the first 3/4 of "calibration/adjustment" data and calculate the GOF metrics (I am using AIC, Area Under ROC curve and pseudo R-squared metrics) two times, calibration and validation periods, being the latter not used to the adjustment of the model parameters.

The calibration/validation of logistic regression models in this manner is something I have not seen very often. I understand that I may lose some information to give the model to adjust coefficients this way, but on the other hand I test it for overfitting or specialization to the given data. I hope to be correct on my trade-off.

The validation period is especially interesting, not only for it containing the most recent observations, but some different patterns have happened within it, so I would be testing if the model that was fit at another period, would capture this different variability. In addition, it looks nice on graphs to plot the fitted model with calibration and validation periods, side by side (below - the light blue bars represent the observed dichotomous variable).

calibration and validation period: fitted model and observations

However, the choice of the calibration/validation period is time-dependent (most ancient and recent observations). Therefore, I was suggested to chose the days (observations) to compose these sets in a more random, time-independent way. Would it indeed be a better option in this case? Is the time-dependency of calibration/validation periods a weakness of this method? Should I avoid it either way? Any further thoughts on this?

Thank you,

EDIT: I see I could use k-folded cross validation (here we are :) ). Still, the subsampling could again be made either randomly or time-dependent. The first option seems to be more commonly used, especially when observations are time-independent by their nature (polls, patients...): not the case here.

$\endgroup$
  • $\begingroup$ If your logistic regression model doesn't incorporate time dependence then your calibration/validation shouldn't, either, and random assignment makes sense. But if you are incorporating time in any way other than your separate daily averages (e.g., multi-day running averages, day-to-day differences, etc.) then you need to use methods designed for time series. $\endgroup$ – EdM Mar 12 '17 at 17:44
  • $\begingroup$ Yep, that makes sense. My days are independent observations. Thank you for the support. $\endgroup$ – ouranos Mar 12 '17 at 19:52
1
$\begingroup$

Default solution for this seems to be k-fold cross validation with randomly assigned data points in each fold.

$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.