Cross Validation Strategy when having subtle time dependence

Question

Lets assume I would like to predict the individual pupil with the best performance in class at an exam. I have around 12000 data samples (12000 pupils in database) over 7 years (10 -16)

I have features like:
-past grades in the last 3 exams,
-past grades of other pupils in the same class,
-the pupils liking towards the subject,
-time since last exam,
-length of the exam,
-the grade the pupil was in when the exam was written,
-number of classmates and so on.

Reading through different cross validation strategies I came across time series splitting approaches and was wondering if they are suitable for my problem.

Generally I could do:
- normal K-fold - just split somewhere
- group K-fold - where one group is a school year
- group K-fold - where one group consists all pupils in the exam (should be almost the same as normal K-fold as the amount of samples is much bigger than the group size)
- time based as in learn year 10; predict 11. Then learn 10, 11; predict 12 etc.
- time based with sliding time window, for instance learn year 10-13 and predict 8 folds, quarter years for the next 2 years. Keep 16 as test.

I'm unsure if I do need time based approaches here, as I only take features which are in the past relative to the current day. I have no direct time or dates as a feature.

One reason I see is that there could be some pattern emerging, like a change in school policy that makes it harder to get good grades in the future. And when I predict exam performance right after start of this policy I normally could not have predicted it that well just based on previous years compared to now taking future folds into account. But then I wonder how important this influence is and if I might loose some of the benefits of doing normal K-fold cross validation.

Any help is greatly appreciated.

Answer 1

You should use time based approaches if the data from year x depends on the data from year x-k . In this case it seems like the data probably does depend on previous years' data so it would make sense to use a time based approach.

Another indicator that you should use time based approaches is if in your real-time application you receive the data in a time based way. That is let's say you build a predictor and now your goal is to follow a class from k-12 and at every year predict who the best performing student will be. In this case you will be receiving the data in a time based way. First you receive grades from the year 1, then you have data from year 1 and year 2, and so on.

So if in "real-life" you receive your data in a time based way, it is always good to perform error analysis using the same method.

One more thing you could check/do: you could check if, or assume if you think it makes sense, the data has some Markovian property, i.e. the performance in year x of a student only depends on on their performance from year x-1 and x-2. Then you could create a feature vector that includes all of the features from year x as well as x-1 and x-2. Now you can just do regular cross validation.

Suggestion independent of question: In fact when you have time based data it is almost always useful to try using "gradient" features- features which are the difference between year x and year x-1 or x and x-2 since perhaps if a student made a big jump in performance between two years, that might mean they also will the following year.

Answer 2

I found this post Cross-validation techniques for time series data

and in there he is referencing some paper Bergmeir et al. "A note on the validity of cross-validation for evaluating time series prediction" explaining that K-fold can be better if only autoregressive models are used.

As far as I can see I only use autoregressive features (this is mathematically more precise for what I meant with "no direct time features"), so I might stick with the normal K-fold for now.