Lets assume I would like to predict the individual pupil with the best performance in class at an exam. I have around 12000 data samples (12000 pupils in database) over 7 years (10 -16)
I have features like:
-past grades in the last 3 exams,
-past grades of other pupils in the same class,
-the pupils liking towards the subject,
-time since last exam,
-length of the exam,
-the grade the pupil was in when the exam was written,
-number of classmates and so on.
Reading through different cross validation strategies I came across time series splitting approaches and was wondering if they are suitable for my problem.
Generally I could do:
- normal K-fold - just split somewhere
- group K-fold - where one group is a school year
- group K-fold - where one group consists all pupils in the exam (should be almost the same as normal K-fold as the amount of samples is much bigger than the group size)
- time based as in learn year 10; predict 11. Then learn 10, 11; predict 12 etc.
- time based with sliding time window, for instance learn year 10-13 and predict 8 folds, quarter years for the next 2 years. Keep 16 as test.
I'm unsure if I do need time based approaches here, as I only take features which are in the past relative to the current day. I have no direct time or dates as a feature.
One reason I see is that there could be some pattern emerging, like a change in school policy that makes it harder to get good grades in the future. And when I predict exam performance right after start of this policy I normally could not have predicted it that well just based on previous years compared to now taking future folds into account. But then I wonder how important this influence is and if I might loose some of the benefits of doing normal K-fold cross validation.
Any help is greatly appreciated.