0
$\begingroup$

Lets assume I would like to predict the individual pupil with the best performance in class at an exam. I have around 12000 data samples (12000 pupils in database) over 7 years (10 -16)

I have features like:
-past grades in the last 3 exams,
-past grades of other pupils in the same class,
-the pupils liking towards the subject,
-time since last exam,
-length of the exam,
-the grade the pupil was in when the exam was written,
-number of classmates and so on.

Reading through different cross validation strategies I came across time series splitting approaches and was wondering if they are suitable for my problem.

Generally I could do:
- normal K-fold - just split somewhere
- group K-fold - where one group is a school year
- group K-fold - where one group consists all pupils in the exam (should be almost the same as normal K-fold as the amount of samples is much bigger than the group size)
- time based as in learn year 10; predict 11. Then learn 10, 11; predict 12 etc.
- time based with sliding time window, for instance learn year 10-13 and predict 8 folds, quarter years for the next 2 years. Keep 16 as test.

I'm unsure if I do need time based approaches here, as I only take features which are in the past relative to the current day. I have no direct time or dates as a feature.

One reason I see is that there could be some pattern emerging, like a change in school policy that makes it harder to get good grades in the future. And when I predict exam performance right after start of this policy I normally could not have predicted it that well just based on previous years compared to now taking future folds into account. But then I wonder how important this influence is and if I might loose some of the benefits of doing normal K-fold cross validation.

Any help is greatly appreciated.

$\endgroup$
1
$\begingroup$

You should use time based approaches if the data from year x depends on the data from year x-k . In this case it seems like the data probably does depend on previous years' data so it would make sense to use a time based approach.

Another indicator that you should use time based approaches is if in your real-time application you receive the data in a time based way. That is let's say you build a predictor and now your goal is to follow a class from k-12 and at every year predict who the best performing student will be. In this case you will be receiving the data in a time based way. First you receive grades from the year 1, then you have data from year 1 and year 2, and so on.

So if in "real-life" you receive your data in a time based way, it is always good to perform error analysis using the same method.

One more thing you could check/do: you could check if, or assume if you think it makes sense, the data has some Markovian property, i.e. the performance in year x of a student only depends on on their performance from year x-1 and x-2. Then you could create a feature vector that includes all of the features from year x as well as x-1 and x-2. Now you can just do regular cross validation.

Suggestion independent of question: In fact when you have time based data it is almost always useful to try using "gradient" features- features which are the difference between year x and year x-1 or x and x-2 since perhaps if a student made a big jump in performance between two years, that might mean they also will the following year.

$\endgroup$
  • $\begingroup$ Thanks, just to clarify: I'm predicting the best student in every exam, not in a year. The year just seemed like a natural way to organize the folds. I'll look more into the gradient features, thanks $\endgroup$ – jens0r Mar 10 '17 at 3:23
  • $\begingroup$ I'm not sure if and to what extend the data is dependend on the previous years, especially since I calculate the feature only relative to the current exam. so having some formal assessment would be good. Do you know about an easy to use implementation for testing the time dependence in features (preferably python)? $\endgroup$ – jens0r Mar 10 '17 at 3:23
  • $\begingroup$ In real life the data comes in batches of 10 exams per week. Would you suggest that LeaveOneGroupOut with all the studends sitting those 10 exams is a good method, or too much variance? For now I tend towards the approach of having time based sliding window and predicing the next quarter of a year in the validation set. I though it might be best to just test all CV strategies and choose the one which gives me best performance on the test set. $\endgroup$ – jens0r Mar 10 '17 at 3:24
0
$\begingroup$

I found this post Cross-validation techniques for time series data

and in there he is referencing some paper Bergmeir et al. "A note on the validity of cross-validation for evaluating time series prediction" explaining that K-fold can be better if only autoregressive models are used.

As far as I can see I only use autoregressive features (this is mathematically more precise for what I meant with "no direct time features"), so I might stick with the normal K-fold for now.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.