1
$\begingroup$

I have data of a class of 50 students. My dataset consists of historical data of students in previous years. Data includes presence, homework grades and final grade of previous 10 years (X, training dataset, has 500 elements).

I have a model trained with this data to predict the final grade.

What would be more accurate way to validate my model? Training it with data a range of years (lets say 2005, 2010) and measuring the error of prediction of 2011 data. Or separating a random sample of the training data and estimating the prediction error of this validation sample?

$\endgroup$
0
$\begingroup$

Why are you not using year as a predictor for the model? Maybe students are getting smarter (or dumber) over time -- hey, you never know!

As to your question of the more accurate way to validate, that is the tricky part. You don't want to have a model that models the training data perfectly, but is so trained for that data, it returns bogus results for other data. Generally, you want to reduce blocking effects in your training data.

In summary:

  1. Recommend adding YEAR as a column in your data set.
  2. Randomly pull a ValidationSet from the full data. What % to pull is a matter of your choice.
  3. Train your model on the leftover data.
  4. Validate against your ValidationSet.

If you don't think you have enough data for separate test and validation sets, you might also consider cross-validation methods.

$\endgroup$
  • $\begingroup$ I can endorse most of this advice and like the "spirit" of it. But I think most practitioners are finding more and more that your recommended method, "split-half" or "training set...test set" validation, is far inferior to more sophisticated cross-validation methods, such as k-fold, which with advances in software and computing power are becoming easier and easier to implement. $\endgroup$ – rolando2 Feb 22 '17 at 18:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.