I am training a multiclass classifier (specifically a Random Forest) from consecutively generated data. It is not a time series, but the data could be seen as subsequent events. Some of my features access the labels from all previous data. This makes a cross-validation approach infeasible, because in each fold (apart from the last) I would access labels of previous data that I would not have in practice.

I am considering evaluating the classifier using something similar to the holdout method:
Say I have 1000 data in my set. I use the first 800 data to train my classifier, and the test it against the data at index 801. I proceed to train the classifier with 801 data, and test it against the data at index 802. And so on. I store each classification results to build a confusion matrix that I can then work with as usual.

Is this a good way of evaluating the model? If not, what would be a good approach given the nature of the data?

  • $\begingroup$ I should probably mention that I have 14 different data sets I can do this with. $\endgroup$ – thrau Aug 27 '16 at 10:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.