I am training a multiclass classifier (specifically a Random Forest) from consecutively generated data. It is not a time series, but the data could be seen as subsequent events. Some of my features access the labels from all previous data. This makes a cross-validation approach infeasible, because in each fold (apart from the last) I would access labels of previous data that I would not have in practice.
I am considering evaluating the classifier using something similar to the holdout method:
Say I have 1000 data in my set. I use the first 800 data to train my classifier, and the test it against the data at index 801. I proceed to train the classifier with 801 data, and test it against the data at index 802. And so on. I store each classification results to build a confusion matrix that I can then work with as usual.
Is this a good way of evaluating the model? If not, what would be a good approach given the nature of the data?