I'm working on a time-series, binary classification dataset, where I'm doing cross validation as a moving window as in the diagram below:
So I'll use the first three "shifts" as cross validation, for example, and leave the last bit as an actual test-set. I don't have a large amount of data, only about 360 rows of data in total with 12 features. I'm using Random Forest. The problem is this: I run cross validation and let's say I get an accuracy of 60%, pretty good. Then, I change my training/validation sets by ONE point. So, for example, initially my sets are like this:
- Training: rows 1-150
- CV Fold 1: rows 151-180
- CV Fold 2 rows 181-210 etc.
Now, I shift it by one so that -
- Training: rows 1-149
- CV Fold 1: rows 150-179
- CV Fold 2 rows 180-209 etc.
This seemingly tiny change has a very large impact on my accuracy - it drops from 60%, to something like 55%. Changing this not by one, but by 5 rows might drop it down to like 53%.
First attempt at fixing the issue - I realized I was leaking data because I was using my cross validation set for feature selection, so within my CV sets, I was leaking information from future "folds" to past folds about which features are best. I stopped doing this. The issue persists. What could be causing this?
Second attempt at fix - PLEASE SEE ATTEMPT 5 INSTEAD. I thought maybe this was some odd feature of Random Forest, so I tried all the same things but with SVM and although the variation is to a lesser degree, the accuracy definitely jumps around a few percentage points when I change the boundary even slightly. With SVM, I noticed that the accuracy doesn't ALWAYS decrease - sometimes it increases slightly when I change the boundary, but again, very unstable.
Third attempt at fix - I thought that maybe the boundary I chose initially just happened to be around some very significant/outlier rows of data that have a disproportional impact on the model. So, I changed the initial start/end points of the training/validation set by 30 rows or so, and the issue more or less still persisted.
Fourth attempt at fix - This whole time, I had a sneaking suspicion that simply the low amount of data was the culprit. So, I added 300 more data points, basically doubling the amount of training data, re-ran the cross validation and STILL a single shift in the boundary results in multiple percentage points loss of accuracy. I have no idea what could be causing such instability of performance.
Fifth attempt at fix - Although in my second attempt, I tried changing the method/algorithm, I tried it again in a more detailed analysis and found that the issue seems to be limited to RandomForest. SVM is slightly unstable, but nowhere near the same degree. KNN isn't affected by the shift nearly as much either. 1 point change results in no change in performance, 5 points decreases performance by several percentage points, 10 doesn't make it any worse than 5. Why is RandomForest so extremely sensitive to the training set? It seems like RandomForest is overfitting to a very strong degree, but why would that be (if other methods don't overfit)?