2
$\begingroup$

I'm working on a time-series, binary classification dataset, where I'm doing cross validation as a moving window as in the diagram below:

enter image description here

So I'll use the first three "shifts" as cross validation, for example, and leave the last bit as an actual test-set. I don't have a large amount of data, only about 360 rows of data in total with 12 features. I'm using Random Forest. The problem is this: I run cross validation and let's say I get an accuracy of 60%, pretty good. Then, I change my training/validation sets by ONE point. So, for example, initially my sets are like this:

  1. Training: rows 1-150
  2. CV Fold 1: rows 151-180
  3. CV Fold 2 rows 181-210 etc.

Now, I shift it by one so that -

  1. Training: rows 1-149
  2. CV Fold 1: rows 150-179
  3. CV Fold 2 rows 180-209 etc.

This seemingly tiny change has a very large impact on my accuracy - it drops from 60%, to something like 55%. Changing this not by one, but by 5 rows might drop it down to like 53%.

First attempt at fixing the issue - I realized I was leaking data because I was using my cross validation set for feature selection, so within my CV sets, I was leaking information from future "folds" to past folds about which features are best. I stopped doing this. The issue persists. What could be causing this?

Second attempt at fix - PLEASE SEE ATTEMPT 5 INSTEAD. I thought maybe this was some odd feature of Random Forest, so I tried all the same things but with SVM and although the variation is to a lesser degree, the accuracy definitely jumps around a few percentage points when I change the boundary even slightly. With SVM, I noticed that the accuracy doesn't ALWAYS decrease - sometimes it increases slightly when I change the boundary, but again, very unstable.

Third attempt at fix - I thought that maybe the boundary I chose initially just happened to be around some very significant/outlier rows of data that have a disproportional impact on the model. So, I changed the initial start/end points of the training/validation set by 30 rows or so, and the issue more or less still persisted.

Fourth attempt at fix - This whole time, I had a sneaking suspicion that simply the low amount of data was the culprit. So, I added 300 more data points, basically doubling the amount of training data, re-ran the cross validation and STILL a single shift in the boundary results in multiple percentage points loss of accuracy. I have no idea what could be causing such instability of performance.

Fifth attempt at fix - Although in my second attempt, I tried changing the method/algorithm, I tried it again in a more detailed analysis and found that the issue seems to be limited to RandomForest. SVM is slightly unstable, but nowhere near the same degree. KNN isn't affected by the shift nearly as much either. 1 point change results in no change in performance, 5 points decreases performance by several percentage points, 10 doesn't make it any worse than 5. Why is RandomForest so extremely sensitive to the training set? It seems like RandomForest is overfitting to a very strong degree, but why would that be (if other methods don't overfit)?

$\endgroup$
10
  • $\begingroup$ In attempt 4, what is the size of the test set? $\endgroup$ Jan 19 at 15:31
  • $\begingroup$ @BenReiniger Initial size of dataset is 360 points, in attempt 4 it was 660, all other attempts, back to 360 rows. $\endgroup$ Jan 19 at 15:34
  • $\begingroup$ Question: how many columns do you have? Are you randomly subsampling, or directly sampling by index? Do you have more than two outputs/labels, that is to say are you accounting for stratification? What is your tree depth? How many minimum leaves per tip? Is your data type a factor, that is to say is your representation consistent with how the software interprets it? How many trees are you using? When you are validating with K nearest neighbors, what is K? Are you estimating the class as the mode of those K neighbors? $\endgroup$ Jan 19 at 15:45
  • $\begingroup$ @EngrStudent As mentioned, I have 12 features/columns. I am not subsampling rows, but I am doing subsampling of columns, the optimal # of columns to sample I arrive at through grid-search cross validation. I only have 2 outputs/labels, binary 0 or 1. My tree depth and minimum leaves per tip are both set at default values. In this R package, I think I have a minimum nodesize (same as leaves per tip?) of 1, with no settings for tree depth (max/min tree depth is only subject to limitation of minimum 1 leaf per tip). Yes, my response variable 0/1 is a factor. Will answer more in next comment. $\endgroup$ Jan 19 at 16:14
  • $\begingroup$ @EngrStudent I am using grid-search cross validation to select the optimal number of trees to be used. Typically, the best result seems to occur around 50, 100 or 200 trees. For k-NN, I am using grid-search cross validation to select the optimal value of K. The best value seems to be k=47. That's correct, in k-NN, I'm just using a majority vote of the nearest 47 neighbors. $\endgroup$ Jan 19 at 16:16
0
$\begingroup$

Ideas:

  • Has the data been shuffled?
  • It could be some kind of off-by-one error in your code, which could be data leakage or the way it's evaluated.
  • Duplicating the data could give you more consistent clues about the nature of the drop in accuracy.
$\endgroup$
2
  • $\begingroup$ Thank you for your suggestions! 1. No, the data has not been shuffled. As mentioned, it's a time-series dataset, and I wanted to preserve the time-series structure. 2. By "off-by-one", do you mean just some kind of indexing error? 3. What do you mean by "duplicating the data"? $\endgroup$ Jan 19 at 14:27
  • $\begingroup$ Please check out my 5th attempt at fix which I just added, I think that pretty much removes the possibility of a data processing error. $\endgroup$ Jan 19 at 15:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.