1
$\begingroup$

this is my first "real" project and I am not understanding a certain behaviour.

My dataset spans from 2017 up to today. What I did is cleaning data, getting rid of missing values etc. There are mixed features numerical and categorical, I use different pipelines for them and a feature union to bring it all together, so far so good.

I looked further into 2019 data. We are talking of ~400k observations. Target Variable is binary and I approached this with RandomForests first.

20% of the data is Test data, I generate this with train_test_split of sklearn. Here it starts to get a bit strange to me. 5-fold cross-validation on the 80% train data gets me quite good results, the score I chose for now is F1 and its always around 94%.

Also the Test data is around 92-94% F1 score.

However, when I manually let a model train with Jan 2019 to October 2019, the Recall for predicting the Rest of 2019 is going down to 60-70%.

It gets even worse when I train the model with all of 2019 data (or the 80% mentioned earlier) and give it a shot for 2020. I get a very very low recall. I have to say that only a small fraction of the data hits the target variable so it's basically a small group we are trying to find. The precision is quite high as is the accuracy but recall is very jumpy.

My main question is why does the 5-fold cross-validation on 2019 data is bringing F1 and recall > 90% and when I select 10 out of 12 months for training and try to predict the other 2 it is so much worse.

I thought of overfitting but shouldn't the 5-fold cross-validation have picked this up then?

Could anybody point me in the right direction?

edit: I seem to have broken it down in a hopefuly basic issue.

So here is what I don't get.

I have this static set of data of 2019 now.

When I split it with

X_train, X_test, y_train, y_test = train_test_split(features, labels, test_size=0.2)
finalpipe.fit(X_train,y_train.values)
predic = finalpipe.predict(X_test)

I get a very high score no matter what I am looking at. Accuracy, Recall, Precision all >78%

Now I am doing cross_val_score on it like this

scores = cross_val_score(finalpipe, features, labels.values.ravel(),cv=5)

and the score is terrible not close to 78% at all, more like 50-60%

I am aware the split of train_test_split and cv is random, but I repeated this multiple times and no matter how the data looks like I should come to the same results, shouldn't I ? Its both splitting data in 80/20 only that cross_val_score is doing it 5 times and averages but again, when I am looping the first one and average myself its still not anywhere close to what cross_val_score gets me

$\endgroup$
8
  • $\begingroup$ Did you use a time series splitter for your cross validation? If not data leakage across your folds will exist $\endgroup$
    – Jon Nordby
    Commented Apr 29, 2020 at 20:26
  • $\begingroup$ Thanks for your comment. I did it now and yes now I can see the bad behavior in the cross validation already. Unfortunately I cannot really explain this. Taken 80% data to learn randomly works like a charm but sequentially through time series splitter it's a Desaster. Also gridsearchcv with the time series splitter didn't reveal any great hyper parameter either so it seems the features are crap. I still wonder how this is since the features are not dependent on time. No trend whatsoever when I plot the values against time axis.. so really a mystery to me $\endgroup$
    – tuxmania
    Commented May 6, 2020 at 6:56
  • $\begingroup$ Do you have concept drift over time? Or perhaps the model is just plain overfitting. Try limiting the RF performance and see if you can get validation performance in range of test. $\endgroup$
    – Jon Nordby
    Commented May 6, 2020 at 13:47
  • 1
    $\begingroup$ You need to use a time series splitter. Doing it randomly on a time series does not make sense. $\endgroup$
    – Jon Nordby
    Commented May 6, 2020 at 20:40
  • 1
    $\begingroup$ If data change substantially over time (e.g., 2020 data are clearly different from 2019 data), then CV based on 2019 will not estimate well what happens in 2020. Using a time series splitter may help, but only if changes over time within 2019 (or whatever you apply the splitter to) work in the same way as from 2019 to 2020. There may also be seasonal changes. $\endgroup$ Commented Aug 7, 2021 at 14:21

1 Answer 1

1
$\begingroup$

I am answering your questions based on the following assumption:

1: For k-fold cross validation you are only using 2019 data.

Following may be the reason for your mentioned problem:

1: The dataset which you are using of only 10 months may be skewed means one class will be in higher majority than other. But in k-fold, it is randomly suffling the data of one year and then from that data splitting the train and validation set. May be the resulting train set is not that much skewed. If your dataset is skewed then model prediction is more biased towards majority class.

2: In k-fold, split is happening to make sure that ratio of two classs will be same in both train and validation set but this is not the case for your manual testing data.

3: There may be data leakage in the k-fold split but there is not data leakage in your manual train and test split.

4: Data of 10 months may be very less than 80% of the whole data, and rest of the two month test data is very large. This is why it is not properly getting trained. .

$\endgroup$
2
  • $\begingroup$ Thanks for the input, I will test your points and make sure I understand the implications. $\endgroup$
    – tuxmania
    Commented Apr 29, 2020 at 21:24
  • $\begingroup$ I got rid of data leakage by using proper pipelines. Also I do cross validation now with a TimeSeriesSplit instead of kfold which shows the bad but probable real scores already in cross validation. It's just a mystery why it behaves well on randomly selected data but very bad when sequentially learning. No feature is related to time so it shouldn't make any difference $\endgroup$
    – tuxmania
    Commented May 6, 2020 at 7:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.