this is my first "real" project and I am not understanding a certain behaviour.
My dataset spans from 2017 up to today. What I did is cleaning data, getting rid of missing values etc. There are mixed features numerical and categorical, I use different pipelines for them and a feature union to bring it all together, so far so good.
I looked further into 2019 data. We are talking of ~400k observations. Target Variable is binary and I approached this with RandomForests first.
20% of the data is Test data, I generate this with train_test_split of sklearn. Here it starts to get a bit strange to me. 5-fold cross-validation on the 80% train data gets me quite good results, the score I chose for now is F1 and its always around 94%.
Also the Test data is around 92-94% F1 score.
However, when I manually let a model train with Jan 2019 to October 2019, the Recall for predicting the Rest of 2019 is going down to 60-70%.
It gets even worse when I train the model with all of 2019 data (or the 80% mentioned earlier) and give it a shot for 2020. I get a very very low recall. I have to say that only a small fraction of the data hits the target variable so it's basically a small group we are trying to find. The precision is quite high as is the accuracy but recall is very jumpy.
My main question is why does the 5-fold cross-validation on 2019 data is bringing F1 and recall > 90% and when I select 10 out of 12 months for training and try to predict the other 2 it is so much worse.
I thought of overfitting but shouldn't the 5-fold cross-validation have picked this up then?
Could anybody point me in the right direction?
edit: I seem to have broken it down in a hopefuly basic issue.
So here is what I don't get.
I have this static set of data of 2019 now.
When I split it with
X_train, X_test, y_train, y_test = train_test_split(features, labels, test_size=0.2)
finalpipe.fit(X_train,y_train.values)
predic = finalpipe.predict(X_test)
I get a very high score no matter what I am looking at. Accuracy, Recall, Precision all >78%
Now I am doing cross_val_score on it like this
scores = cross_val_score(finalpipe, features, labels.values.ravel(),cv=5)
and the score is terrible not close to 78% at all, more like 50-60%
I am aware the split of train_test_split and cv is random, but I repeated this multiple times and no matter how the data looks like I should come to the same results, shouldn't I ? Its both splitting data in 80/20 only that cross_val_score is doing it 5 times and averages but again, when I am looping the first one and average myself its still not anywhere close to what cross_val_score gets me