How come model prediction accuracy high but model does not generalise well I have trained a couple of models which I'm experimenting with. One is Logistic Regression and the other Random Forest. I've got 10s of 1000s of samples in my dataset (which has 4 features) and I've experimented with how many samples throw up the best out-of-sample test accuracy. I have done 10 k-fold validation, and some gridsearch optimisation of hyperparameters ... and I'm consistently(**) getting about 82% accuracy predicting on test data. I am splitting my dataset 70:30, training on the 70% and then testing on the unseen 30%. Both models give me roughly 82% accuracy predicting on test data. I was thinking this was a good result and because k-fold validation is giving me a nice accuracy, I am not overfitting or underfitting. But, I must be ...
... when I try predicting on new data samples captured very soon after I train the model ... I am getting nowhere near 82% accuracy. In fact, I'm getting less than 40% success rate when I compare my model prediction with what outcome actually transpires.
So I guess my model does not generalise well. Where can I go from here? I would like to first of all confirm what the problem is exactly. Is the 82% accuracy misleading? How can my live results be so much worse? Could it be that the 4 features are simply not good enough? In which case how can I get 82% accuracy in testing? Are there tests that I can do on the model(s) to gain insights for further work?
(** I retrain the model quite often as new data comes in realtime)
 A: It's hard to say without digging deeply into your model and your data. However, it seems like you have been doing a lot of cross-validation, model tuning, cross-validation, model tuning and so forth. That, together with bad out-of-sample performance, suggests that you are overfitting to your test set. That is harder than overfitting in-sample (which is easy indeed), but it is quite possible to do. Essentially, if this is the problem, then your repeated model tuning cycles simply fitted it to the idiosyncrasies of the full dataset.
As to what to do now: you should dig into your data. Did anything change drastically between the training and the testing data? Are there any strong predictors in the new data which did not show up as strongly in the training data? Stuff like that. But remember that the more you tweak your model, the more likely you are to overfit, so proceed with caution.
Incidentally, you should be able to get at least 50% accuracy by always predicting the majority class in your holdout dataset, assuming you can identify this class beforehand. Thus, an accuracy of only 40% is a big red flag. It looks like something has changed in a major way. (Also, this simple benchmark is one reason why accuracy is not a good evaluation measure.)
A: I second Stephan's answer that the likely culprit is overfitting the entire dataset. That said, another thing to validate is that there are no differences between data processing pipelines in your training vs. production code. E.g. are you normalizing the features before training? If so, do you record the means and standard deviations and apply the same normalization to live data?
A: To me it seems to be a data problem. You are splitting the data 70:30, but are all data from the 70% prior to data from the 30% set?
It can be a problem if you mix older and newer data in the training set. If time is involved in the generation of data, which seems to be the case as you have live data, test set should never contain data that were generated before those of the training set.
