I'm building a classifier to predict a binary label on a dataset with 30 features and around 60000 samples of measurements from a car assembly process.
While experimenting with some baseline models without any feature selection or engineering, hyperparameter tuning or anything really, just using all features, I'm getting CV acc scores of around 85%.
Problem is when I try to predict the labels using the holdout set, if I built this set by randomly sampling from the original dataset (and not using these samples for training) I get similar results, but if the holdout set was built using for example the most recent 100 samples from production, or by selecting (and removing) for instance row 45000-45099 from the original dataset, the model predicts all labels as 1.
If anybody could shed some light as to why this happens I'd be incredibly grateful. Thanks!
Edit: To clarify, bad performance happens not only for new values, but also if the holdout is taken from the middle of the existing dataset for example instead of randomly sampling it.