I'm building a classifier to predict a binary label on a dataset with 30 features and around 60000 samples of measurements from a car assembly process.

While experimenting with some baseline models without any feature selection or engineering, hyperparameter tuning or anything really, just using all features, I'm getting CV acc scores of around 85%.

Problem is when I try to predict the labels using the holdout set, if I built this set by randomly sampling from the original dataset (and not using these samples for training) I get similar results, but if the holdout set was built using for example the most recent 100 samples from production, or by selecting (and removing) for instance row 45000-45099 from the original dataset, the model predicts all labels as 1.

If anybody could shed some light as to why this happens I'd be incredibly grateful. Thanks!

Edit: To clarify, bad performance happens not only for new values, but also if the holdout is taken from the middle of the existing dataset for example instead of randomly sampling it.

  • $\begingroup$ It seems pretty obvious that the most recent data is significantly different than the other data doesn’t it? $\endgroup$ – astel Mar 13 at 23:34
  • $\begingroup$ I considered it and that's why I also tried to select an holdout set from 100 values for example from the middle of the existing dataset (ensuring they weren't used for training), and the results are as poor as when the model tries to predict on new data from production. However creating the holdout by randomly sampling from that dataset seems to yield results in line with those from CV (around 85% prediction accuracy), hence my confusion. $\endgroup$ – Chineru91 Mar 14 at 0:10
  • $\begingroup$ Right, so it appears that your data is ordered by some measurement of time (though I really have no idea since I can't see your data). So the middle behaves differently from the beginning which behaves differently from the end... $\endgroup$ – astel Mar 14 at 14:00
  • $\begingroup$ Indeed, that would appear to be the case and I will check the distribution of the data over time, thanks. I do have timestamps associated with when the data was collected, however, the order of the cars can change during the assembly process, so sequential measurements aren't necessarily correlated. The goal is only to check if deviations in certain measurements will result in a problem being detected downstream. $\endgroup$ – Chineru91 Mar 14 at 14:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.