Training machine learning models on an unbalanced dataset: about 3% positive labels, and 97% negative. The modeling goal is to get as many examples as possible with 60% precision on a holdout test set per model predictions. (all numbers are hypothetical.)

I've got some general questions on difference between the two approaches:

  • A) Train and test ONE model on the entire dataset, and
  • B) Train and test MULTIPLE segmentation models on disjoint subsets of the same dataset. These subsets may be based on values of a certain feature, or multiple features.

Q1: Are Approaches A and B essentially the same? I found a segmentation model (trained and tested on a subset of the data based on a certain feature values) that can reach comparable performance to Approach A, per above-mentioned modeling goal, i.e. with 60% precision, the segmentation model has almost as many examples, while the subset is less than 5% of the train set of Approach A. (That is why I started wondering about building more segmentation models on disjoint subsets.)

Q2: If Approach A can not satisfy the modeling goal, what are some sensible next steps, e.g. improve data unbalancedness? improving data quality? try different types of model?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.