Training machine learning models on an unbalanced dataset: about 3% positive labels, and 97% negative. The modeling goal is to get as many examples as possible with 60% precision on a holdout test set per model predictions. (all numbers are hypothetical.)
I've got some general questions on difference between the two approaches:
- A) Train and test ONE model on the entire dataset, and
- B) Train and test MULTIPLE segmentation models on disjoint subsets of the same dataset. These subsets may be based on values of a certain feature, or multiple features.
Q1: Are Approaches A and B essentially the same? I found a segmentation model (trained and tested on a subset of the data based on a certain feature values) that can reach comparable performance to Approach A, per above-mentioned modeling goal, i.e. with 60% precision, the segmentation model has almost as many examples, while the subset is less than 5% of the train set of Approach A. (That is why I started wondering about building more segmentation models on disjoint subsets.)
Q2: If Approach A can not satisfy the modeling goal, what are some sensible next steps, e.g. improve data unbalancedness? improving data quality? try different types of model?