Background: I'm facing a 1 : 40 000 class imbalanced problem. It's a binary classification problem with positive class around ~500-700 instances and negative class in the tens of millions instances.
I have read several other questions, references, etc.:
Binary classification with strongly unbalanced classes
https://machinelearningmastery.com/tour-of-evaluation-metrics-for-imbalanced-classification/
What is the root cause of the class imbalance problem?
and others etc...
Several things I have tried based on those references (and many more):
Under/over-sampling: I undersampled the data to 10:1 and the performance improved from 0.1% precision to ~1-5% (for highly imbalanced problem, that's quite good in terms of absolute numbers, especially since my data is in the tens of millions). Recall went up from 0.1% to ~5-10% as well.
Tweaking with several hyperparameters generally adjusted for class imbalances: In most learning algorithms (XGB, CatBoost, LGBM), adjusting scale_pos_weight, min_child_weight, max_delta_step were especially "great" (however, result is still as above).
Tweaking with objective function: Using cost-sensitive functions such as logloss and AUC
Feature engineering and selection: 33 base/raw features, engineered to 142. From top 10 features, only 2 of the base features have high SHAP values (see: https://shap.readthedocs.io/en/latest/example_notebooks/api_examples/plots/bar.html).
Further feature selection: I also used VarianceThreshold and tree-based feature selection based on scikit-learn's guide (https://scikit-learn.org/1.5/modules/feature_selection.html)
Stacking: I built hundreds of base learners in XGBoost, each trained with different hyperparameters (basically, using numpy.random to tweak these around some sensible values), and with different sets of negative classes (i.e. undersampling the majority class with different seeds on each training round), and with different sets of training features (i.e. selecting a random number of features out of my 142 features, although I always keep top 8 features based on SHAP). I then averages out each of these learner's predict_proba, and set a final probability threshold at 99.9%.
Now, the best model (after I combine ALL the methods above) performs at 20-30% recall with 5-10% precision. The target is 50% recall with 50-90% precision.
The next thing I want to try is the following:
Using the hundreds of learners I trained from point 6 (keeping in mind points 1-5 while training each base learners), I'll apply model.predict_proba on my training set.
And then, I'll use that value of predict_proba (from hundreds of base learners) as features to train a final model, on the same training set. The idea is, instead of averaging all the predict_proba of the base learners (same weightage), I'll let a final model decide the weightage of each base learner.
I would like to ask around, if the above idea makes sense?
Will there be data leakage concerns? (because I trained each base learner on my train_set. And then asked it to predict the same train_set. And then use this output to train a final model, also on the train_set).
Do you think, it would also makes sense if I train, say, 100 CatBoost, 100 XGBoost, and 100 LGBM, and then train a neural network based on the total 300 predict_proba outputs? I have read that some model-stacking architectures did this, but I'm not sure if that is a widely-used practice?
Thanks!
