When is an unbalanced dataset large enough for calculating a decision threshold? I have a (large i.e. >1M rows) very unbalanced (1% event label, binary classification) dataset with data from various institutions. At the moment, I train an XGBoost model on this data and get good performance on average (in production) across all institutions. The decision threshold is calculated on the test dataset to give a model with a defined target level of precision (we care about false positives).
This means, however, that for some institutions the performance is subpar and being more or less strict i.e. having a lower or higher decision threshold would be beneficial. I understand I'm walking into overfitting territory, but at the same time the individual institutions complain they're not getting good performance.
So my question is: how would you determine if a dataset (in this case, the data relative to a certain institution) is large and diversified enough (enough rows, but also enough event labels) to undertake that threshold optimisation (the model is the same, I'm only asking about the decision threshold)?
I understand the answer is usually "depends on the data", but I don't have a heuristic to determine what factors to look into.
 A: This is fun and not a trivial problem, I would attack as follows: 1. Find formulas where the variance of the confidence intervals of the statistic of interest is a function of $n$, the sample size available. 2. Despite this being a binary classification task, I would treat it as being multi-class during the evaluation. That is because while the "micro-average" is good (i.e. when aggregating all institution samples together), we want assurances about this estimator's performance at a macro-level (i.e. when each institution's performance is (almost) equally important).
Assuming that we care for the $F_1$ score, for example, we can use the work in:  Confidence interval for micro-averaged $F_1$ and macro-averaged $F_1$ scores  (2021) by Takahashi et al., to look at the formulas for computing micro- and macro-averaged $F_1$ confidence intervals. For example, the micro-averaged
$F_1$ score's variance is given as $\text{Var}(\hat{miF_1}) = \frac{(\sum_{i=1}^r p_{ii})(1-\sum_{i=1}^r p_{ii})}{n}$. Here $r$ is the number of classes, and $p_{ii}$ is cell probability in the confusion matrix for class $i$; this is Eq. 4 in the paper; a similar (but longer) formula exists for the macro-averaged $F_1$ score's variance. With this in mind, we can define a target variance for our metric and solve for $n$. (Again, do note that what the institutions are complaining about relates much closer to the macro-average. Also in case, it is missed, Takahashi et al. have R code in its appendix. It goes without saying that the same rationale can be applied if want to use other performance metrics, for example, if we are interested in AUC-ROC we would use something akin to: Variance estimation for two-class and multi-class ROC analysis using operating point averaging (2008) by Paclik et al.)
To elucidate some points on the "multiple institution" caveat: We pick a multi-class evaluation metric because we treat each institution as a class on its own. To important points: 1. the confusion matrix has zero entries in cells $p_{ij}$ when those entries refer to misclassifying samples from one institution to another (as we can't misclassify a sample to another institution anyway). The only non-zero non-diagonal entries will be at the margin of the "Negative" label. 2. the binary to multiclass change is only relevant when computing our performance metrics; during training, we still use a binary classifier as we want our classifier to learn the same representation for all positive samples irrespective of their institutional membership. To that extent, we might want to use the institution as a label and ensure it has very low explanatory power.
The above being said I suspect that some assumptions might be a bit over-optimistic about the asymptotic behaviour in imbalanced settings. (e.g. I have seen a lot of normal approximations being used in the Takahashi paper) With that in mind, as a first step, I would aim to estimate the variance estimator of the underlying multi-class estimates, communicate those estimates to each respective institution, and then work backwards. (i.e. given this variance and this sample size how much bigger the sample would need to be to have half the variance, etc.) In that regard, the variance estimates from institution $A$ having $2N$ samples could be used to also give an idea to institution $B$ having "only" $N$ sample how much "better" estimates they would get if they doubled their throughput. (Tantithamthavorn et al. (2016) An Empirical Comparison of Model Validation Techniques for Defect Prediction Models is a nice recent view  on the matter.)
A: 1% of $10^6$ is 10,000, which is a pretty good sample size. You could be having problems because 10,000 samples isn't enough to fill the feature space you're dealing with. If I had to guess, though, I'd say that your problem isn't the imbalance, it's that the imbalance varies from institution to institution. If you're wedded to XGBoost, or similar, I would recommend adding the institution to the list of labels you train on, and see if that helps.
If you're allowed to change model, though, I would recommend training a model that outputs probabilities (i.e. trained via logistic regression). Then you can use the balance of the overall set as a Bayesian prior probability to remove from the prediction and replace with a prior reflective of the individual institution.
You can then do a ROC analysis to set decision boundary, do some form of expected value calculation, or do game theory if multiple people are involved.
If the 2008 financial crash taught us anything, it's that the data can have unexpected correlations. That is, there may be some important time-series information in here. For instance, the classes of the previous $k$ samples may have relevant information about whether the next sample is of the rare class (examples: a machine on the line broke, so the usually rare defective part is now common; or a good experience by one person who falls in the rare class means they've spread good word-of-mouth among their friends who are more likely than average to be from the rare class, etc).
