(In the context of predictive models) Caveat: I'm aware that imbalanced data questions are a dead horse, but I haven't found an answer to this flavor of it directly.
When working with highly imbalanced data (e.g. binary class cases), the common wisdom is to try training on an upsampling of the minority class, downsampling of the majority class, some combination of the two, or something like SMOTE. The potential gains of doing so are obvious if one accepts default model thresholds and tunes for matrix-based metrics like accuracy or f1. However...
It is my experience that if one instead optimizes for a threshold-invariant metric first (like ROC AUC or PR AUC or similar) and then adjusts the threshold to meet the specific business need, resampling loses its benefit swiftly and often hurts more than helps.
Is the common wisdom of balancing class data (or its various flavors) then folklore given the above strategy? If not, in what cases should one generally expect it to help, and in what cases hurt?
