I have a database of data that is labelled good. I can make new data myself and I'll label it bad (I know how to transform a bad data from a good data).
I want my model to determine if a piece of data is good (not modified) or bad (modified), I wanted to do 50/50 in my training dataset but I know that in real life, bad data only occurs like around 0.1% of the time.
Should I reproduce this in my training dataset? Why? Any intuition about why?
Because I've read that using 0.01/99.9 would lead to the algorithm always returning the class of the majority label.
Most models, such as logistic regressions, neural networks, and random forests, give predictions on a continuum, rather than making hard classifications. The way software packages get a categorical prediction is by comparing that continuous prediction to some kind of threshold and deciding on the category by seeing the side of the threshold on which the prediction falls (above the prediction goes to the positive category, below to the negative). These continuous predictions often lie in the interval $[0,1]$ and can have an interpretation as probability, so the typical threshold is $0.5$.
In an unbalanced problem, it might be that the probability of being in the minority class never exceeds $0.5$, meaning that correct performance might be to having every prediction fall below this standard threshold. Consequently, some suggest to artificially balance the categories so the predictions can be higher. After all, a higher prior probability (the proportion of observations that belong to the minority class) leads to a higher posterior probability by Bayes' theorem, all else equal. Below, $P(Y=1)$ is taken as the prior probability.
$$ P(Y = 1\vert X = x) = \dfrac{ P(X = x\vert Y = 1)P(Y = 1) }{ P(X = x) } $$
Not all else is equal, but this is the idea behind artificial balancing (e.g., ROSE, SMOTE), and the results show that artificial balancing indeed increases predicted probabilities.
Consequently, this artificial balancing can be expected to be successful in making the model make higher predictions that exceed the $0.5$ threshold more often. However, your predictions are now, in some sense, not telling the truth. A prediction of $0.8$ does not mean the event is expected to happen with probability $0.8$. Consequently, the rich information present in probabilistic predictions from your model no longer tell the truth. While you might be able to calibrate these predictions to reflect the reality of how often events occur, fiddling with the class ratio and then calibrating (maybe) just to make sure uncalibrated predictions fall above an arbitrary threshold seems inefficient at best (there could be edge cases where this winds up working better, sure) and damaging to the education of aspiring predictive modelers at worst by eliminating an emphasis on the valuable information available in the original, non-categorical predictions.
If you start out with the natural class ratio, you do not start the modeling by misleading the model about the prior probability. Consequently, most "fixing" of class imbalance is unnecessary (though this question has an answer with an interesting edge case) and due to an unnecessary obsession with making sure the predictions are on the correct side of $0.5$ when it is fine to use a different threshold (the prior probability seems like a natural one) or consider the direct predictions without considering any threshold at all.
(I know, I know: sklearn uses predict to return categorical predictions. I believe that sklearn, for all of its positives (and I have used it to do good work at my job), has caused damage to the field of statistics by using the predict method to return the category with the highest probability and the predict_proba method to return the raw probabilities, rather than having predict return the probabilities and something like predict_category to return the category with the highest probability.)
I will close with some standard links about class imbalance and evaluating classifiers.
Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?
Profusion of threads on imbalanced data - can we merge/deem canonical any?
Why is accuracy not the best measure for assessing classification models?
Academic reference on the drawbacks of accuracy, F1 score, sensitivity and/or specificity
