Is there a standard way to make the ratio of prediction classes match the ratio of training classes? I have a dataset with several classes, let's say class 1 and class 2. In the data (which are split into training and testing), 90% are class 1 and 10% are class 2. I want the predictions to come close to the ratio. However, I'm getting something like 99% class 1 and 1% class 2.
I can use sklearn.utils.class_weight.compute_sample_weight or sklearn.utils.class_weight.compute_class_weight to get balanced classes, depending on the classifier. However, my understanding is that this tries to make the predictions 50% class 1 and 50% class 2. I want to make it 90% class 1 and 10% class 2, just like the input data.
I tried doing something like:
weights = num_samples / num_classes / counts_of_each_class
weights += (1 - weights) / x

x is a number greater than 1. The shifts every weight closer to 1.
This produces almost the same effect:
weights = sum(counts_of_each_class ** y) / num_classes / counts_of_each_class ** y

y is a number between 0 and 1. This also shifts every weight closer to 1.
Empirically, x = 8 or y = 0.95 works for me, but this seems like a very hacky solution. Is there a standard way to adjust the weights so the ratio of the prediction classes match the ratio of the classes in the input data?
 A: This is not an easy problem and in fact, many of the commonly used techniques (oversampling, undersampling, class-weighting) are hacky and are often the wrong approach which might help only in the short-term.
I would generally advise 4 things:
1) Don't focus on accuracy, but consider other metrics which are better suited for imbalanced data. Specifically: precision and recall. Problem with accuracy is that in case of high imbalance, even trivial models can achieve high accuracy.
2) Thing about the loss function. Maybe one of the classes is more important to you than the other one. If so, this can be encoded in the loss function or sample weights.
3) Introduce more information to the model (new features, more data) or allow more complex models. Strong models with adequate loss functions can usually learn the imbalance by themselves.
4) I would advise to stay away from undersampling. You are basically giving the model less information to work with, which is bad. It might help in short term, but it is not a good long term solution.
A: In Python you can use imbalanced-learn. It contains both undersampling and oversampling strategies, and even some more elaborate techniques that generate new samples. For all these methods you can specify ratios for the classes. See this user guide.
