The validation set includes few positive labels I'm training a classifer on an unbalanced dataset. The test dataset's positive proportion is 0.02%.
For that reason, the validation data set labels proportions are the same. Because the validation set size is much smaller than the test dataset, it contains less than ten positive labels. The test set includes 25 positive labels. I tune the model hyperparameters by using the F-beta score.
I'm not sure that a sample with less than ten positive labels, is a valid sample for tuning and evaluating the classifer. Indeed, the classifer has terrible results when applied to the validation and test sets.
Since the training set is more balanced from the validation and test sets, I can move positive labels from the train set to the validation set (and test set). However, in that way, they will not represent the real data.
What do you recommend me to do?
 A: You’ve split the data wrong. Since class imbalance is unlikely to be a problem for your work, even if it appears to be when you use improper scoring rules, there is no need to fiddle with the data. Just split the data, perhaps stratifying to ensure the exact same ratio in both the training-sample and out-of-sample sets.
When you do this, you do not deplete your minority-class samples by artificially balancing the training data, leaving you with plenty of samples for an out-of-sample assessment (especially if you have a lot of data like you have posted is the case for your work).
An alternative to splitting the data is a bootstrap approach. Not everyone agrees with this, with an interesting debate here, and my take on it is that I am torn. However, it is worth knowing that such an approach to validation does exist.
A: If the training data is more balanced, why do you not consider adding more positive samples to the validation and test set from the training set? If training data is also not balanced, maybe using representation learning first to learn discriminative features better and then taking these features to apply classification can be an option.
A: Having a balanced training set is important as the model may otherwise learn to be biased towards the negative class. If it is not possible to have a balanced training set then I suggest you look into adjusting the weight each class has on the error criteria. This can be done in sklearn by using the class_weight model parameter. In turn, giving the model a much larger penalty during training for misclassifying positive examples to make up for the lack of positive examples in the training set.
As for the validation set, 10 positive examples is quite small. Try to increase that to 20 if it's possible. However, having an unbalanced validation and test set is not as much of an issue. You can look at the confusion matrix or other metrics derived from the confusion matrix to validate the performance of the model - https://towardsdatascience.com/performance-metrics-confusion-matrix-precision-recall-and-f1-score-a8fe076a2262.
Furthermore, have you looked into augmenting the data?
