# Reliably evaluate model performance with very few positive samples

I do a binary classification in the domain of predictive maintenance.

Setup

• My dataset is highly imbalanced with only 17 samples of the positive class, but an nearly indefinite amount of negative ones.
• My target variable is precision of class 1, as I would like to correctly classify as many "1" cases as possible, but a false positive would be bad and related to unecessary maintenance costs.
• As the data comes from six different vehicles, I use a train-test-split which is grouped per vehicle to prevent the model from learning vehicle-specific characteristic.
• In my grid search to find the best model I implement some measures to counter this class imbalance, like oversampling, class weights and optimizing the classification threshold (by using sklearn's .predict_proba() and evaluating the model performance on different thresholds).

Question 1: How to obtain a reliable estimate of the model's predictive power while still using as much information about class 1 as possible during training?

• So far, I use LeavePGroupsOut cross-validation with P set to 2 to tune hyperparameters
• As can be expected, the results of the cross-validation runs are not very stable with this few positive samples in cv training and cv validation sets (they fluctuate quite a bit)
• I fear that evaluating my model against the test set might not deliver a good estimate of the model's performance due to the test set having not many positive cases (thus, there is probably a lot of random variation coming into play).
• Would for example a nested cross-validation contribute to make my model's performance estimate more reliable?

Question 2: Which other methods could make my model better, taking into account the very small number of positive cases? I already implemented a semi-supervised Mahalanobis classifier which fits on class "0" only and then predicts on the whole X_train, and I use this classifier's prediction as an additional feature.