Reliably evaluate model performance with very few positive samples

Question

I do a binary classification in the domain of predictive maintenance.

Setup

• My dataset is highly imbalanced with only 17 samples of the positive class, but an nearly indefinite amount of negative ones.
• My target variable is precision of class 1, as I would like to correctly classify as many "1" cases as possible, but a false positive would be bad and related to unecessary maintenance costs.
• As the data comes from six different vehicles, I use a train-test-split which is grouped per vehicle to prevent the model from learning vehicle-specific characteristic.
• In my grid search to find the best model I implement some measures to counter this class imbalance, like oversampling, class weights and optimizing the classification threshold (by using sklearn's .predict_proba() and evaluating the model performance on different thresholds).

Question 1: How to obtain a reliable estimate of the model's predictive power while still using as much information about class 1 as possible during training?

• So far, I use LeavePGroupsOut cross-validation with P set to 2 to tune hyperparameters
• As can be expected, the results of the cross-validation runs are not very stable with this few positive samples in cv training and cv validation sets (they fluctuate quite a bit)
• I fear that evaluating my model against the test set might not deliver a good estimate of the model's performance due to the test set having not many positive cases (thus, there is probably a lot of random variation coming into play).
• Would for example a nested cross-validation contribute to make my model's performance estimate more reliable?

Question 2: Which other methods could make my model better, taking into account the very small number of positive cases? I already implemented a semi-supervised Mahalanobis classifier which fits on class "0" only and then predicts on the whole X_train, and I use this classifier's prediction as an additional feature.

Answer 1

Question 1: in your case, since I assume that you don't have more historical positive samples I would say you can either create some synthetic positive samples with something like Smote for example (although this is more like an experiment rather than a definite answer) and use those samples to either train or test your model on. There is no harm in trying how your model performs against only negative samples specially if what you are interested to find out is how good/bad it is with false positives.

Question 2: I would say that since you are trying to model a time-to-event problem you can try a survival analysis approach which specialises in very small number of positive samples and it takes time into account