I have a heavily imbalanced binary classification task, where the prevalence of the negative event is around 3%. I shuffled then stratified split my data in train, test and validation, ensuring that I have the same percentage of samples from the negative class in each set. Since some negative samples are repeated (higher importance), I further ensured that the copies all lie either in training, or in validation or test (in order to avoid sample memorization). I resampled the training data using the SmoteENN technique, leaving validation and test unbalanced. For accuracy tracking, I use metrics which account for the prevalence of the negative event.
My problem is that I have perfect metric values for the training data, with a significant gap to the metric values on the validation and test data, which are very similar. The hyperparameter optimization tracks the model performance on the validation data.
First thought - overfitting. I tried using different resampling techniques on the training data, using cross validation and also using heavy regularization. The result is the same - even when the training metric values drop below perfection, I see the same performance drop in validation and test. The gap is however still present. I tried using different models, ranging from SVM to neural networks. The gap is present everywhere, but the more complex models still offer better metric values.
Second thought - different train/test/validation distributions. This should be covered by the initial shuffling of the data, before the stratified split. Since I however only have a limited amount of data (~500 of the negative events against thousands of positive events), can it be that this is really the explanation?
Are there any other explanations for the observed metrics? Is there anything I can do except gathering more data (not a possibility currently)?