I'm not a statistician, so please excuse a possibly wrong use of terminology here.
My dataset has about 400 - 800 samples and about 800 features. The samples are ordered by time, although it is not a time series. There are three class labels (exclusive), which are not highly imbalanced, but slightly imbalanced.
I have a custom metric to check how well a model performs, which is not usable for training, but can be used to evaluate it afterwards (very computationally expensive).
I'm using a training set, a validation set, and a test set. The test set is fixed (last x recent samples), but depending on how i choose the training and validation sets, e.g., for a MLPClassifier, the outcome on my custom metric varies greatly.
To deal with this, i build a few hundred models, each with a differently shuffled training/validation set. Then i compare them on my custom metric with the fixed test set. Among the models, i can then select the best performing one.
The disadvantage is that i have to train a few hundred models every time a new number of samples are available and then select the best one for the next prediction period.
Is the overall approach sound? Is there a better approach?
I read things like stochastic weight averaging might help, but as far as i understand this only helps to reduce the randomness introduced by the different starting weights of a neural network, not with variance introduced by having different training/validation sets.