The shorter and more general version of this question: If tuning a model via cross-validation (within training set) produces worse results on the test set than my previous default/baseline model, do I stick with the tuned model (to avoid over-fitting on the training data) or do I go back to the baseline (even though it seems like I'm overfitting by doing so)?
Long more detailed scenario: Let's say I have an 80/20 train/test split set. Then I build a model with default model hyperparameters and obtain an F1 score of 0.35.
Then I use cross-validation on the train set to identify the best hyperparameters and build a new model on all the training data using those hyperparameters found optimal per the cross-validation. However, when I evaluate this "optimal" tuned model on the test set I get an F1 score of 0.23.
In such cases, should I stick to the default hyperparameters that had produced the higher F1 score on the test set or stick with the tuned model since it was tuned using cross-validation?
In case such variation in numbers is unlikely - I guess then I'm wondering whether there may be some other factor at play, such as too small a dataset (e.g. around total 1000 datapoints total with fewer than 200 of them being in the class of interest) or imbalanced fold partitioning.