Are there any mathematical features that an evaluation metric must have? I'm trying to optimize the hyperparameters of my model using the Bayesian approach with the hyperopt library. I have to code a loss to evaluate each iteration of the optimization, and the classic metrics are usually chosen, like
loss = 1 - accuracy

Now, since I want to consider both a good model on test data and a not-overfitted model, I came to define the loss as this
train_loss = 1 - train_f1_score
test_loss = 1 - test_f1_score
loss = test_loss * 10^{test_loss - train_loss}

where the test f1 is calculated based on the mean f1 on a 3-fold cross-validation. The idea is that the metric becomes higher with an overfitted model, even if the test score is good.
I have a doubt: am I missing some particular feature that a good evaluation metric needs to have?
 A: Accuracy is a misleading KPI for predictive performance. Note that every criticism raised in that thread against accuracy applies equally to the $F1$ (or more generally, any $F\beta$) score.
As Sycorax comments, use proper scoring rules on probabilistic predictions. That is also my recommendation in the linked thread.
Contrary to user2672299, I'm all for cross-validation. (Note that you can of course work with probabilistic predictions and scoring rules in cross-validation.) I would just recommend that you keep a validation sample that you evaluate your final model on, because, as user2672299 notes, you can overfit "to a cross-validation".
As to particular features a good metric should have: it should reward calibrated and sharp probabilistic predictions. Accuracy does not. The tag wiki for our scoring-rules tag contains pointers to literature explaining this point and why it makes sense.
A: What you are missing is the purpose of the evaluation metric and the "test" set (actually it is validation set). You are not allowed to use the "test" loss in your loss function, because than your "test" set is not an independent sample anymore.
If you use your "test_loss" in the training step your "test" error is confounded  (i.e. worthless).
Therefore, what you are doing is using "test" data as training data.
