Those days, I have been reviewing different academic papers which mainly compare the performance of different machine learning methods on a particular problem. And I was surprised by the variety of techniques used as a strategy to optimize the hyperparameters of each model and to compare the different models generated (in other words if I want to measure the predictive power of different models such as random forest, neural network, etc. as I optimize their parameters to maximize the predictive power of each one and then compare the different models to choose the best one).
Specifically, the techniques I have seen used can be classified into 3 groups:
Using nested cross validation: outer loop is used to train and compare the different models and inner loop is used to tune the model hyperparameters (an example of this procedure can be found in )
Using flat cross validation: flat cross validation is used to tune each of the different models and to train them, using the performance metrics obtained from each cross validation to compare the different models (an example of this procedure can be found in ). I assume that in this case in order to correctly compare the models the folds in each Cross Validation should be equal. This procedure also seems to be one of the most commons in internet posts.
Using flat cross validation + hold out set: initially partitioning the data between train and test, using the train with CV to tune and train the models. After this, the different models with the optimized hyperparameters were tested on the test set, selecting the model that outperformed the others.
Also, I found some recent papers that still use a simple hold-out approach. For this reason, I find myself quite confused about what would be an adequate (or the most accepted) procedure to tune and train different models and then select the best one, specially in the academic context.
 Wang, H., Zhou, Z., Li, Y., Chen, Z., Lu, P., Wang, W., ... & Yu, L. (2017). Comparison of machine learning methods for classifying mediastinal lymph node metastasis of non-small cell lung cancer from 18 F-FDG PET/CT images. EJNMMI research, 7(1), 11.
 Li, J., Dai, W., Metze, F., Qu, S., & Das, S. (2017, March). A comparison of deep learning methods for environmental sound detection. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 126-130). IEEE.