What are some of the most correct/accepted ways to tune and compare different models in an academic context?

Question

Those days, I have been reviewing different academic papers which mainly compare the performance of different machine learning methods on a particular problem. And I was surprised by the variety of techniques used as a strategy to optimize the hyperparameters of each model and to compare the different models generated (in other words if I want to measure the predictive power of different models such as random forest, neural network, etc. as I optimize their parameters to maximize the predictive power of each one and then compare the different models to choose the best one).

Specifically, the techniques I have seen used can be classified into 3 groups:

1. Using nested cross validation: outer loop is used to train and compare the different models and inner loop is used to tune the model hyperparameters (an example of this procedure can be found in [1])

2. Using flat cross validation: flat cross validation is used to tune each of the different models and to train them, using the performance metrics obtained from each cross validation to compare the different models (an example of this procedure can be found in [2]). I assume that in this case in order to correctly compare the models the folds in each Cross Validation should be equal. This procedure also seems to be one of the most commons in internet posts.

3. Using flat cross validation + hold out set: initially partitioning the data between train and test, using the train with CV to tune and train the models. After this, the different models with the optimized hyperparameters were tested on the test set, selecting the model that outperformed the others.

Also, I found some recent papers that still use a simple hold-out approach. For this reason, I find myself quite confused about what would be an adequate (or the most accepted) procedure to tune and train different models and then select the best one, specially in the academic context.

Thanks!

[1] Wang, H., Zhou, Z., Li, Y., Chen, Z., Lu, P., Wang, W., ... & Yu, L. (2017). Comparison of machine learning methods for classifying mediastinal lymph node metastasis of non-small cell lung cancer from 18 F-FDG PET/CT images. EJNMMI research, 7(1), 11.

[2] Li, J., Dai, W., Metze, F., Qu, S., & Das, S. (2017, March). A comparison of deep learning methods for environmental sound detection. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 126-130). IEEE.

Answer 1

Welcome to this community, Fran.

From what I see there's no universal or constant consensus on what's adequate, nor can we really speak of most accepted (and as scientists we should always be wary, even if something is "most accepted" – the Ptolemaic system was most accepted for centuries). Some of the techniques you mention are widely used by a school or group of authors and criticized by another (often unawares of one another). And their use and popularity seem to be fluctuating with time.

Also, the techniques you mention are not the only ones. For example there's the Bayesian technique that uses the evidence (technical sense). Take a look at MacKay's beautiful article Bayesian interpolation (see also http://www.inference.org.uk/mackay/PhD.html), where he explains and compares the "evidence" and other cross-validation-like techniques.