I want to pick the best model among a set of candidate models. Often, I've come across two approaches to do the same:

(1) Split the data into k-folds and iteratively use k-fold CV to arrive at a model with the optimal error metric.

(2) Split the data into 3 folds: training set, on which you train the various models, use the validation set to pick the best model based on which has the optimal error metric and report the error metrics of the test model.

Are these two approaches independent or inter-twined? If they're independent, when would one choose one over the other? Does the size of the dataset determine this? If they're inter-twined, then what exactly is the sequence of steps leading to picking the final model? If (2) is the approach used, then I understand that the final model you would want to use is the one with the parameters selected based on the training data. If (1) is the approach used, what is your final hypothesis? i.e., the set of parameters in your final model? Of which iteration?

Finally, which metric is best for picking a model? I've often seen people reporting a bunch of metrics - AIC/BIC based on complexity, Precision/Recall as the error metric, a McFadden's pseudo R-squared, or a measure of goodness of fit based on likelihood ratio test. What if all of these measures are not consistent with each other? Which model would one pick at the model picking stage if one has, say, more Recall, but also a is a more complex model or something?

I want to pick the best model among a set of candidate logistic regression models. Often, I've come across two approaches to do the same:

(1) Split the data into k-folds and iteratively use k-fold CV to arrive at a model with the optimal error metric.

(2) Split the data into 3 folds: training set, on which you train the various models, use the validation set to pick the best model based on which has the optimal error metric and report the error metrics of the test model.

Are these two approaches independent or inter-twined? If they're independent, when would one choose one over the other? Does the size of the dataset determine this? If they're inter-twined, then what exactly is the sequence of steps leading to picking the final model? If (2) is the approach used, then I understand that the final model you would want to use is the one with the parameters selected based on the training data. If (1) is the approach used, what is your final hypothesis? i.e., the set of parameters in your final model? Of which iteration?

Finally, which metric is best for picking a model? I've often seen people reporting a bunch of metrics - AIC/BIC based on complexity, Precision/Recall as the error metric, a McFadden's pseudo R-squared, or a measure of goodness of fit based on likelihood ratio test. What if all of these measures are not consistent with each other? Which model would one pick at the model picking stage if one has, say, more Recall, but also a is a more complex model or something?