Questions reg. data partitioning, error metrics and model selection

Question

I want to pick the best model among a set of candidate logistic regression models. Often, I've come across two approaches to do the same:

(1) Split the data into k-folds and iteratively use k-fold CV to arrive at a model with the optimal error metric.

(2) Split the data into 3 folds: training set, on which you train the various models, use the validation set to pick the best model based on which has the optimal error metric and report the error metrics of the test model.

Are these two approaches independent or inter-twined? If they're independent, when would one choose one over the other? Does the size of the dataset determine this? If they're inter-twined, then what exactly is the sequence of steps leading to picking the final model? If (2) is the approach used, then I understand that the final model you would want to use is the one with the parameters selected based on the training data. If (1) is the approach used, what is your final hypothesis? i.e., the set of parameters in your final model? Of which iteration?

Finally, which metric is best for picking a model? I've often seen people reporting a bunch of metrics - AIC/BIC based on complexity, Precision/Recall as the error metric, a McFadden's pseudo R-squared, or a measure of goodness of fit based on likelihood ratio test. What if all of these measures are not consistent with each other? Which model would one pick at the model picking stage if one has, say, more Recall, but also a is a more complex model or something?

Answer 1

You probably want to use ideas from both approaches together:

• cross validation is more efficient than a single split in estimating performance. You don't need it if you have enough cases available. Enough may be in the range of several thousands for a single estimate up to hundred thousands for comparing multiple models as in selecting the best.

• If you do data-driven model selection, you need to measure the performance of the finally selected model with a test set that is independent of the whole training (incl. data-driven model selection). In the second approach that is the test set. With cross validation it would be the outermost loop of a nested cross validation.

---

Which metric to use depends on your application. However, for classification you typically want to look at at least two corresponding measures such as sensitivity/specificicity or positive and negative predictive values, because these pairs are trade-offs (by selecting a working point or cut-off for e.g. the predicted probability): you can get arbitrarily good sensitivity at the cost of specificity and vice versa.

However, for data-driven optimization, proper scoring rules are the way to go - whatever other figures of merit you look at for the final validation.