Generalization of model performance (AUC) and tuning of a catboost classifier I was wondering if it is good practice to overfit on the training data while tuning a catboost classifier for a binary outcome.
Wouldn't it be better to reguralize until validation error equals training error, espescially if the test set has a different distribution of the predictors?
In general how should I compare the train/val/test AUC scores when training (and tuning) a catboost model for binary classification? Is there a tuning strategy that will ensure the best generalization error (other than having i.i.d. splits)?
 A: I would suggest to avoid over-fitting your training data and focus on your (cross-) validation error. Our training error is not entirely useless but it is very prone to mislead us (i.e. if two classifiers have qualitative similar validation errors, pick the classifier with the lower training error but don't put too much faith on that choice between those two). Ultimately we want our classifier to perform well to new unseen data so training set performance by itself is not a strong indicator.
You mention that the test-set in question "has a different distribution of the predictors", this is potentially problematic because it suggests the occurrence of covariate shift, the input variables distribution changes. That is almost always detrimental to our learner performance and if anything it suggest we should focus even more in a robust validation procedure (e.g. repeated K-fold) instead of trying to lower our training error by over-fitting the training set. A good first reference on the subject is: A unifying view on dataset shift in classification (2012) by Moreno-Torres et al. if you want to probe this a bit further.
Regarding overall strategy: Unless there is a situation where there is an abundance of data, I would strongly suggest using cross-validation instead of a single validation set, so you can estimate the variability in the validation performance as well as have a more realistic view of the expected performance. To that regard, even if "a lot of data" is available, I would say, try a 2-fold CV just to ensure that on both partitions provide qualitative similar performance. Finally do note that random partitioning might not be the best practice and maybe stratifying the cross-validation procedure (e.g. by geographical regions, testing site, etc.) allows a more realistic scenario to evaluate generalisation performance.
