Cross validation : hyper-parameter tuning ? or model validation?

Question

I have been seaching internet for exact definition of cross validation . I have come acrossed a few different ideas, with different terminology. I don't know if I have understand correctly.

Basically, what I understand now is, there are two major applications of cross-validation.

1. Hyper-parameter tuning. Lasso has a parameter $\lambda$ . We don't know which $\lambda$ we should use. So we split the into training set and testing set. Try different $\lambda$ with these 'sub-problems' and see which $\lambda$ gives the best performance.

2. Model validation. Imaging that I have implemented both Lasso and Gradient Boosted Regression Tree. I want to know which one would work better in real-life ( predicting new , unseen data). So I split the data into training/testing parts. I will choose the one that yields better out-of-sample performance in cross validation.

Is my understanding correct?

Thanks

Answer 1

I'd say mostly the first (i.e. Hyper-parameter tuning).

If you have a sufficiently large hold-out test set you can evaluate the models pretty reliably. When wanting to select hyperparameters, having a validation set could cause your model to overfit on that. CV makes it much harder to do so.

Answer 2

Model Selection and Model Hyperparams Tuning are conceptually equivalent.

If a model has a single Hyperparam, which in turn can have 3 possible values, it's like you have 3 different models. Very similar models maybe, but for the purpose of optimization they are treated as if they are 3, completely different and independent.

So wheter your hyperparam is saying "use N=1/2/3 neurons per layer" or "use model 1=linear regression/2=lasso/3=ridge" you have indeed 3 different models, all the same.

Crossvalidation would then be a way to compare these 3 competing models and select the one that is "most adept" to learn the behavior you want to resemble. You can see that it is the "most adept", because everytime yor train it on a subset of your data, it then performs well on the remaining data.

Answer 3

Suppose you have split your data into train-validation-test sets. You do not usually split it into train-test unless your models have no hyperparameters.

Validation set is always used to tune hyperparameters of your models. Test set is used to assess the final performance of your model, and compare different classes of models (e.g. random forest vs neural network vs svm) by their performance.

Cross-validation is tightly connected to the validation set and hyperparameter selection. In general, cross-validation splits your large mass of (train + validation) into training and validation sets repeatedly. This should provide you with an out-of-sample performance approximation, and based on it you choose your hyperparameters (model). However, you can treat your model class (random forest, svm, neural network) as hyperparameter. In this way, you can choose your final model class using cross-validation.

Test set is used purely for reporting (how well your chosen model performs). According to some authors, you cannot use test set for ANY model selection, even if it is model class selection (svm vs random forest vs etc.).

However, I would not compare broad model classes using cross-validation. It is best to decide which model to use based on your test set. By using your test set purely for reporting as statisticians suggest, you are losing a lot of data. However, using it to choose a model class as the last step does not influence the out-of-sample overfitting much, and this is what all scientific papers do when they claim that their state-of-the-art method outperformed some other method on some large dataset.

As for #1) You are making a sympathetic error by confusing cross-validation division procedure and train-validation-test sets. Train-validation-test is a data set division, while cross-validation is a specific way HOW you can divide your mass of data into training and validation sets (while test set has nothing to do with cross-validation, and allocated separately).

As for #2) Yes, you can do Lasso and Gradient Boosted Regression Tree comparison using validation set (and cross-validation split method), but it would be better to compare them on the test set, while cross-validation (validation set) is used to find hyperparameters of your GRT and Lasso regression separately.