4
$\begingroup$

There is one thing in data science that I cannot understand. When we have algorithms like Random Forest, Gradient Boosting, Neural Networks, we split our data into three parts - train (to train our model), validation (to choose best hyperparameters), and test (to get most accurate out of sample error).

In standard OLS linear regression it's obvious that we won't split our data to training, validation and testing, but only training and testing, since there is no hyperparameters to optimize. However, the game changes when we consider regularization, which implies need of estimation of hyperparameters associated with certain regularization (L1, L2 or Elastic Net). I've never seen in my life, that someone would split data into three parts, to search on the validation set for the best hyperparameters for regularization. In contrast everyone does it on test set.

But this is a little problematic isn't it? Observe, that if you choose regularization hyperparameters on test set, then you don't posses any accurate out of sample result (and this is argument, why you should provide a validation set).

Could you please explain to me this phenomenon?

Improve this question
$\endgroup$
4
  • 7
    $\begingroup$ The question is based on flawed reasoning. Regularized regression doesn’t differ from any other model, you use the same procedures as in other cases. The fact that you “never seen it” proves nothing, if you search better you would easily find many examples. Absence of evidence is not evidence of absence. $\endgroup$
    – Tim ♦
    Aug 27, 2022 at 15:55
  • 1
    $\begingroup$ even in OLS you will use crossvalidation to iterate over inputs (eg choosing the order of a polynomial model). The only difference is that ols and regularised regression are less computationally intense and therefore you use crossvalidation rather than a single split of training and validation data $\endgroup$
    – seanv507
    Aug 27, 2022 at 17:23
  • 3
    $\begingroup$ Some people misunderstand how hyperparameter search is performed, and use test set to find best performing hyperparam (instead of a validation set and reserving a separate test set for model evaluation). One of those people is unfortunately my undergrad supervisor. It's wrong, but this practice exists. $\endgroup$
    – Nuclear03020704
    Aug 28, 2022 at 5:48
  • 1
    $\begingroup$ @Nuclear03020704 Mrs Marsupial and I wrote a paper for people like your undergraduate supervisor, it used to be quite a common practice that should have died out faster as more computational power became available. jmlr.org/papers/volume11/cawley10a/cawley10a.pdf $\endgroup$
    – Dikran Marsupial
    Sep 2, 2022 at 12:57

1 Answer 1

Reset to default
16
$\begingroup$

It’s very common to use a validation set or a cross-validation scheme (k-fold CV, nested CV) to select the regularization hyper parameter. The software for regularization sometimes abstracts this step away. Glmnet is the gold standard for regularization in R. You can choose the penalty parameter(s) with cv.glmnet(). The cv in cv.glmnet() refers to cross validation. In Python, GridSearchCV,RandomSearchCV, or EstimatorCV (as suggested by @BenReiniger), can be used to find the best penalty hyperparameter of a lasso/ridge/elastic-net estimator.

However, for linear ridge regression models you can calculate leave-one-out cross validation with a closed form expression. You don’t need to go through the process of actually doing leave-one-out cross validation. This is called generalized cross validation.

Improve this answer
$\endgroup$
4
  • 2
    $\begingroup$ And unless the sample size is huge (sometimes > 20,000 independent observations) it is often unwise to split at all, as more precise model performance metrics can be estimated using resampling (e.g., 100 repeats of 10-fold cross-validation) and nested resampling. The idea of splitting, especially when splitting twice, needs to be considered carefully. Leave-out-one cross-validation can be an excellent approach but it does not properly capture the extreme volatility in feature selection. $\endgroup$
    – Frank Harrell
    Sep 2, 2022 at 12:03
  • 2
    $\begingroup$ There is a closed form leave-one-out CV approximation for e.g. logistic regression (including kernel variants) as well. It seems to work reasonably well, but it's main advantage is that it is cheap (so good as the inner loop in nested cross-validation). Agree with @FrankHarrell about the resampling if you can afford it. Is there and advantage of repeated k-fold over just repeated random test/train splits (which seems more uniform)? $\endgroup$
    – Dikran Marsupial
    Sep 2, 2022 at 12:54
  • $\begingroup$ @Ben Reinger I included your link in the comment. $\endgroup$
    – Eli
    Sep 2, 2022 at 13:05
  • $\begingroup$ @Fran kHarrell I agree with your comment. I wrote "validation set" as a catchall for all a single validation set, k-fold CV, nested CV, etc. I tried to make that more explicit in an edit. $\endgroup$
    – Eli
    Sep 2, 2022 at 13:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.