When to use regularization vs. cross validation [closed]

Regularization and Cross validation are two of the most important techniques for preventing overfitting, but it's not clear to me when one should be used over the other, or when both should be used together.

So my questions are:

(1) When would you use regularization over CV?

(2) When would you use CV over regularization?

(3) When would you use CV and regularization together?

(4) When would you use some other method to prevent overfitting?

• Regularization and cross validation are used for completely different purposes, so it is not clear why exactly are you asking about using one "instead" of another? Could you clarify? – Tim Jun 15 '20 at 6:34
• @Tim So my understanding is cross validation has multiple purposes, mostly centered around validating the hypothesis you obtained from a training set, by using a dataset that wasn't part of the training. By doing this, we can identify if the model is overfitting. If we identify that it's overfitting, we can choose a different hypothesis. So isn't cross validation technically a technique for preventing overfitting, which is also the purpose of regularization? – David Jun 15 '20 at 20:18
• There is a difference between can be used to and is used to. There are lots of uses for regularization, for example to find a well posed integral, to reduce propagated error of a parameter, which is not the same goal as finding goodness of fit to a curve, to reduce the variance of residuals of fitting, etc. – Carl Jun 30 '20 at 9:48

This was clarified in comment

So isn't cross validation technically a technique for preventing overfitting, which is also the purpose of regularization?

Not exactly. For example, imagine that your data has $$n$$ samples and $$p$$ features with $$n \ll p$$. In such case fitting a multivariate regression model to all the features would overfit, so you need a model with less features. You could compare all the possible models, i.e. all the possible combinations of the $$p$$ features, and use cross validation to pick the best one, but this would likely be time consuming. Regularization, does this in single step by penalizing the overtly complex models. One example is LASSO regression that will push the regression coefficients of the "unnecessary" variables to zero (so technically, remove them). With fitting regularized multivariate regression you need to fit one model, instead of $$2^p$$ models, so that is much faster solution. Also, the regularized model still can use all the features (parameters do not need to be shrinked to exact zero), rather then selecting best features, such model is not possible when doing feature selection alone. Additionally, you can check this thread for learning why this is not that simple.

More generally, cross validation and regularization serve different tasks. Cross validation is about choosing the "best" model, where "best" is defined in terms of test set performance. Regularization is about simplifying the model. They could, but do not have to, result in similar solutions. Moreover, to check if the regularized model works better then unregularized you would still need cross validation.

• Ah right. Sorry for the ill-posed question. I realize $p > n$ would lead to overfitting, or in the case of least squares linear regression, the matrix to be inverted actually becomes singular (unless you add a constant positive diagonal component to the matrix prior to inversion), and a unique solution can't be obtained. Is there any other point to regularization if overfitting and singularity isn't a concern? – David Jun 15 '20 at 23:48
• @David in machine learning overfitting always is a concern and some kind of regularization is used almost as a default. – Tim Jun 16 '20 at 6:32

As Tim mentioned in the comment, regularization and cross validation are different things. And they often are used together.

For example, we know we have the overfitting problem, and we want to regularize the model. But how much to regularize? We need cross validation to tell us, i.e., the we split the data set into two sets, and tune the regularization parameter in one set and check the performance on both sets.