Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Regularization and Cross validation are two of the most important techniques for preventing overfitting, but it's not clear to me when one should be used over the other, or when both should be used together.
So my questions are:
(1) When would you use regularization over CV?
(2) When would you use CV over regularization?
(3) When would you use CV and regularization together?
(4) When would you use some other method to prevent overfitting?
This was clarified in comment
So isn't cross validation technically a technique for preventing overfitting, which is also the purpose of regularization?
Not exactly. For example, imagine that your data has $n$ samples and $p$ features with $n \ll p$. In such case fitting a multivariate regression model to all the features would overfit, so you need a model with less features. You could compare all the possible models, i.e. all the possible combinations of the $p$ features, and use cross validation to pick the best one, but this would likely be time consuming. Regularization, does this in single step by penalizing the overtly complex models. One example is LASSO regression that will push the regression coefficients of the "unnecessary" variables to zero (so technically, remove them). With fitting regularized multivariate regression you need to fit one model, instead of $2^p$ models, so that is much faster solution. Also, the regularized model still can use all the features (parameters do not need to be shrinked to exact zero), rather then selecting best features, such model is not possible when doing feature selection alone. Additionally, you can check this thread for learning why this is not that simple.
More generally, cross validation and regularization serve different tasks. Cross validation is about choosing the "best" model, where "best" is defined in terms of test set performance. Regularization is about simplifying the model. They could, but do not have to, result in similar solutions. Moreover, to check if the regularized model works better then unregularized you would still need cross validation.
As Tim mentioned in the comment, regularization and cross validation are different things. And they often are used together.
For example, we know we have the overfitting problem, and we want to regularize the model. But how much to regularize? We need cross validation to tell us, i.e., the we split the data set into two sets, and tune the regularization parameter in one set and check the performance on both sets.
