Initially I thought that "regularization" referred to specific methods to reduce overfitting by putting a penalizer term in the cost function that uses a norm eg. L1, L2 norm. But recently I've seen people use the the term "regularization" to refer to any method that reduces overfitting eg. I've heard people call random dropout in neural networks as a regularization method. So which usage of the term is correct?
$\begingroup$
$\endgroup$
3
asked Apr 15, 2021 at 20:57
-
1$\begingroup$ Well, dropout can be framed as regularization under your first definition. $\endgroup$– Arya McCarthyCommented Apr 15, 2021 at 21:24
-
2$\begingroup$ I would say that "regularization" is any method that constrains parameters and keeps them from floating freely. As @AryaMcCarthy notes, dropout sets some parameters to zero (just as the Lasso does). As such, one could even call feature selection a kind of regularization. But I'm definitely not an expert, so I'd rather not post this as an answer. $\endgroup$– Stephan KolassaCommented Apr 16, 2021 at 4:11
-
$\begingroup$ I think there's a connection to adding constraints based on some inductive bias for posing your optimization problem -- i.e. Tikhonov and norm-based regularization methods are just scalarizations (i.e. of the form $\text{objective} + \lambda \lVert \cdot \rVert$) of a multi-objective problem, which is also the framing used in the paper Arya links. I've heard of regularization used in all the contexts OP mentions, but in this light, I got a little stuck on the distinction between "constraint" and "regularization." $\endgroup$– chang_trentonCommented Apr 16, 2021 at 8:38
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
I agree with the comment that regularization is a way to constrain parameters. In that regard, not only are $L_1$ and $L_2$ penalties regularization, but neural network dropout is regularization, since it forces certain parameters to be zero. Likewise, convolutional neural networks could be argued to perform regularization by forcing some parameters to be equal and others to be zero.
Even if the terminology is a bit loose, the field of machine learning seems to do okay. Many fields are loose with terminology, too. For instance, ask a musician to explain the difference between a "riff" and a "lick".
