# What does the term "regularization" refer to specifically?

Initially I thought that "regularization" referred to specific methods to reduce overfitting by putting a penalizer term in the cost function that uses a norm eg. L1, L2 norm. But recently I've seen people use the the term "regularization" to refer to any method that reduces overfitting eg. I've heard people call random dropout in neural networks as a regularization method. So which usage of the term is correct?

• Commented Apr 15, 2021 at 21:24
• I would say that "regularization" is any method that constrains parameters and keeps them from floating freely. As @AryaMcCarthy notes, dropout sets some parameters to zero (just as the Lasso does). As such, one could even call feature selection a kind of regularization. But I'm definitely not an expert, so I'd rather not post this as an answer. Commented Apr 16, 2021 at 4:11
• I think there's a connection to adding constraints based on some inductive bias for posing your optimization problem -- i.e. Tikhonov and norm-based regularization methods are just scalarizations (i.e. of the form $\text{objective} + \lambda \lVert \cdot \rVert$) of a multi-objective problem, which is also the framing used in the paper Arya links. I've heard of regularization used in all the contexts OP mentions, but in this light, I got a little stuck on the distinction between "constraint" and "regularization." Commented Apr 16, 2021 at 8:38

I agree with the comment that regularization is a way to constrain parameters. In that regard, not only are $$L_1$$ and $$L_2$$ penalties regularization, but neural network dropout is regularization, since it forces certain parameters to be zero. Likewise, convolutional neural networks could be argued to perform regularization by forcing some parameters to be equal and others to be zero.