It is very well known that L1 and L2 regularization can help in reducing the generalization error, and their effectiveness has been empirically demonstrated across a large set of machine learning methods.

However, when I try to understand how these regularizations help in a specific setting, say logistic regression, I found a surprisingly small amount of resources (maybe I just googled the wrong keywords).

For example, it is very well-known that L2 regularization introduces biases and reduces the variances, we can derive the bias and variance in closed-form in linear regression case, but when I try to learn the same problem under logistic regression setting, I cannot find anything. Yes, deriving closed-form solution in logistic regression itself is unsolvable, but I cannot find good explanation even without closed-form. (I have also tried to learn many other properties in these regularizations in logistic regression, but let's just use this simplest question as an example).

So, I wonder if the community has actually cared to work on these theoretical questions regarding logistic regression, or did we just directly borrow the conclusions from linear regressions? If we did, what justifies doing so?

  • $\begingroup$ Have you read the 7.1.1 of Deep Learning Book by Goodfellow? The expalation there seems very clear. Highly recommend. $\endgroup$ – Lerner Zhang Apr 23 '19 at 5:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.