I'm am using the h2o.deeplearning() function in R, and in the parameter setting, there is l1 and l2. The documentation for L1/L2 regularization says:

L1 regularization (can add stability and improve generalization, causes many weights to become 0). Defaults to 0. L2 regularization (can add stability and improve generalization, causes many weights to be small. Defaults to 0. Intuition: L1 lets only strong weights survive (constant pulling force towards zero), while L2 prevents any single weight from getting too big.

I understand the basic concept of L1 and L2 from both intuition and the sparsity perspective, but I don't understand what does 0 mean here? In some of the coding I saw online, like the code below, it sets L1/L2 as 1e-5...i see they are small numbers...but again what they mean and how it's different to zero?

Sincerely wish for some insights. Thank you.

m3 <- h2o.deeplearning(model_id = "dl_model_tuned", 
                       hidden = c(128,128,128), 
                       l1 = 1e-5,  ## add some L1/L2 regularization
                       l2 = 1e-5)

When you use L1/L2 regularization with parameter $\lambda$ it means that instead of your regular loss $L(w)$ you use $$L(w) + \lambda\|w\|_p$$ for appropriate $p$.

Here 'parameter l1 defaults to 0' means that $\lambda$ for $\|w\|_1$ is equal to 0 by default.

  • $\begingroup$ Thank you so much Jakub for you reply. What kind of value is usually for this lamda, and what intuitive difference it makes on the DL model? such as the larger the lamda is, the more general the model fit is?? and again, how large this lamda can go up to? $\endgroup$ – MeiNan Zhu Nov 3 '17 at 16:23
  • $\begingroup$ The higher the lambda is, the more you penalize big weights. It can be also interpreted as imposing some kind of prior (look up Maximum A Posteriori & regularization). If you take too big lambda the weights are going to be really small, or for L1 regularization even most of them will become zero (this can be formulated as enforcing sparsity). $\endgroup$ – Jakub Bartczuk Nov 3 '17 at 18:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.