I'm am using the
h2o.deeplearning() function in R, and in the parameter setting, there is l1 and l2. The documentation for L1/L2 regularization says:
L1 regularization (can add stability and improve generalization, causes many weights to become 0). Defaults to 0. L2 regularization (can add stability and improve generalization, causes many weights to be small. Defaults to 0. Intuition: L1 lets only strong weights survive (constant pulling force towards zero), while L2 prevents any single weight from getting too big.
I understand the basic concept of L1 and L2 from both intuition and the sparsity perspective, but I don't understand what does 0 mean here? In some of the coding I saw online, like the code below, it sets L1/L2 as 1e-5...i see they are small numbers...but again what they mean and how it's different to zero?
Sincerely wish for some insights. Thank you.
m3 <- h2o.deeplearning(model_id = "dl_model_tuned", hidden = c(128,128,128), l1 = 1e-5, ## add some L1/L2 regularization l2 = 1e-5)