Caveat: this is just a thought that occurred to my while I was driving to work, so maybe it's not well-considered.

One challenge to deep neural networks is the credit assignment problem: a node at one layer might have many, many inputs from previous iterations of the network, so disentangling how to adjust parameters during training can be very challenging.

There are other approaches to solving this problem, such as parameter sharing in CNNs and dropout/drop connect (which zero out some connections or nodes during training to approximate the separate effects of different paths through the network). But $L_1$ would seem to help to ameliorate this also, since it will zero out some weights entirely, and attribute the outcome to solely the nonzero paths.

Clearly $L_1$ regularization alone won't solve problems like co-adaptation (where two or more neurons work to negate each other), but does it at least help address the credit assignment problem? Has there been any research in this area?

I want to emphasize that I'm not asking about whether or not practitioners use $L_1$ regularization in general. I'm specifically asking whether $L_1$ can help ameliorate the credit assignment problem.

  • $\begingroup$ L1 (and L2) regularisation will help coadaptation, because using eg only one neuron will have lower cost $\endgroup$ – seanv507 Oct 4 '18 at 16:06

I can't point you to papers that estimate the strength of L1 regularization at solving this problem, but I can point you to tutorials and textbooks, such as chapter 3 of Neural Networks and Deep Learning and this Tensorflow tutorial, which suggests to me that L1 regularization is widely used.

  • 1
    $\begingroup$ Sure -- L1 induces sparsity, and sparsity is useful for reasons beyond (maybe) solving CAP. My question is not about whether L1 is used in (D)NNs, because the answer is "obviously." Instead, my question is specifically about whether L1 is also helpful for CAP. $\endgroup$ – Sycorax says Reinstate Monica Oct 11 '16 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.