How does one implement lasso regularization or elastic net on neural networks? (feed forward in particular). I know that closed form solutions for this problems don't exist, still how are they implemented? How do I compute the gradients?
Attention, I'm familiar with both procedures and what they accomplish, what I'm asking is how to implement them, how to formulate the gradients.
You could take a look at sparse autoencoders, which sometimes put a L1 penalty on the neural activations, which from an optimization point of view is similar to Lasso (L1 penalty on weights). Here is a Theano implementation. An alternative is given from the UFLDL tutorial:
This objective function presents one last problem - the L1 norm is not differentiable at 0, and hence poses a problem for gradient-based methods. While the problem can be solved using other non-gradient descent-based methods, we will "smooth out" the L1 norm using an approximation which will allow us to use gradient descent. To "smooth out" the L1 norm, we use $\sqrt{x^2 + \epsilon}$in place of $\left| x \right|$, where ε is a "smoothing parameter" which can also be interpreted as a sort of "sparsity parameter" (to see this, observe that when ε is large compared to x, the x + ε is dominated by ε, and taking the square root yields approximately $\sqrt{\epsilon}$).
So you could follow their approach using the smooth approximation, but you can also go for the exact gradient, which is discontinuous at 0, but sometimes that may not be a problem. For a example, the popular ReLU neuron also has a gradient that is discontinuous at 0, but that it's not a problem for most applications.
Also, you can look at Extreme Learning Machines (ELM), which are MLPs that only learn the weights of the final layers and use random hidden layer weights. This seems odd, but it can achieve reasonable results in a very fast time (see right frame). Since the optimization problem to train the ELMs is only linear regression, you could use any Lasso tool for this.
