Should I use drop connect and L2 regularization?

Question

I've just learned about the drop connect technique for neural networks. It is my understanding that it is intended to reduce over fitting and is referred to as regularization. Is there additional benefit to using both drop connect and L2 regularization?

Answer 1

I can't give you an answer specifically for DropConnect, but can for Dropout (which is the technique from which DropConnect was derived, and is actually a special case of DropConnect). Whereas DropConnect zeros individual connections, Dropout zeros the activity of entire units.

Srivastava et al. (2014). Dropout: A Simple Way to Prevent Neural Networks from Overfitting.

They combine Dropout with what they call 'max-norm regularization', which simply means constraining the $l_2$ norm of the weights. E.g. for a weight vector $w$, they impose $\|w\|_2 \le c$, where the constant $c$ is a tunable hyperparameter. They impose this constraint by renormalizing the weights to have norm $c$ if the norm exceeds this bound. They found that combining Dropout with max-norm regularization gave better generalization performance than using Dropout alone. They hypothesize that the reason this works is that constraining the norm of the weights allows larger learning rates to be used without the weights blowing up. If this hypothesis is true, then the success of the technique should also depend on the details of the update rules.

Given that this works for Dropout, I'd say you might as well try and see if it works with DropConnect, unless you find information to the contrary.