The literature discusses tanh and relu activation. Does drop connect not work with sigmoid activation?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Yes, they compare relu, tanh, and sigmoid activation functions in the original DropConnect paper (see table 2).
Wan et al. (2013). Regularization of neural networks using dropconnect.
answered Jun 27, 2016 at 2:46
$\begingroup$
$\endgroup$
Dropout and dropconnect can be seen as 2 things:
- Training a pseudo ensemble of networks when training one network. You slightly modify the network each time and ultimately use an approximate of their geometric mean as the network output.
- Making your network more robust and regularized by forcing your network parameters to not overly rely on each other.
Systematically, both of these methods are independent of which activation you choose to use because they are akin to general training strategies over the entire network. In short, you shouldn't have any problem using an activation unit of your choice.
