What is the difference between dropout and drop connect?
AFAIK, dropout randomly drops hidden nodes during training but keeps them in testing, and drop connect drops connections.
But isn't dropping connections equivalent to dropping the hidden nodes? Aren't the nodes (or connections) just a set of weights?
DropOut and DropConnect are both methods intended to prevent "co-adaptation" of units in a neural network. In other words, we want units to independently extract features from their inputs instead of relying on other neurons to do so.
Suppose we have a multilayered feedforward network like this one (the topology doesn't really matter). We're worried about the yellow hidden units in the middle layer co-adapting.
To apply DropOut, we randomly select a subset of the units and clamp their output to zero, regardless of the input; this effectively removes those units from the model. A different subset of units is randomly selected every time we present a training example.
Below are two possible network configurations. On the first presentation (left), the 1st and 3rd units are disabled, but the 2nd and 3rd units have been randomly selected on a subsequent presentation. At test time, we use the complete network but rescale the weights to compensate for the fact that all of them can now become active (e.g., if you drop half of the nodes, the weights should also be halved).
DropConnect works similarly, except that we disable individual weights (i.e., set them to zero), instead of nodes, so a node can remain partially active. Schematically, it looks like this:
These methods both work because they effectively let you train several models at the same time, then average across them for testing. For example, the yellow layer has four nodes, and thus 16 possible DropOut states (all enabled, #1 disabled, #1 and #2 disabled, etc).
DropConnect is a generalization of DropOut because it produces even more possible models, since there are almost always more connections than units. However, you can get similar outcomes on an individual trial. For example, the DropConnect network on the right has effectively dropped Unit #2 since all of the incoming connections have been removed.
The original papers are pretty accessible and contain more details and empirical results.
Yes, but they are slightly different in terms of how the weights are dropped.
These are the formulas of DropConnect (left) and dropout (right).
So dropout applies a mask to the activations, while DropConnect applies a mask to the weights.
The DropConnect paper says that it is a generalization of dropout in the sense that
DropConnect is the generalization of Dropout in which each connection, instead of each output unit as in Dropout, can be dropped with probability p.
Based on what I saw on Tensorflow 2.5. From the workflow tf.keras.layer.Dropout to tf.python.ops.nn_ops.py function dropout and dropout_v2 (Line 5059 -> 5241), The code does not adjust the shape of the input layer but to mask and do matmul instead with a mask of 0 and 1 to gain speed of continuous vector/matrix/tensor, which is compatible as DropConnect. In another word, although not completely guarantee, the real implementation of Dropout is to fit with the real-world problem (training speed).
By the way, the idea of Dropout is just a general scheme (removing neuron unit of a FC layer). Matt Krause has already well explained it
