1
$\begingroup$

I'm implementing dropout using the paper by Srivastava et al. In it, the authors suggest that when a neuron is dropped out, it is temporarily removed from the network completely:

However, when explaining the feedforward operation, the Bernoulli distribution is not applied to the bias:

This suggests that the activation function would be $y_i^{(l+1)}=f(b_i^{(l+1)})$, and therefore non-zero.

Further (ignoring the above bias question), assuming that $z_i^{(l+1)}$ is zero; if the sigmoid function is used for $f(x)$, wouldn't the activation function be non-zero as, for the sigmoid function, $f(x)=\frac{1}{2}$ for $x=0$.

I'm wondering if I'm understanding this correctly. I thought that the activation function should become zero, but I don't understand how the activation function could be zero.

Thanks for any help.

$\endgroup$
1
$\begingroup$

I've just spotted my mistake and I understand now. The activation function of the next layer uses the modified value of the previous layer using the Bernoulli distribution vector. Therefore, if dropout is only applied to the hidden layer of a network with one hidden layer:

  • The values of the neurons in each hidden layer are calculated normally
  • After, a vector generated by Bernoulli distribution is multiplied with the vector of hidden layer neurons using the Hadamard product. This creates the new vector where the value of some hidden layer neurons is zero.
  • The values of output layer neurons are calculated normally, except some of the hidden layer neurons do not contribute due to having a value of zero.

Sorry for any spam, I realised my mistake right after posting this question but felt I should leave it up in case others have the same confusion.

Thanks.

$\endgroup$
1
  • 1
    $\begingroup$ Not a spam, feel free to ask. Answering your own questions is also absolutely ok, since it often helps others to learn something new. $\endgroup$
    – Tim
    Jun 19 '20 at 8:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.