3
$\begingroup$

Specifically, I know that in SGD one sums all the gradients for weights/biases for each minibatch and divides by the mini batch size, would one do the same thing for dropout networks? Or would they divide by the number of networks in the mini-batch for which the weight/bias is not dropped out?

$\endgroup$
0

1 Answer 1

2
$\begingroup$

Typically, given a 50% dropout (standard), the weights are multiplied by 0.5 when predicting new instances. The gradient is always computed as the average over minibatches, without scaling. That said, dropout networks use a far higher learning rate (10 to 100 times the usual values), so you can think of that as a form of scaling.

You can read all about it in the paper that introduced dropout.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.