1
$\begingroup$

I see two options:

  1. Apply gradient noise before you apply an optimization method such as Adam (or just SGD with momentum or something else). I.e. you calculate the gradients, then you add noise to it, and then you pass the resulting term over to the optimization method (such as Adam) which will return the updates for the parameters.

  2. You apply the noise afterwards, which means just adding noise to the parameters. I.e. you calculate the gradients, pass them over to the optimization method (such as Adam) and then you add the noise to the updates of the parameters.

In the case of standard SGD without momentum, this is equivalent.

I think I have seen both variants implemented but it's not exactly clear to me which of them makes more sense, or should be better from a theoretical point of view (could still look different empirically).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.