In SGD algorithms such as Adam you generally make a bad estimate of the gradient of the loss function and take that gradient to move the parameters in the desired direction. Gradient free methods such as SPSA do basically the same, they make a bad estimate of the gradient, by randomly perturbing the parameter vector in two opposite directions. Then they move the parameters along the direction of the gradient.

My question is: Would it possible to combine the gradient calculation from SPSA with the "step taking" from Adam and are there any papers on such an approach?

SPSA only uses a decaying step sequence, and Adam seems superior it this regard.

Edit: The resulting combined algorithm is not supposed to be used to train a ANN but for simulation based optimization.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.