I understand the intended use cases for both stochastic approximation algorithms like SPSA or FDSA, and for SGD algorithms like Adam. SPSA is intended for noisy objective functions, and Adam for randomized mini batches.

So for me it looks like the only difference between both of them is where the randomness comes from. You can also use SPSA for mini batches and you can use Adam for noisy objective function. And both of them can be used for mini batches where each training example is noisy by itself.

So with that said they both look very semilar in their application, yet in the literature they are always kind of seperate from each other.

Why is that? What fundamental difference distinguishes them from each other?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.