# Is it possible to combine SPSA and Adam?

In SGD algorithms such as Adam you generally make a bad estimate of the gradient of the loss function and take that gradient to move the parameters in the desired direction. Gradient free methods such as SPSA do basically the same, they make a bad estimate of the gradient, by randomly perturbing the parameter vector in two opposite directions. Then they move the parameters along the direction of the gradient.

My question is: Would it possible to combine the gradient calculation from SPSA with the "step taking" from Adam and are there any papers on such an approach?

SPSA only uses a decaying step sequence, and Adam seems superior it this regard.

Edit: The resulting combined algorithm is not supposed to be used to train a ANN but for simulation based optimization.

With a bit more analysis, we could argue that SPSA isn't built at all towards something like Adam, even if we try to average out the gradient estimates to get different magnitudes for each component. With a quick Taylor series, we may see that $$\hat g_1$$, the gradient for the first component, is estimated as:
$$\hat g_1\simeq(g\cdot e)e_1+\epsilon_1=g_1\pm g_2\pm g_3\pm\dots+\epsilon_1$$
where $$g$$ is the true gradient, $$e$$ is the signs of the perturbations, and $$\epsilon_1$$ is an additional error term (including things like noisy evaluations and the fact that the perturbation step size needs to vanish). This means the main noise in the estimate of the first component depends on all of the other components combined. For components which are large, this noise is negligible and we get an accurate estimate. For components which are small, this noise is significant and we need a large amount of iterations before our averaged estimate starts to look good.