5
$\begingroup$

Look at the definition of ADAM optimizer, from the original paper of Kingma and Ba (paper):

See algorithm $1$ for pseudo-code of our proposed algorithm Adam. Let $f(\theta)$ be a noisy objective function: a stochastic scalar function that is differentiable w.r.t. parameters $\theta$. We are interested in minimizing the expected value of this function, $\mathbb{E}[f(\theta)]$ w.r.t. its parameters $\theta$.

Of course, this definition is given in the context of minimization of loss function. But what I don't understand, is how can we say that we want to minimize $E[f(\theta)]$, when $f$ has clear formula (for example MSE). In this paper it make sense, since they say, that $f(\theta)$ is a stochastic function, but I don't understand where we have this stochastic part, when we just want to minimize MSE (for example, which does not have any stochastic part).

Could you please explain to me where is the stochastic part of the loss function, and why it make sense to take the expectation? What is the random variable?

$\endgroup$
3
  • 3
    $\begingroup$ The sentence right below your screenshot is the answer. “The stochasticity might come from the evaluation at random subsamples (minibatches) of datapoints, or arise from inherent function noise.” $\endgroup$ Jun 11 at 13:59
  • 4
    $\begingroup$ @AryaMcCarthy This seems to be a complete answer to the question $\endgroup$
    – Sycorax
    Jun 11 at 14:38
  • $\begingroup$ Also, since f is stochastic, then minimizing E[f] means minimizing a stochastic function. $\endgroup$
    – justhalf
    Jun 12 at 13:50

1 Answer 1

11
$\begingroup$

Converting my comment into an answer.

The sentence right below your screenshot in the paper is the answer.

The stochasticity might come from the evaluation at random subsamples (minibatches) of datapoints, or arise from inherent function noise.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.