0
$\begingroup$

In the paper "ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION" they write in the introduction "Often, objective functions are stochastic."

I know what a stochastic gradient descent is, but what does it mean that the objective function is stochastic?

$\endgroup$
2
$\begingroup$

In the case of linear regression, we try to minimize the mean square error between the true dependent variable and OLS estimate of the dependent variable. The mean square error is the objective function in the case of the linear regression.

The mean square error is summed over observations in the training set. The observations are stochastic because they come from the population probability distribution. Therefore, the objective function is stochastic also which depends on the random samples you observed.

In the algorithm of ADAM, the training sample changes with time, and the objective function changes also because it is defined on the given train sample. Therefore, we have a series of objective functions that form a stochastic process.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.