I was considering various loss functions and optimization methods, for my neural network, and while watching a youtube video about them this illustration was shown:

loss functions

(You can find the animated version here.)

I already know that a marble in a smooth hilly terrain is an often used analogy for gradient decent type of optimization. It made me wonder what optimization method a literal marble in smooth hilly terrain would be. My thought is that it would have to include at least:

  • Gravity
  • Friction
  • Momentum

Using typical and simple physics laws, what common optimization method would comes closest?

  • $\begingroup$ In what sense do you mean that a "marble" (thought of metaphorically as some iterative procedure to seek a minimum) could be a "loss function"? What do you mean by "loss function" in this context? $\endgroup$ – whuber Jun 3 '17 at 15:46
  • $\begingroup$ I meant it more general. I was thinking "loss function" and "optimization method" in combination. An example would be to use mean square error as a loss function, and stochastic gradient descent as the optimization method. What combination of loss function and optimizer would most closely fit the marble in a hilly terrain analogy, where you want to minimize the elevation of the marble, given gravity, friction and momentum. Each iteration of the optimization would be analogous to time. I'll update the question to make it more clear. $\endgroup$ – André Christoffer Andersen Jun 3 '17 at 16:23
  • $\begingroup$ In that sense, "loss function" determines the surface. It doesn't otherwise seem to have any connection with the optimization procedure. That's why I cannot understand what you're trying to ask. $\endgroup$ – whuber Jun 3 '17 at 16:25
  • $\begingroup$ @whuber Good point. I'll do another update, in order not to conflate the concepts. $\endgroup$ – André Christoffer Andersen Jun 3 '17 at 16:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.