1
$\begingroup$

I was considering various loss functions and optimization methods, for my neural network, and while watching a youtube video about them this illustration was shown:

loss functions

(You can find the animated version here.)

I already know that a marble in a smooth hilly terrain is an often used analogy for gradient decent type of optimization. It made me wonder what optimization method a literal marble in smooth hilly terrain would be. My thought is that it would have to include at least:

  • Gravity
  • Friction
  • Momentum

Using typical and simple physics laws, what common optimization method would comes closest?

Improve this question
$\endgroup$
4
  • $\begingroup$ In what sense do you mean that a "marble" (thought of metaphorically as some iterative procedure to seek a minimum) could be a "loss function"? What do you mean by "loss function" in this context? $\endgroup$
    – whuber ♦
    Commented Jun 3, 2017 at 15:46
  • $\begingroup$ I meant it more general. I was thinking "loss function" and "optimization method" in combination. An example would be to use mean square error as a loss function, and stochastic gradient descent as the optimization method. What combination of loss function and optimizer would most closely fit the marble in a hilly terrain analogy, where you want to minimize the elevation of the marble, given gravity, friction and momentum. Each iteration of the optimization would be analogous to time. I'll update the question to make it more clear. $\endgroup$
    – André Christoffer Andersen
    Commented Jun 3, 2017 at 16:23
  • $\begingroup$ In that sense, "loss function" determines the surface. It doesn't otherwise seem to have any connection with the optimization procedure. That's why I cannot understand what you're trying to ask. $\endgroup$
    – whuber ♦
    Commented Jun 3, 2017 at 16:25
  • $\begingroup$ @whuber Good point. I'll do another update, in order not to conflate the concepts. $\endgroup$
    – André Christoffer Andersen
    Commented Jun 3, 2017 at 16:27

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.