9
$\begingroup$

A loss function is a function that measures the distance between the expected value and the actual value of a model (an example of a loss function is the cross entropy).

An energy function can be defined as a function that we want to minimise or maximise and it is a function of the variables of the system. It is referred to as "energy function" because it is often related or compared to the concept of "energy" in physics.

These two expression seem to refer to the same concept. Is there any difference between a loss function and an energy function (especially, in machine learning and computer vision)?

$\endgroup$
1
  • 2
    $\begingroup$ Your characterization of loss functions differs (sharply) from that in the Wikipedia article you reference (in the sense that your description is a very special case of the general concept), so reading that article more closely might be a good next step. $\endgroup$
    – whuber
    May 20, 2019 at 16:34

1 Answer 1

8
$\begingroup$

Yes I found these quotes by Yann LeCun particularly helpful:

  • "A distinction should be made between the energy function, which is minimized by the inference process, and the loss functional (introduced in Section 2), which is minimized by the learning process."
  • "A loss functional, minimized during learning, is used to measure the quality of the available energy functions."

An analogy to psychology: This implies that energy functions are analogous to your intuition/habits: first you train your intuition/habits through experience, & external feedback serves as the 'loss function' that your intuition/habits adapt to. If your intuition is well trained the right decisions will become the easiest & thus require the least 'mental/activation energy' from you. Therefore decision making in this ideal scenario becomes mostly a matter of 'minimizing the energy' required by your chosen decision.

Further Reading: A blog post by OpenAI describing energy based learning can be found here. And the full document where I found these quotes is here; it is a thorough tutorial on energy based learning by scholar Yann LeCun which you may find useful to read for a deeper understanding.

P.S. By convention the energy function is minimized. I believe this is because in physics: systems naturally seek low-energy states.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.