19
$\begingroup$

I was reading about optimization for an ill-posed problem in computer vision and came across the explanation below about optimization on Wikipedia. What I don't understand is, why do they call this optimization "Energy minimization" in Computer Vision?

An optimization problem can be represented in the following way:

Given: a function $f: A \to R$ from some set $A$ to the real numbers

Sought: an element $x_0$ in $A$ such that $f(x_0) ≤ f(x)$ for all $x$ in $A$ ("minimization") or such that $f(x_0) ≥ f(x)$ for all $x$ in $A$ ("maximization").

Such a formulation is called an optimization problem or a mathematical programming problem (a term not directly related to computer programming, but still in use for example in linear programming – see History below). Many real-world and theoretical problems may be modeled in this general framework. Problems formulated using this technique in the fields of physics and computer vision may refer to the technique as energy minimization, speaking of the value of the function $f$ as representing the energy of the system being modeled.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
12
$\begingroup$

Energy-based models are a unified framework for representing many machine learning algorithms. They interpret inference as minimizing an energy function and learning as minimizing a loss functional.

The energy function is a function of the configuration of latent variables, and the configuration of inputs provided in an example. Inference typically means finding a low energy configuration, or sampling from the possible configuration so that the probability of choosing a given configuration is a Gibbs distribution.

The loss functional is a function of the model parameters given many examples. E.g., in a supervised learning problem, your loss is the total error at the targets. It's sometimes called a "functional" because it's a function of the (parametrized) function that constitutes the model.

Major paper:

Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, and F. J. Huang, “A tutorial on energy-based learning,” in Predicting Structured Data, MIT Press, 2006.

Also see:

LeCun, Y., & Huang, F. J. (2005). Loss Functions for Discriminative Training of Energy-Based Models. In Proceedings of the 10th International Workshop on Artificial Intelligence and Statistics (AIStats’05). Retrieved from http://yann.lecun.com/exdb/publis/pdf/lecun-huang-05.pdf

Ranzato, M., Boureau, Y.-L., Chopra, S., & LeCun, Y. (2007). A Unified Energy-Based Framework for Unsupervised Learning. Proc. Conference on AI and Statistics (AI-Stats). Retrieved from http://dblp.uni-trier.de/db/journals/jmlr/jmlrp2.html#RanzatoBCL07

Improve this answer
$\endgroup$
7
  • 5
    $\begingroup$ Can you expand on what "They interpret inference as minimizing an energy function and learning as minimizing a loss function" means? How is an energy function different than a loss function? $\endgroup$
    – Cliff AB
    Commented Feb 7, 2016 at 21:42
  • $\begingroup$ Could you please elaborate your answer $\endgroup$
    – iamprem
    Commented Feb 7, 2016 at 21:54
  • $\begingroup$ @CliffAB Hopefully that's clearer? $\endgroup$
    – Neil G
    Commented Feb 7, 2016 at 21:55
  • $\begingroup$ @NeilG: to be honest, I'm still slightly confused. To me, it sounds like the "energy function" is essentially the same thing as the likelihood function in statistics. Is that a reasonable interpretation or am I missing something more subtle? $\endgroup$
    – Cliff AB
    Commented Feb 7, 2016 at 22:11
  • 1
    $\begingroup$ @NeilG Still hard to understand . The energy function is a function of the configuration of latent variables, and the configuration of inputs provided in an example. Inference typically means finding a low energy configuration... this sentence is confusing for me (not a native English speaker) . Could you give a real example(maybe iris classification? ) to show what part of a machine learning contain energy function or energy minimization . $\endgroup$
    – Mithril
    Commented Dec 7, 2017 at 8:53
2
$\begingroup$

In signal detection literature, the energy of a signal $x_t$ is defined as $$ E = \Sigma x_t^2 $$

When predicting some response y from some features x, a very common and simple way to proceed is to minimise the sum of the squared errors $$ SSE= \Sigma (y-\hat{y})^2 $$ where $\hat{y}$ is the fitted response. Notice the similarity? The SSE is energy. This energy is minimised by the fitted parameters.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ I think you're confusing the loss with the energy $\endgroup$
    – Neil G
    Commented Feb 7, 2016 at 21:59
  • $\begingroup$ I am using the standard definition of energy from signal processing. Computer science / machine learning people do tend to redefine terms, I guess. I come from stats and signal processing background $\endgroup$
    – stan
    Commented Feb 7, 2016 at 22:18
  • $\begingroup$ Your first formula is an energy function. The second formula is loss function since it's not a function of the configuration. $\endgroup$
    – Neil G
    Commented Feb 7, 2016 at 22:22
  • $\begingroup$ @Neil I'm sure you are using the terminology correctly as defined in the papers you cited. It's just a different terminology from what I'm used to where SSE is energy $\endgroup$
    – stan
    Commented Feb 7, 2016 at 22:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.