Objective function, cost function, loss function: are they the same thing?

Question

In machine learning, people talk about objective function, cost function, loss function. Are they just different names of the same thing? When to use them? If they are not always refer to the same thing, what are the differences?

Answer 1

These are not very strict terms and they are highly related. However:

• Loss function is usually a function defined on a data point, prediction and label, and measures the penalty. For example:
  • Square loss: $l(f(x_i|\theta),y_i) = \left (f(x_i|\theta)-y_i \right )^2$, used in linear regression
  • Hinge loss: $l(f(x_i|\theta), y_i) = \max(0, 1-f(x_i|\theta)y_i)$, used in SVM
  • 0/1 loss: $l(f(x_i|\theta), y_i) = 1 \iff f(x_i|\theta) \neq y_i$, used in theoretical analysis and definition of accuracy
• Cost function is usually more general. It might be a sum of loss functions over your training set plus some model complexity penalty (regularization). For example:
  • Mean Squared Error: $MSE(\theta) = \frac{1}{N} \sum_{i=1}^N \left (f(x_i|\theta)-y_i \right )^2$
  • SVM cost function: $SVM(\theta) = \|\theta\|^2 + C \sum_{i=1}^N \xi_i$ (there are additional constraints connecting $\xi_i$ with $C$ and with training set)
• Objective function is the most general term for any function that you optimize during training. For example, a probability of generating training set in maximum likelihood approach is a well defined objective function, but it is not a loss function nor cost function (however you could define an equivalent cost function). For example:
  • MLE is a type of objective function (which you maximize)
  • Divergence between classes can be an objective function but it is barely a cost function, unless you define something artificial, like 1-Divergence, and name it a cost

Long story short, I would say that:

A loss function is a part of a cost function which is a type of an objective function.

All that being said, thse terms are far from strict, and depending on context, research group, background, can shift and be used in a different meaning. With the main (only?) common thing being "loss" and "cost" functions being something that want wants to minimise, and objective function being something one wants to optimise (which can be both maximisation or minimisation).

Answer 2

Quoting from section 4.3 in "Deep Learning" book by Ian Goodfellow, Yoshua Bengio, Aaron Courville (emphasis in the original):

The function we want to minimize or maximize is called the objective function, or criterion. When we are minimizing it, we may also call it the cost function, loss function, or error function. In this book, we use these terms interchangeably, though some machine learning publications assign special meaning to some of these terms.

In this book at least, loss and cost are the same.

Answer 3

In Andrew NG's words-

"Finally, the loss function was defined with respect to a single training example. It measures how well you're doing on a single training example. I'm now going to define something called the cost function, which measures how well you're doing an entire training set. So the cost function J which is applied to your parameters W and B is going to be the average with one of the m of the sum of the loss function applied to each of the training examples and turn."

Answer 4

According to Prof. Andrew Ng (see slides on page 11),

Function h(X) represents your hypothesis. For fixed fitting parameters theta, it is a function of features X. I'd say this can also be called the Objective Function.

The Cost function J is a function of the fitting parameters theta. J = J(theta).

According to the Hastie et al.'s textbook "Elements of Statistical Learning", by p.37:

"We seek a function f (X) for predicting Y given values of the input X." [...] the loss function L(Y, f(X)) is "a function for penalizing the errors in prediction",

So it seems "loss function" is a slightly more general term than "cost function". If you seek for "loss" in that PDF, I think that they use "cost function" and "loss function" somewhat synonymously.

Indeed, p. 502

"The situation [in Clustering] is somewhat similar to the specification of a loss or cost function in prediction problems (supervised learning)".

Maybe these terms exist because they evolved independently in different academic communities. "Objective Function" is an old term used in Operations Research, and Engineering Mathematics. "Loss function" might be more in use among statisticians. But I'm speculating here.

Answer 5

The loss function computes the error for a single training example, while the cost function is the average of the loss functions of the entire training set.

Answer 6

The terms cost and loss functions are synonymous. Some people also call them the error function. The more general scenario is to define an objective function first that we want to optimize. This objective function could be to:

1. maximize the posterior probabilities (e.g., naive Bayes)
2. maximize a fitness function (genetic programming)
3. maximize the total reward/value function (reinforcement learning)
4. maximize information gain/minimize child node impurities (CART decision tree classification)
5. minimize a mean squared error cost (or loss) function (CART, decision tree regression, linear regression, adaptive linear neurons, …
6. maximize log-likelihood or minimize cross-entropy loss (or cost) function minimize hinge loss (support vector machine)

Answer 7

Actually to be simple If you have m training data like this (x(1),y(1)),(x(2),y(2)), . . . (x(m),y(m)) We use loss function L(ycap,y) to find loss between ycap and y of a single training set If we want to find loss between ycap and y of a whole training set we use cost function.

Note:- ycap means output from our model And y means expected output

Note:- Credit goes Andrew ng Resource: coursera neural network and deep learning

Answer 8

To give you a short answer, according to me they are synonymous. However, the cost function is used more in optimization problem and loss function is used in parameter estimation.

Answer 9

How about Score function?

Not related directly to the question, but I wanted to add this here, to have a completed reference to all these computational terminologies.

In statistics, the score (or informant[1]) is the gradient of the log-likelihood function with respect to the parameter vector.

This term is used specifically with the Maximum likelihood estimation under the econometric modeling field. So, the Score function can also be considered as an objective function.

Reference

https://en.wikipedia.org/wiki/Score_(statistics)