# Objective function, cost function, loss function: are they the same thing?

In machine learning, people talk about objective function, cost function, loss function. Are they just different names of the same thing? When to use them? If they are not always refer to the same thing, what are the differences?

• Oct 28, 2015 at 18:05

These are not very strict terms and they are highly related. However:

• Loss function is usually a function defined on a data point, prediction and label, and measures the penalty. For example:
• square loss $$l(f(x_i|\theta),y_i) = \left (f(x_i|\theta)-y_i \right )^2$$, used in linear regression
• hinge loss $$l(f(x_i|\theta), y_i) = \max(0, 1-f(x_i|\theta)y_i)$$, used in SVM
• 0/1 loss $$l(f(x_i|\theta), y_i) = 1 \iff f(x_i|\theta) \neq y_i$$, used in theoretical analysis and definition of accuracy
• Cost function is usually more general. It might be a sum of loss functions over your training set plus some model complexity penalty (regularization). For example:
• Mean Squared Error $$MSE(\theta) = \frac{1}{N} \sum_{i=1}^N \left (f(x_i|\theta)-y_i \right )^2$$
• SVM cost function $$SVM(\theta) = \|\theta\|^2 + C \sum_{i=1}^N \xi_i$$ (there are additional constraints connecting $$\xi_i$$ with $$C$$ and with training set)
• Objective function is the most general term for any function that you optimize during training. For example, a probability of generating training set in maximum likelihood approach is a well defined objective function, but it is not a loss function nor cost function (however you could define an equivalent cost function). For example:
• MLE is a type of objective function (which you maximize)
• Divergence between classes can be an objective function but it is barely a cost function, unless you define something artificial, like 1-Divergence, and name it a cost

Long story short, I would say that:

A loss function is a part of a cost function which is a type of an objective function.

All that being said, thse terms are far from strict, and depending on context, research group, background, can shift and be used in a different meaning. With the main (only?) common thing being "loss" and "cost" functions being something that want wants to minimise, and objective function being something one wants to optimise (which can be both maximisation or minimisation).

• +1. I've not seen a source for this but I have guessed that "objective" is the term used because it is your goal or objective to optimise that function, which could mean maximising something good or minimising something bad, although that difference is trivial, as any function can be negated. In contrast, the pejorative overtones of "loss" and "cost" do bite: I'd say it would be perverse to use either term except for something to be minimised. These points are tacit in your fine answer but deserve a little more emphasis. Oct 28, 2015 at 15:36
• The "M" in "MLE" stands for "maximum" not "minimum." I only mention this pedantic detail because this question was migrated from stackoverflow, and I've been bitten by the bug of minimizing the wrong function before Dec 26, 2016 at 22:33
• Actually, the objective function is the function (e.g. a linear function) you seek to optimize (usually by minimizing or maximizing) under the constraint of a loss function (e.g. L1, L2). Examples are ridge regression or SVM. You can also optimize the objective function without any loss function, e.g. simple OLS or logit.
– g3o2
Jun 10, 2017 at 16:54
• @ Nick Cox wrote ' the pejorative overtones of "loss" and "cost" do bite: I'd say it would be perverse to use either term except for something to be minimise' I disagree, loss or cost can be maximized in order to find the worst possible case (subject to whatever constraints). This can be useful for worst case analysis. Jun 10, 2017 at 22:40
• @Stephen mnemonic: loss=lonely (one data point), cost = comprehensive. Aug 11, 2018 at 9:58

Quoting from section 4.3 in "Deep Learning" book by Ian Goodfellow, Yoshua Bengio, Aaron Courville (emphasis in the original):

The function we want to minimize or maximize is called the objective function, or criterion. When we are minimizing it, we may also call it the cost function, loss function, or error function. In this book, we use these terms interchangeably, though some machine learning publications assign special meaning to some of these terms.

In this book at least, loss and cost are the same.

In Andrew NG's words-

"Finally, the loss function was defined with respect to a single training example. It measures how well you're doing on a single training example. I'm now going to define something called the cost function, which measures how well you're doing an entire training set. So the cost function J which is applied to your parameters W and B is going to be the average with one of the m of the sum of the loss function applied to each of the training examples and turn."

According to Prof. Andrew Ng (see slides on page 11),

Function h(X) represents your hypothesis. For fixed fitting parameters theta, it is a function of features X. I'd say this can also be called the Objective Function.

The Cost function J is a function of the fitting parameters theta. J = J(theta).

According to the Hastie et al.'s textbook "Elements of Statistical Learning", by p.37:

"We seek a function f (X) for predicting Y given values of the input X." [...] the loss function L(Y, f(X)) is "a function for penalizing the errors in prediction",

So it seems "loss function" is a slightly more general term than "cost function". If you seek for "loss" in that PDF, I think that they use "cost function" and "loss function" somewhat synonymously.

Indeed, p. 502

"The situation [in Clustering] is somewhat similar to the specification of a loss or cost function in prediction problems (supervised learning)".

Maybe these terms exist because they evolved independently in different academic communities. "Objective Function" is an old term used in Operations Research, and Engineering Mathematics. "Loss function" might be more in use among statisticians. But I'm speculating here.

• loss function is nowhere near being "more general" than cost function. f(X) is in particular the function of your parameters (thus J(theta)), making it (loss function) a particular type of cost function. Furthermore, Hastie has a simplification there, he assumes additive loss functions, which create a particular class of cost functions Oct 28, 2015 at 10:13
• I just tried to answer this question with references from the academic literature, sources which are easy to understand. Your point of "additive loss functions" might be right, but is well beyond the scope of the question asked, and I can't find this specific term in the ESL book
– knb
Oct 28, 2015 at 11:47
• Esl is a great book, but not the only source of Ml knowledge. Oct 28, 2015 at 11:48
• Is this "I'd say" from Ng or you? h is the model (h for hypothesis). The objective is that h performs well. The objective function measures how well h does and is usually different from h. Feb 1, 2018 at 17:58
• link to esl is broken Jan 6, 2019 at 8:10

The loss function computes the error for a single training example, while the cost function is the average of the loss functions of the entire training set.

• Look at Nick Cox's link. Sep 16, 2017 at 18:39

Actually to be simple If you have m training data like this (x(1),y(1)),(x(2),y(2)), . . . (x(m),y(m)) We use loss function L(ycap,y) to find loss between ycap and y of a single training set If we want to find loss between ycap and y of a whole training set we use cost function.

Note:- ycap means output from our model And y means expected output

Note:- Credit goes Andrew ng Resource: coursera neural network and deep learning

The terms cost and loss functions are synonymous. Some people also call them the error function. The more general scenario is to define an objective function first that we want to optimize. This objective function could be to:

1. maximize the posterior probabilities (e.g., naive Bayes)
2. maximize a fitness function (genetic programming)
3. maximize the total reward/value function (reinforcement learning)
4. maximize information gain/minimize child node impurities (CART decision tree classification)
5. minimize a mean squared error cost (or loss) function (CART, decision tree regression, linear regression, adaptive linear neurons, …
6. maximize log-likelihood or minimize cross-entropy loss (or cost) function minimize hinge loss (support vector machine)

To give you a short answer, according to me they are synonymous. However, the cost function is used more in optimization problem and loss function is used in parameter estimation.