People uses these 4 terms when talking about deep learning, machine learning, data science.

error (such as training error, test error), cost (such as training cost, test cost), loss (such as empirical loss), risk (such as empirical risk)

Are those 4 terms the same in the context of machine learning?

Answers with authoritative reference would be better, such as published paper, text book, user guide or documentation.


2 Answers 2

  • Cost and loss are often used interchangeably. However, sometimes people deliberately use the term loss to refer to a single training example, and cost is used in the context of the whole training set. For instance, for the $i$th training example, you may have a loss function $\mathcal{L}(x_i, y_i)$, and you may write the cost as $\frac{1}{n} \sum_{i}^{n}\mathcal{L}(x_i, y_i)$.

  • Cost and error can be the same, depending what your loss function is. For example, say you have a logistic regression classifier, the classification cost and classification error are two different things. Here, you would define the cost via the negative log-likelihood loss, and the error is based on the 0/1 loss. The 0/1 loss is intuitive but not differentiable. In other cases, like OLS regression, you minimize the mean-squared error and can also report it as the training error. So in that case they are the same

  • The "risk" is usually the expected loss. However, since that's hard to compute in most cases, people use the term "empirical risk" which is basically again just averaging over the losses (so, basically similar as cost)

  • $\begingroup$ Thanks for your answer. Would you please give some authoritative reference, such as published paper, text book, user guide or documentation? $\endgroup$
    – JJJohn
    Aug 29, 2019 at 15:31
  • $\begingroup$ Not sure if it exists, and I would have to search for that ... However, this is based on years of reading ML literature. You can basically just use Google Scholar and search for each term within the ML literature and see that these terms are exactly used the way I described. $\endgroup$
    – resnet
    Aug 29, 2019 at 17:40

Loss: Discrepancy between the response of superviser and learning machine for a given input x (Vapnik, 2000, p. 19). Different for regression, classficiation..

risk functional: Probability of "error" of the learning machine (Vapnik, 2000, p. 20). As described by @resnet this is replaced by empirical risk empirical risk functional, i.e., the frequency of "error" (Vapnik, 2000, p. 20).

Error: Usually, it is used to describe the error of your ML algorithm (train error, test error, and so on). Vapnik seems to be a bit inconsistent with usual usage of error. However, error is also not a very technical term. Note: Empirical risk is only the test error (not train error).

Cost is used in ML terminology to describe the cost for miss-classification of an example (e.g., soft-margin SVM) - compared to the value that is added to the loss function from the regularization term. The "cost" parameter allows to balance between higher regularization error vs. miss-classification "cost" of the complete error term. The cost parameter is a very SVM specific term. For other ML algorithms it is often called the regularization parameter.

Note: Cost function is not very specifically defined in literature see @resnet comment below. @resnet: "The term cost function is more general. It's like objective function but defined for minimization (whereas the objective function can either be defined as an maximization, in case of reward, or minimization)"

  • $\begingroup$ Regarding cost; the Vapnik usage of that term is very specific to SVMs in the context he uses it, but it is generally used differently. For instance, in the paper "LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436." (collaboratively written by some of the most recognized experts in the field), the term "cost function" is used synonymously with "loss function", which they use in some other of their papers. $\endgroup$
    – resnet
    Aug 29, 2019 at 17:46
  • $\begingroup$ I agree I will slightly modify it $\endgroup$ Aug 29, 2019 at 18:02
  • $\begingroup$ Yeah, there is the distinction between a cost parameter of a particular algorithm and a cost function to measure model performance. $\endgroup$
    – resnet
    Aug 29, 2019 at 18:03
  • $\begingroup$ Are you sure that cost function does not describe the error term that is not the regularization term? $\endgroup$ Aug 29, 2019 at 18:04
  • $\begingroup$ In LeCun 2015 it could be interpreted as that? $\endgroup$ Aug 29, 2019 at 18:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.