# What is the difference between a loss function and decision function?

I see that both functions are part of data mining methods such as Gradient Boosting Regressors. I see that those are separate objects too.

How is the relationship between both in general?

A decision function is a function which takes a dataset as input and gives a decision as output. What the decision can be depends on the problem at hand. Examples include:

• Estimation problems: the "decision" is the estimate.
• Hypothesis testing problems: the decision is to reject or not reject the null hypothesis.
• Classification problems: the decision is to classify a new observation (or observations) into a category.
• Model selection problems: the decision is to chose one of the candidate models.

Typically, there are an infinite number of decision functions available for a problem. If we for instance are interested in estimating the height of Swedish males based on ten observations $\mathbf{x}=(x_1,x_2,\ldots,x_{10})$, we can use any of the following decision functions $d(\mathbf{x})$:

• The sample mean: $d(\mathbf{x})=\frac{1}{10}\sum_{i=1}^{10}x_i$.
• The median of the sample: $d(\mathbf{x})=\mbox{median}(\mathbf{x})$
• The geometric mean of the sample: $d(\mathbf{x})=\sqrt[10]{x_1\cdots x_{10}}$
• The function that always returns 1: $d(\mathbf{x})=1$, regardless of the value of $\mathbf{x}$. Silly, yes, but it is nevertheless a valid decision function.

How then can we determine which of these decision functions to use? One way is to use a loss function, which describes the loss (or cost) associated with all possible decisions. Different decision functions will tend to lead to different types of mistakes. The loss function tells us which type of mistakes we should be more concerned about. The best decision function is the function that yields the lowest expected loss. What is meant by expected loss depends on the setting (in particular, whether we are talking about frequentist or Bayesian statistics).

In summary:

• Decision functions are used to make decisions based on data.
• Loss functions are used to determine which decision function to use.
• For parametric decision functions (eg: logistic regression, threshold decision) you basically have one possible function for each parameter combination, and the loss function is used to find the best one. Common example: if you use gradient descent to explore the parameter space, you derive the loss with respect to the parameters and descend to a (local) minimum of the loss. Commented Jun 17, 2016 at 10:03

The loss function is what is minimized to obtain a model which is optimal in some sense. The model itself has a decision function which is used to predict.

For example, in SVM classifiers:

• loss function: minimizes error and squared norm of the separating hyperplane $\mathcal{L}(\mathbf{w}, \xi) =\frac{1}{2}\|\mathbf{w}\|^2 + C\sum_i \xi_i$
• decision function: signed distance to the separating hyperplane: $f(\mathbf{x}) = \mathbf{w}^T\mathbf{x} + b$
• Isn´t the norm equal to the distance, or do i mix up something here... So the decision function is always a part of the loss function i use to "compare" to the real values i try to fix a model to? And the goal is minimizing this "difference"? Commented Jun 27, 2014 at 9:28
• @Hiatus the norm of the separating hyperplane (which is being optimized when training an SVM) is not used in the decision function. The hyperplane itself is used. Minimizing the norm during training is basically a form of regularization. Commented Jun 27, 2014 at 9:30
• It would be better to give a more generic answer that isn't tied to any specific classifier.
– smci
Commented May 12, 2016 at 18:26