What is the difference between a loss function and decision function?

Question

I see that both functions are part of data mining methods such as Gradient Boosting Regressors. I see that those are separate objects too.

How is the relationship between both in general?

Answer 1

A decision function is a function which takes a dataset as input and gives a decision as output. What the decision can be depends on the problem at hand. Examples include:

• Estimation problems: the "decision" is the estimate.
• Hypothesis testing problems: the decision is to reject or not reject the null hypothesis.
• Classification problems: the decision is to classify a new observation (or observations) into a category.
• Model selection problems: the decision is to chose one of the candidate models.

Typically, there are an infinite number of decision functions available for a problem. If we for instance are interested in estimating the height of Swedish males based on ten observations $\mathbf{x}=(x_1,x_2,\ldots,x_{10})$, we can use any of the following decision functions $d(\mathbf{x})$:

• The sample mean: $d(\mathbf{x})=\frac{1}{10}\sum_{i=1}^{10}x_i$.
• The median of the sample: $d(\mathbf{x})=\mbox{median}(\mathbf{x})$
• The geometric mean of the sample: $d(\mathbf{x})=\sqrt[10]{x_1\cdots x_{10}}$
• The function that always returns 1: $d(\mathbf{x})=1$, regardless of the value of $\mathbf{x}$. Silly, yes, but it is nevertheless a valid decision function.

How then can we determine which of these decision functions to use? One way is to use a loss function, which describes the loss (or cost) associated with all possible decisions. Different decision functions will tend to lead to different types of mistakes. The loss function tells us which type of mistakes we should be more concerned about. The best decision function is the function that yields the lowest expected loss. What is meant by expected loss depends on the setting (in particular, whether we are talking about frequentist or Bayesian statistics).

In summary:

• Decision functions are used to make decisions based on data.
• Loss functions are used to determine which decision function to use.

Answer 2

The loss function is what is minimized to obtain a model which is optimal in some sense. The model itself has a decision function which is used to predict.

For example, in SVM classifiers:

• loss function: minimizes error and squared norm of the separating hyperplane $\mathcal{L}(\mathbf{w}, \xi) =\frac{1}{2}\|\mathbf{w}\|^2 + C\sum_i \xi_i$
• decision function: signed distance to the separating hyperplane: $f(\mathbf{x}) = \mathbf{w}^T\mathbf{x} + b$