Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I see that both functions are part of data mining methods such as Gradient Boosting Regressors. I see that those are seperate objects. But how is the relationship between both in general?

share|improve this question
up vote 14 down vote accepted

A decision function is a function which takes a dataset as input and gives a decision as output. What the decision can be depends on the problem at hand. Examples include:

  • Estimation problems: the "decision" is the estimate.
  • Hypothesis testing problems: the decision is to reject or not reject the null hypothesis.
  • Classification problems: the decision is to classify a new observation (or observations) into a category.
  • Model selection problems: the decision is to chose one of the candidate models.

Typically, there are an infinite number of decision functions available for a problem. If we for instance are interested in estimating the average height of Swedish males based on ten observations $\mathbf{x}=(x_1,x_2,\ldots,x_n)$, we can use any of the following decision functions $d(\mathbf{x})$:

  • The sample mean: $d(\mathbf{x})=\frac{1}{10}\sum_{i=1}^{10}x_i$.
  • The median of the sample: $d(\mathbf{x})=\mbox{median}(\mathbf{x})$
  • The geometric mean of the sample: $d(\mathbf{x})=\sqrt[10]{x_1\cdots x_{10}}$
  • The function that always returns 1: $d(\mathbf{x})=1$, regardless of the value of $\mathbf{x}$. Silly, yes, but it is nevertheless a valid decision function.

How then can we determine which of these decision functions to use? One way is to use a loss function, which describes the loss (or cost) associated with all possible decisions. Different decision functions will tend to lead to different types of mistakes. The loss function tells us which type of mistakes we should be more concerned about. The best decision function is the function that yields the lowest expected loss. What is meant by expected loss depends on the setting (in particular, whether we are talking about frequentist or Bayesian statistics).

In summary:

  • Decision functions are used to make decisions based on data.
  • Loss functions are used to determine which decision function to use.
share|improve this answer
Ok, thanks for this answer! – Hiatus Jun 27 '14 at 11:25

The loss function is what is minimized to obtain a model which is optimal in some sense. The model itself has a decision function which is used to predict.

For example, in SVM classifiers:

  • loss function: minimizes error and squared norm of the separating hyperplane $\mathcal{L}(\mathbf{w}, \xi) =\frac{1}{2}\|\mathbf{w}\|^2 + C\sum_i \xi_i$
  • decision function: signed distance to the separating hyperplane: $f(\mathbf{x}) = \mathbf{w}^T\mathbf{x} + b$
share|improve this answer
Isn´t the norm equal to the distance, or do i mix up something here... So the decision function is always a part of the loss function i use to "compare" to the real values i try to fix a model to? And the goal is minimizing this "difference"? – Hiatus Jun 27 '14 at 9:28
@Hiatus the norm of the separating hyperplane (which is being optimized when training an SVM) is not used in the decision function. The hyperplane itself is used. Minimizing the norm during training is basically a form of regularization. – Marc Claesen Jun 27 '14 at 9:30
It would be better to give a more generic answer that isn't tied to any specific classifier. – smci May 12 at 18:26

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.