Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I see that both functions are part of data mining methods such as Gradient Boosting Regressors. I see that those are separate objects too.

How is the relationship between both in general?

share|improve this question
up vote 26 down vote accepted
+50

A decision function is a function which takes a dataset as input and gives a decision as output. What the decision can be depends on the problem at hand. Examples include:

  • Estimation problems: the "decision" is the estimate.
  • Hypothesis testing problems: the decision is to reject or not reject the null hypothesis.
  • Classification problems: the decision is to classify a new observation (or observations) into a category.
  • Model selection problems: the decision is to chose one of the candidate models.

Typically, there are an infinite number of decision functions available for a problem. If we for instance are interested in estimating the height of Swedish males based on ten observations $\mathbf{x}=(x_1,x_2,\ldots,x_{10})$, we can use any of the following decision functions $d(\mathbf{x})$:

  • The sample mean: $d(\mathbf{x})=\frac{1}{10}\sum_{i=1}^{10}x_i$.
  • The median of the sample: $d(\mathbf{x})=\mbox{median}(\mathbf{x})$
  • The geometric mean of the sample: $d(\mathbf{x})=\sqrt[10]{x_1\cdots x_{10}}$
  • The function that always returns 1: $d(\mathbf{x})=1$, regardless of the value of $\mathbf{x}$. Silly, yes, but it is nevertheless a valid decision function.

How then can we determine which of these decision functions to use? One way is to use a loss function, which describes the loss (or cost) associated with all possible decisions. Different decision functions will tend to lead to different types of mistakes. The loss function tells us which type of mistakes we should be more concerned about. The best decision function is the function that yields the lowest expected loss. What is meant by expected loss depends on the setting (in particular, whether we are talking about frequentist or Bayesian statistics).

In summary:

  • Decision functions are used to make decisions based on data.
  • Loss functions are used to determine which decision function to use.
share|improve this answer
    
Ok, thanks for this answer! – Hiatus Jun 27 '14 at 11:25
    
For parametric decision functions (eg: logistic regression, threshold decision) you basically have one possible function for each parameter combination, and the loss function is used to find the best one. Common example: if you use gradient descent to explore the parameter space, you derive the loss with respect to the parameters and descend to a (local) minimum of the loss. – pixelou Jun 17 at 10:03

The loss function is what is minimized to obtain a model which is optimal in some sense. The model itself has a decision function which is used to predict.

For example, in SVM classifiers:

  • loss function: minimizes error and squared norm of the separating hyperplane $\mathcal{L}(\mathbf{w}, \xi) =\frac{1}{2}\|\mathbf{w}\|^2 + C\sum_i \xi_i$
  • decision function: signed distance to the separating hyperplane: $f(\mathbf{x}) = \mathbf{w}^T\mathbf{x} + b$
share|improve this answer
    
Isn´t the norm equal to the distance, or do i mix up something here... So the decision function is always a part of the loss function i use to "compare" to the real values i try to fix a model to? And the goal is minimizing this "difference"? – Hiatus Jun 27 '14 at 9:28
    
@Hiatus the norm of the separating hyperplane (which is being optimized when training an SVM) is not used in the decision function. The hyperplane itself is used. Minimizing the norm during training is basically a form of regularization. – Marc Claesen Jun 27 '14 at 9:30
    
It would be better to give a more generic answer that isn't tied to any specific classifier. – smci May 12 at 18:26

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.