I am trying to understand the idea of Loss functions For Regression Task perfectly.

I have read many textbooks and articles, and I came up with questions related to this subject.

Several different uses of loss functions can be distinguished.

(a) In prediction problems: a loss function depending on predicted and observed value defines the quality of a prediction.

(b) In estimation problems: a loss function depending on the true parameter and the estimated value defines the quality of estimation.

(c) Many estimators (such as least squares or M-estimators) are defined as optimizers of certain loss functions which then depend on the data and the estimated value.

Now, since my focus is on Loss Functions For Regression Task

($y_i=\theta_0 +\theta_1x_{i1}+\dots + \theta_px_{ip} +\epsilon_i$ ,y is the dependent variable and x is the independent one)

My questions are as follows.

  1. Should I write the loss function formula as a function of the parameter or of the variables ($\mathrm{L}\big(\theta,\hat\theta\big)$ or $\mathrm{L}\big(y,\hat y\big)$ )?

  2. Should I consider the Loss function formula for one point or Not (with sums or not)?


My thought is to introduce Loss function first and then to use the standard notation for all Loss functions (least square, absolute value and Huber Loss, Quntile Loss and so on).


I did the following but I am not sure

L2 Loss

$$ \mathrm{L}\big(\{y_{(i)}, \hat{y}_{(i)}\}_{i=1}^n\big) = \mathrm{\sum}_{i=1}^n\big(y_i-\hat y_i\big)^2;$$

L1 Loss

$$ \mathrm{L}\big(\{y_{(i)}, \hat{y}_{(i)}\}_{i=1}^n\big) = \mathrm{\sum}_{i=1}^n|y_i-\hat y_i|;$$

Huber Loss

$$ \mathrm{L}\big(\{y_{(i)}, \hat{y}_{(i)}\}_{i=1}^n\big) = \begin{cases} \frac{1}{2}\mathrm{\sum}_{i=1}^n(y_i-\hat y_i)^2 & |y_i-\hat y_i| \leq \delta \\ \delta\mathrm{\sum}_{i=1}^n|y_i-\hat y_i|-\frac{1}{2}\delta^2 & \text{otherwise} \end{cases} $$

Log-Cosh Loss

$$\mathrm{L}\big(\{y_{(i)}, \hat{y}_{(i)}\}_{i=1}^n\big)= \log[\cosh\mathrm{\sum}_{i=1}^n(y_i-\hat y_i)];$$

Quantile Loss

$$\mathrm{L}\big(\{y_{(i)}, \hat{y}_{(i)}\}_{i=1}^n\big) = \big[\tau \mathrm{\sum}_{i=1}^n|y_i-\hat y_i| + (1 + \tau)\mathrm{\sum}_{i=1}^n(y_i-\hat y_i)\big].$$

  • 1
    $\begingroup$ In case a) I would use $y$ and $\hat{y}$. Notice that if you have a parametrized model $f(x;\theta)$ then this loss function also depends on theta as it becomes $L(y,f(x;\theta))$. In the Second case I would go for $\theta$ because it is intertwined in my mind with the parameter of a density/distribution/... Concerning the second question: I tend to use $l$ (lower case) for a single example and $L$ for the sum. This may be subject to personal style though... $\endgroup$ – Fabian Werner Aug 15 at 11:22
  • $\begingroup$ @FabianWerner, what about case c? $\endgroup$ – jeza Aug 15 at 12:46
  • $\begingroup$ I do not really understand case c but I guess that you actually mean $L(y, f(x;\theta))$ which is case a basically... So my guess is that there is actually no difference between case a and c... (?) Could you elaborate more on c? $\endgroup$ – Fabian Werner Aug 15 at 16:02
  • 1
    $\begingroup$ Actually, there is no 'y', it is just a placeholder in order to make it clear how the loss function looks like. In reality, we are given pairs of input vectors $x_i$ with 'true answers' $y_i$, i.e. $(x_1, y_1), ..., (x_N, y_N)$. Now for a single $y$ (that might or might not be one of the $y_i$) you specify the loss function with a lowercase letter, for example $l(y, \hat{y}) = (y - \hat{y})^2$. For the whole vector $\mathbf{y} = (y_1, ..., y_N)$ and a vector of predictions produced by some model $\mathbf{\hat{y}} = (\hat{y}_1, ..., \hat{y}_N)$ you use ... $\endgroup$ – Fabian Werner Aug 17 at 22:26
  • 1
    $\begingroup$ $L(\mathbf{y}, \mathbf{\hat{y}}) = \sum_{i=1,...,N} l(y_i, \hat{y}_i)$ (maybe up to a normalizing factor) and you optimize the model (in the parameter $\theta$) by minimizing $L(\mathbf{y}, (f(x_i;\theta))_{i=1,...,N})$. $\endgroup$ – Fabian Werner Aug 17 at 22:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.