# Questions tagged [loss-functions]

A function used to quantify the difference between observed data and predicted values according to a model. Minimization of loss functions is a way to estimate the parameters of the model.

1,170 questions
Filter by
Sorted by
Tagged with
1 vote
20 views

### Can a regularizer be reverse engineered to induce precise modifications to the associated unregularized regression problem's solution?

The following is a picture of regularization in a regression problem... The blue line is unregularized or less regularized, and the green line is more regularized. Problems of this sort are often ...
• 133
1 vote
11 views

### Is covariance estimation via the inverse hessian method generalizable (or possible) for loss functions other than least squares?

I know from other resources such as here that the scaled inverse hessian of your least squares loss can be used to estimate your model's parameter uncertainty (specifcally, covariance), but I can't ...
• 11
15 views

34 views

### About the hinge loss and slack variables

I'll be denoting the $ith$ training example, target label and slack variable as $\mathbf{\vec x}^{(i)}$, $y^{(i)}$ and $\xi_i$ respectively. Hinge Loss : The hinge loss function in the context of ...
1 vote
35 views

### Accuracy "overfits" but loss doesn't?

I'm perplexed as to why my loss doesn't go up when the accuracy goes down (after about 40 epochs). Isn't it possible to tell overfitting from the loss curve alone? (I'm of course referring the ...
• 795
7 views

### Computing Test Loss in Kernel Ridge Regression

In Kernel Ridge regression we have the standard loss function $$L(\beta) = \|Y-K\beta\|_2^2 + \alpha \beta^T K \beta$$ Here, $K$ is the kernel (gram) matrix. If I compute $\beta$ on a training set, so ...
• 1,501
20 views

### Optimizing parameters for a non-standard probability density function

We have a non-standard multivariate probability density function, P(x | q), where x is a vector, and q are the parameters of the density. I get events ...
• 101
53 views

### On using the loss as a metric?

The context is model evaluation in supervised learning. I am coming from a numerical optimisation background. For me it is quite natural to use the loss of the model (what we optimise during training) ...
• 1,665
68 views

### Huber-Loss optimisation using Stochastic Gradient Descent to estimate intercept and coefficient of regression line

What: I'm trying to minimise the Huber-Loss for a linear regression using Stochastic Gradient Descent from scratch. Problem: It seems like that the coeffcient $m$ doesn't get optimised, therefore the ...
• 111
1 vote
42 views

### What happens if I use a single scalar output with MSE Loss for classification tasks? [closed]

Rather than using Cross Entropy Loss and one hot encoding for neural network classification tasks, if my model outputs a single scalar value and I use mean squared error loss what will happen?
• 236
141 views

### Which form of cross-entropy loss is correct?

For classification problems with more than two classes, I've seen these two forms of cross-entropy loss: -$\sum_k y_k \log(a_k)$ -$\sum_k y_k \log(a_k) + (1-y_k) \log(1-a_k)$ Here $y_i$ are the true ...
• 677
9 views

### Choosing Distortion Measures for Decision Rules with Logarithmic Posteriors

I've been delving into Bayesian decision theory and specifically looking at scenarios where we work with the logarithm of the posterior distribution (log-posterior). My understanding is that in such ...
• 113
52 views

### Loss function for volatility forecasts from GARCH

What are the options for loss functions, when trying to compare the volatility (sigma) forecasts from different GARCH models? I was thinking about the Qlike function but am not sure if this would give ...
• 703
110 views

• 143
15 views

### Increasing the clarity in the tasks of image generation using CNN

What methods exist to improve the quality of generated images and the clarity of contours in the tasks of image denoising/debluring (using CNN), style transfer etc? I am interested in approaches that ...
44 views

### How to penalize disagreement between two classification loses?

I am working with a multi-head, multi-loss neural network. Each of the two heads is associated with a multi-class classification loss. The losses are combined additively. Assume loss 1 is trained to ...
154 views

### Why do we work with factor of likelihoods instead of e.g. a sum for a batch in the negative log likelihood loss function?

In a classification task, at a certain stage of the training process, we get a likelihood of sampling proper class Y for a particular data point X. For batch, we get many independent likelihoods. Let'...
179 views

### Scenario where minimizing 0-1 loss is different than minimizing hinge loss

Suppose we're using linear predictors. I'm trying to conceptually understand how minimizing hinge loss and 0-1 loss aren't necessarily the same. For instance I was told that one can choose a set of ...
36 views

### Training loss reach to zero, then suddenly increases, then decreases to zero

I get the following loss behavior when training multilayer perceptron with mean squared error loss on some synthetic data using default Adam with default learning. (I am working on 1 demention data) I ...
36 views

### Is there a (lower) limit/minimum for learning rate values?

I'm building a model for traffic prediction with ConvLSTM and A3T-GCN cells. Since the input data is highly complex and the model is relatively big, I can only load ...
• 101
128 views

### Real-world example of quantile loss used for evaluation

We can use quantile loss (a.ka. tick or pinball loss) for training a model or for evaluating predictions. (It is helpful to distinguish the two clearly, e.g. as done here.) I am interested in the ...
• 68.6k
40 views

### Solving a system of equalities using a neural network

Assume $P$ is a set of pairs $(x, y)$, where both $x$ and $y$ are in $\mathbb{R}^n$. Assume $P'$ is a subset of $P$. I want to train a neural network $N: \mathbb{R}^n \to \mathbb{R}^m$ such that, for ...
• 21
30 views

1 vote
64 views

• 21
842 views

### How to incentivise AI to make risky predictions

I'm trying to build a weather forecasting AI. I have a dataset that contains the peak temperatures for each day. I have trained it with MSE as the loss function and it has worked fairly well. I do ...
• 193
1 vote
51 views

### Does the matrix notation for OLS loss function assume that indexing a row from $X$ return a column or row vecotr?

I'm confused about the matrix notation of the loss function in ordinary least squares regression. In matrix form, the expression for a linear model is: $$\hat{y} = Xw$$ Where $\hat{y} \rightarrow$ ...
520 views

### Ordinal log-loss in a multiclass classification in XGBoost?

I have a multi-class problem that which classes are simultaneously mutually exclusive and have ordering. You can think of the classes as being some score: 0 (Low), 1 (Medium), 2 (High). What I would ...
• 243
1 vote
76 views

### Is model distillation an ill-defined problem?

Model distillation (or knowledge distillation) consist in making a student model learn from a teacher model in order to eventually use the student model as an alternative to the original teacher model....
• 11