Skip to main content

Questions tagged [loss-functions]

A function used to quantify the difference between observed data and predicted values according to a model. Minimization of loss functions is a way to estimate the parameters of the model.

Filter by
Sorted by
Tagged with
1 vote
0 answers
20 views

Can a regularizer be reverse engineered to induce precise modifications to the associated unregularized regression problem's solution?

The following is a picture of regularization in a regression problem... The blue line is unregularized or less regularized, and the green line is more regularized. Problems of this sort are often ...
user10478's user avatar
  • 133
1 vote
0 answers
11 views

Is covariance estimation via the inverse hessian method generalizable (or possible) for loss functions other than least squares?

I know from other resources such as here that the scaled inverse hessian of your least squares loss can be used to estimate your model's parameter uncertainty (specifcally, covariance), but I can't ...
Will's user avatar
  • 11
0 votes
0 answers
15 views

difference between l2 penalty and l2 loss in SAE

I was reading this paper from Anthropic https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html and in the paper loss is defined like this :$$ L = \mathbb{E}_x \left[ \| x - \hat{x} \|...
Mrnobody's user avatar
1 vote
0 answers
22 views

Multi-task learning-Loss function

0 I am training a convolutional autoencoder with two velocity fields (2D array) as inputs and outputs. These fields represent wind velocities in both the x and y directions within a square domain. My ...
Sarah's user avatar
  • 11
0 votes
0 answers
23 views

Information coefficient as loss function of XGBoost

I am trying to train an XGBoost regressor for stock price prediction. I want to customize the objective function to be Information Coefficient (IC). The definition of IC is the Pearson correlation ...
atlantic0cean's user avatar
0 votes
0 answers
25 views

How to show that RMSE is more sensitive to outliers than the MAE?

I am reading this book where it states that for $\ell_p$ norms: The higher the norm index, the more it focuses on large values and neglects small ones. This is why the RMSE is more sensitive to ...
ado sar's user avatar
  • 477
3 votes
1 answer
38 views

Taking into account a non-symmetric loss function in a classification problem

Consider a binary classification method that estimates the class probability and where the observation weights can be specified (e.g. Logistic Regression). To accommodate the difference losses from TP ...
James's user avatar
  • 2,742
6 votes
1 answer
123 views

There are infinitely many proper scoring rules. Are they all equally valid? Or is log loss superior because of its connection to max likelihood?

I'm kind of obsessed with binary loss functions. How to create a (binary) loss function (scoring rule): Create a function, $f: [0,1] \rightarrow \mathbb{R}_{\geq 0} $, that is symmetric about $x=\...
Chinmay The Math Guy's user avatar
2 votes
1 answer
50 views

BerHu custom loss function for XGBoost

I would like a loss function that penalizes outliers like the squared loss, while treating small errors less sharply, like the absolute loss. It seems that I am looking for a Huber loss function, but ...
Mr. Ivan's user avatar
0 votes
0 answers
17 views

Is it a good idea to incorporate a feature into the loss function when training a neural network model that does regression?

When training a neural network that does regression, assuming I have 3 features called "a", "b", and "c". The corresponding target is called "d". In theory, ...
sensationti's user avatar
3 votes
2 answers
158 views

Estimating Smooth Density Field from Limited Sampled Data

I want to estimate a “density field”, specifically $P(y|x, m)$, for binary labels $y$ associated with 2D points characterized by spatial coordinates $m$ and additional spatio-temporal features $x$. ...
Xaume's user avatar
  • 81
0 votes
0 answers
17 views

Cost function for time series anomaly detection with limited labelled anomalies

Given a time series $y_1, \dots, y_n$, I will fit some models to the data and I want to choose one for anomaly identification. I'm interested in a cost function that rewards a model whose fitted ...
Alex's user avatar
  • 722
5 votes
2 answers
363 views

Expected loss function from bias variance trade off (integral help)

I have a hard time understanding this formula. It's from bias-variance trade-off proof. and the expected loss function is as follows: $$L(\hat f) := \mathbb E_D\mathbb E_{(x,y)}[(y-\hat f(x))^2]=\...
Taewooo Kim's user avatar
0 votes
0 answers
34 views

About the hinge loss and slack variables

I'll be denoting the $ith$ training example, target label and slack variable as $\mathbf{\vec x}^{(i)}$, $y^{(i)}$ and $\xi_i$ respectively. Hinge Loss : The hinge loss function in the context of ...
Sagnik Taraphdar's user avatar
1 vote
1 answer
35 views

Accuracy "overfits" but loss doesn't?

I'm perplexed as to why my loss doesn't go up when the accuracy goes down (after about 40 epochs). Isn't it possible to tell overfitting from the loss curve alone? (I'm of course referring the ...
Tfovid's user avatar
  • 795
0 votes
0 answers
7 views

Computing Test Loss in Kernel Ridge Regression

In Kernel Ridge regression we have the standard loss function $$L(\beta) = \|Y-K\beta\|_2^2 + \alpha \beta^T K \beta$$ Here, $K$ is the kernel (gram) matrix. If I compute $\beta$ on a training set, so ...
WeakLearner's user avatar
  • 1,501
0 votes
0 answers
20 views

Optimizing parameters for a non-standard probability density function

We have a non-standard multivariate probability density function, P(x | q), where x is a vector, and q are the parameters of the density. I get events ...
Niteya Shah's user avatar
2 votes
1 answer
53 views

On using the loss as a metric?

The context is model evaluation in supervised learning. I am coming from a numerical optimisation background. For me it is quite natural to use the loss of the model (what we optimise during training) ...
Lucas Morin's user avatar
  • 1,665
0 votes
0 answers
68 views

Huber-Loss optimisation using Stochastic Gradient Descent to estimate intercept and coefficient of regression line

What: I'm trying to minimise the Huber-Loss for a linear regression using Stochastic Gradient Descent from scratch. Problem: It seems like that the coeffcient $m$ doesn't get optimised, therefore the ...
Corbjn's user avatar
  • 111
1 vote
1 answer
42 views

What happens if I use a single scalar output with MSE Loss for classification tasks? [closed]

Rather than using Cross Entropy Loss and one hot encoding for neural network classification tasks, if my model outputs a single scalar value and I use mean squared error loss what will happen?
Fine-Tuning's user avatar
2 votes
1 answer
141 views

Which form of cross-entropy loss is correct?

For classification problems with more than two classes, I've seen these two forms of cross-entropy loss: -$\sum_k y_k \log(a_k)$ -$\sum_k y_k \log(a_k) + (1-y_k) \log(1-a_k)$ Here $y_i$ are the true ...
theQman's user avatar
  • 677
0 votes
0 answers
9 views

Choosing Distortion Measures for Decision Rules with Logarithmic Posteriors

I've been delving into Bayesian decision theory and specifically looking at scenarios where we work with the logarithm of the posterior distribution (log-posterior). My understanding is that in such ...
Alireza's user avatar
  • 113
2 votes
1 answer
52 views

Loss function for volatility forecasts from GARCH

What are the options for loss functions, when trying to compare the volatility (sigma) forecasts from different GARCH models? I was thinking about the Qlike function but am not sure if this would give ...
statwoman's user avatar
  • 703
3 votes
1 answer
110 views

Why is the regularization term multiplied by the error term in the cost function of SVM?

The cost function of the Optimal Margin Classifier(non-kernelized SVM) is given as : $$ J(\mathbf{\vec w}, b) = \frac{1}{2}\|\mathbf{\vec w}\|_{2}^{2} + C \sum_{i=1}^{n}\max(0, 1-y ^{(i)}(\mathbf{\vec ...
Sagnik Taraphdar's user avatar
0 votes
0 answers
56 views

What are the benefits of using pseudo-residuals in Gradient Boosting?

At each iteration $t$ of the Gradient Boosting algorithm, we're basically trying to add the weak learner $f_t$ that minimizes: $$ \mathcal{L}_t = \sum\limits_{i=1}^{n} l(y_i, \hat{y}_i^{(t-1)} + f_t(\...
Druudik's user avatar
  • 143
0 votes
0 answers
15 views

Increasing the clarity in the tasks of image generation using CNN

What methods exist to improve the quality of generated images and the clarity of contours in the tasks of image denoising/debluring (using CNN), style transfer etc? I am interested in approaches that ...
Alimagadov K.'s user avatar
0 votes
0 answers
44 views

How to penalize disagreement between two classification loses?

I am working with a multi-head, multi-loss neural network. Each of the two heads is associated with a multi-class classification loss. The losses are combined additively. Assume loss 1 is trained to ...
Gertrude Porter's user avatar
4 votes
4 answers
154 views

Why do we work with factor of likelihoods instead of e.g. a sum for a batch in the negative log likelihood loss function?

In a classification task, at a certain stage of the training process, we get a likelihood of sampling proper class Y for a particular data point X. For batch, we get many independent likelihoods. Let'...
Maciek Gruszczyński's user avatar
3 votes
1 answer
179 views

Scenario where minimizing 0-1 loss is different than minimizing hinge loss

Suppose we're using linear predictors. I'm trying to conceptually understand how minimizing hinge loss and 0-1 loss aren't necessarily the same. For instance I was told that one can choose a set of ...
redbull_nowings's user avatar
2 votes
1 answer
36 views

Training loss reach to zero, then suddenly increases, then decreases to zero

I get the following loss behavior when training multilayer perceptron with mean squared error loss on some synthetic data using default Adam with default learning. (I am working on 1 demention data) I ...
Rahim Brahimi's user avatar
0 votes
1 answer
36 views

Is there a (lower) limit/minimum for learning rate values?

I'm building a model for traffic prediction with ConvLSTM and A3T-GCN cells. Since the input data is highly complex and the model is relatively big, I can only load ...
olenscki's user avatar
  • 101
7 votes
2 answers
128 views

Real-world example of quantile loss used for evaluation

We can use quantile loss (a.ka. tick or pinball loss) for training a model or for evaluating predictions. (It is helpful to distinguish the two clearly, e.g. as done here.) I am interested in the ...
Richard Hardy's user avatar
2 votes
0 answers
40 views

Solving a system of equalities using a neural network

Assume $P$ is a set of pairs $(x, y)$, where both $x$ and $y$ are in $\mathbb{R}^n$. Assume $P'$ is a subset of $P$. I want to train a neural network $N: \mathbb{R}^n \to \mathbb{R}^m$ such that, for ...
Mahyar's user avatar
  • 21
0 votes
0 answers
30 views

Forcing NN to have fixed or identity output in a region of state space

I have a transition system defined over state space $X$, with a transition function $f: X \to X$. Let us assume my task is to learn a function $G: X\to \mathbb{R}$ such that $G$ is decreasing, i.e. $G(...
Mahyar's user avatar
  • 21
0 votes
0 answers
37 views

How does the chain-rule look for the gradient of a loss function?

When we are computing the gradient of the loss function, $L$, of a Word2Vec model, for the context word-embedding, $w_i$, and the target word-embedding, $t$. Where the loss function, $L$, looks like: $...
ZenPyro's user avatar
1 vote
0 answers
64 views

Modification of square loss analogous to absolute and vs pinball loss: what is elicited?

Quantile regression at quantile $\tau$ minimizes the following "pinball" loss function, $L_{\tau}$, and elicits conditional quantile $\tau$. $$ l_{\tau}(y_i, \hat y_i) = \begin{cases} \...
Dave's user avatar
  • 65k
0 votes
0 answers
102 views

Exponentially Weighted Covariance Matrix with Ledoit Wolf Shrinkage

The Ledoit Wolf paper "Honey, I Shrunk the Sample Covariance Matrix" presents the formulation for the shrinkage intensity parameter estimate in Appendix B. The formula for a weighted ...
nka5we's user avatar
  • 49
1 vote
0 answers
130 views

Theoretically optimal "loss" function for continuous distributions? [closed]

Classification problems have the benefit of being discrete, so it's easy to calculate how much information your model has. For example, in LLMs your training data has inputs $\mathbf{x}_i$, a list of ...
programjames's user avatar
0 votes
1 answer
24 views

What loss function should I use to fit a distribution of points with a function with latent variables?

For simplicity I am going start with a toy example. Lets suppose we have a set of $n$ points $\vec{Y}$ in the 2d space, distributed with the shape of the letter M. ...
Iván's user avatar
  • 101
3 votes
2 answers
159 views

Support Vector machine - Hingeloss

What does it mean that 'The SVM hinge loss estimates the mode of the posterior class probabilities'(Elements of statistical Learning p.427). The decision function f(x) assigns to the positive class(+...
J.doe's user avatar
  • 359
0 votes
0 answers
19 views

How to split the data for speaker verification using AAM Softmax loss?

There are several models for speaker verificaiton task (wavlm-ecapa / xvector / ...). Some of those model where trained with AAM Softmax loss, which gets the number of labels as input. When training ...
user3668129's user avatar
1 vote
0 answers
41 views

Derivations of loss functions for learning loss attenuation in Bayesian DL

I'm fairly new to Bayesian deep learning, so sorry if this is a silly question. I'm trying to implement the work in this paper: What Uncertainties Do We Need in Bayesian Deep Learning for Computer ...
suyash0's user avatar
  • 11
3 votes
1 answer
134 views

is there hidden cost function for hidden layer in the neural network?

In the case of a neural network,are there different cost functions for different hidden layers? or is there one cost function for the final layer ? For example, in the neural network, the hidden layer ...
farhana hossain's user avatar
0 votes
0 answers
64 views

Instance segmentation using a discriminative loss?

I have been reading this paper and I was wondering if their discriminitive loss definition is correct for instance segmentation ? From what I understand they map the image pixels into a higher ...
KFkf's user avatar
  • 111
2 votes
1 answer
76 views

Calculation of the Generalized Dice Loss Gradient

I am trying to understand the gradient of the Generalized Dice Loss (GDL) shown here Link. It says that the GDL for two classes is: $$ GDL = 1 - 2 \frac{\sum_{l=1}^2w_l \sum_{n=1}^{N} r_{ln}p_{ln}}{\...
sephlink's user avatar
6 votes
2 answers
842 views

How to incentivise AI to make risky predictions

I'm trying to build a weather forecasting AI. I have a dataset that contains the peak temperatures for each day. I have trained it with MSE as the loss function and it has worked fairly well. I do ...
n-l-i's user avatar
  • 193
1 vote
1 answer
51 views

Does the matrix notation for OLS loss function assume that indexing a row from $X$ return a column or row vecotr?

I'm confused about the matrix notation of the loss function in ordinary least squares regression. In matrix form, the expression for a linear model is: $$\hat{y} = Xw$$ Where $\hat{y} \rightarrow$ ...
another_student's user avatar
7 votes
1 answer
520 views

Ordinal log-loss in a multiclass classification in XGBoost?

I have a multi-class problem that which classes are simultaneously mutually exclusive and have ordering. You can think of the classes as being some score: 0 (Low), 1 (Medium), 2 (High). What I would ...
deblue's user avatar
  • 243
1 vote
1 answer
76 views

Is model distillation an ill-defined problem?

Model distillation (or knowledge distillation) consist in making a student model learn from a teacher model in order to eventually use the student model as an alternative to the original teacher model....
Ben's user avatar
  • 11
0 votes
0 answers
20 views

cGAN: Discriminator loss going to zero while Generator's going always up but the result is very good

I have a Conditional Generative Adversarial Network for Quantum State Tomography. The metrics I am monitoring during the training process are the losses and the Fidelity (the degree of similarity ...
Dimitri's user avatar
  • 15

1
2 3 4 5
24