All Questions
Tagged with cost-function or loss-functions
1,174 questions
3
votes
1
answer
120
views
Custom Loss function Overfits to the Wrong Output but MSE Doesn't
I have a simple function that I want to approximate with a neural network:
N(1) = -1
N(2) = -1
N(3) = 1
N(4) = -1
Instead of using the MSE or cross-entropy losses, ...
- 33
4
votes
2
answers
63
views
Smooth Thresholding a Loss Function
Suppose we have an objective function with a fixed "threshold" $\delta > 0$, for example
$$ L(y_i, \hat{y}_i) = \begin{cases} (y_i - \hat{y}_i)^2 & if \ |\epsilon_i|:= |y_i - \hat{y}...
- 478
3
votes
1
answer
406
views
Why is "unbiased" estimator more important than min-error estimator?
According to Edwin Jaynes (Chapter 17 of his book Probability Theory: the Logic of Science), the mean squared error of an estimator consists of bias term and variance term, that is:
$$L =E[(\beta - \...
- 471
0
votes
0
answers
21
views
Low initial validation loss from the first epoch?
The initial validation loss is low from the first epoch and then decreases slightly. What does this actually mean? Does it indicate that the model can effectively and quickly identify patterns for ...
- 101
1
vote
0
answers
41
views
Loss function - computed on one data point or on all data points?
I'm trying to read Jerome H. Friedman's lecture on Gradient Boosting Decision Trees. Unfortunately I have very rudimentary understanding of statistics and mathematics and I am confused, so I hope ...
- 11
0
votes
0
answers
8
views
How do one test for absence of performance drop?
I'll try to keep this general... Say I calibrate a Supervised ML model and observe some performance locally (in and out of sample). Then I put the model into production and observe some 'live' ...
- 1,665
0
votes
0
answers
19
views
Recursive Random Search and Categorical Cost Functions
I'm currently working on a project that involves optimizing the default Spark-submit configurations to minimize execution time. I've developed two models to aid in this process:
Binary Classification ...
- 175
1
vote
0
answers
19
views
Best loss function for predicting relative speed of two (or more) options
I want to create a model that tries to predict which execution engine will be faster for a given user query. The model does not have enough information to predict the actual run time, but we want to ...
- 141
0
votes
0
answers
14
views
Loss functions for target values very close to 0 [duplicate]
I am currently building a regression MLP model to predict a target variable between [0,3]. The distribution for target variable is normal for the most part with a slight left skew. My model is ...
- 1
0
votes
0
answers
54
views
Maximum Likelihood Estimation with Gradient Descent and Squarred Loss
My goal is to learn parameters $\mu$ and $\sigma$ of a univariate Gaussian distribution using gradient descent to validate my understanding of the algorithm by deriving all the formulas from scratch. ...
- 1
0
votes
0
answers
53
views
Should the target be standardized in gradient descent?
Suppose that we have a general loss function that depends on some parameters $w$ (e.g. neural network weights):
$$L_w =\frac{1}{N} \sum_i \ell(\hat{y}_i, y_i)$$
Is it beneficial to standardize the ...
- 621
1
vote
1
answer
30
views
Loss functions for unsupervised clustering of gaussian data
I got data that was generated from $n$ multivariate gaussians with different means and covariances, which means that they are separateble, and I'm tasked with classifying them using neural network in ...
- 157
3
votes
2
answers
155
views
Beta distribution loss function [closed]
I'm trying to build a model to predict the ratio of two numbers. The distribution of this ratio takes the form of a beta distribution, between 0 and 1. So far, I've been using different existing loss ...
- 33
1
vote
0
answers
30
views
Can a regularizer be reverse engineered to induce precise modifications to the associated unregularized regression problem's solution?
The following is a picture of regularization in a regression problem...
The blue line is unregularized or less regularized, and the green line is more regularized. Problems of this sort are often ...
- 133
1
vote
0
answers
39
views
Is covariance estimation via the inverse hessian method generalizable (or possible) for loss functions other than least squares?
I know from other resources such as here that the scaled inverse hessian of your least squares loss can be used to estimate your model's parameter uncertainty (specifcally, covariance), but I can't ...
- 11
0
votes
0
answers
23
views
difference between l2 penalty and l2 loss in SAE
I was reading this paper from Anthropic https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html and in the paper loss is defined like this :$$
L = \mathbb{E}_x \left[ \| x - \hat{x} \|...
- 3
1
vote
0
answers
36
views
Multi-task learning-Loss function
0
I am training a convolutional autoencoder with two velocity fields (2D array) as inputs and outputs. These fields represent wind velocities in both the x and y directions within a square domain. My ...
- 11
0
votes
0
answers
41
views
Information coefficient as loss function of XGBoost
I am trying to train an XGBoost regressor for stock price prediction. I want to customize the objective function to be Information Coefficient (IC). The definition of IC is the Pearson correlation ...
- 101
0
votes
0
answers
62
views
How to show that RMSE is more sensitive to outliers than the MAE?
I am reading this book where it states that for $\ell_p$ norms:
The higher the norm index, the more it focuses on large values and
neglects small ones. This is why the RMSE is more sensitive to
...
- 621
3
votes
1
answer
55
views
Taking into account a non-symmetric loss function in a classification problem
Consider a binary classification method that estimates the class probability and where the observation weights can be specified (e.g. Logistic Regression). To accommodate the difference losses from TP ...
- 2,754
10
votes
1
answer
178
views
There are infinitely many proper scoring rules. Are they all equally valid? Or is log loss superior because of its connection to max likelihood?
I'm kind of obsessed with binary loss functions.
How to create a (binary) loss function (scoring rule):
Create a function, $f: [0,1] \rightarrow \mathbb{R}_{\geq 0}
$, that is symmetric about $x=\...
2
votes
1
answer
101
views
BerHu custom loss function for XGBoost
I would like a loss function that penalizes outliers like the squared loss, while treating small errors less sharply, like the absolute loss. It seems that I am looking for a Huber loss function, but ...
- 77
0
votes
0
answers
18
views
Is it a good idea to incorporate a feature into the loss function when training a neural network model that does regression?
When training a neural network that does regression, assuming I have 3 features called "a", "b", and "c". The corresponding target is called "d". In theory, ...
- 111
3
votes
2
answers
165
views
Estimating Smooth Density Field from Limited Sampled Data
I want to estimate a “density field”, specifically $P(y|x, m)$, for binary labels $y$ associated with 2D points characterized by spatial coordinates $m$ and additional spatio-temporal features $x$. ...
- 81
0
votes
0
answers
18
views
Cost function for time series anomaly detection with limited labelled anomalies
Given a time series $y_1, \dots, y_n$, I will fit some models to the data and I want to choose one for anomaly identification. I'm interested in a cost function that rewards a model whose fitted ...
- 722
5
votes
2
answers
607
views
Expected loss function from bias variance trade off (integral help)
I have a hard time understanding this formula. It's from bias-variance trade-off proof. and the expected loss function is as follows:
$$L(\hat f) := \mathbb E_D\mathbb E_{(x,y)}[(y-\hat f(x))^2]=\...
- 89
0
votes
0
answers
65
views
About the hinge loss and slack variables
I'll be denoting the $ith$ training example, target label and slack variable as $\mathbf{\vec x}^{(i)}$, $y^{(i)}$ and $\xi_i$ respectively.
Hinge Loss :
The hinge loss function in the context of ...
- 115
1
vote
1
answer
39
views
Accuracy "overfits" but loss doesn't?
I'm perplexed as to why my loss doesn't go up when the accuracy goes down (after about 40 epochs). Isn't it possible to tell overfitting from the loss curve alone? (I'm of course referring the ...
- 805
0
votes
0
answers
9
views
Computing Test Loss in Kernel Ridge Regression
In Kernel Ridge regression we have the standard loss function $$L(\beta) = \|Y-K\beta\|_2^2 + \alpha \beta^T K \beta$$
Here, $K$ is the kernel (gram) matrix.
If I compute $\beta$ on a training set, so ...
- 1,531
0
votes
0
answers
22
views
Optimizing parameters for a non-standard probability density function
We have a non-standard multivariate probability density function, P(x | q), where x is a vector, and q are the parameters of the density. I get events ...
- 101
2
votes
1
answer
101
views
On using the loss as a metric?
The context is model evaluation in supervised learning. I am coming from a numerical optimisation background. For me it is quite natural to use the loss of the model (what we optimise during training) ...
- 1,665
0
votes
0
answers
114
views
Huber-Loss optimisation using Stochastic Gradient Descent to estimate intercept and coefficient of regression line
What: I'm trying to minimise the Huber-Loss for a linear regression using Stochastic Gradient Descent from scratch.
Problem: It seems like that the coeffcient $m$ doesn't get optimised, therefore the ...
- 111
1
vote
1
answer
75
views
What happens if I use a single scalar output with MSE Loss for classification tasks? [closed]
Rather than using Cross Entropy Loss and one hot encoding for neural network classification tasks, if my model outputs a single scalar value and I use mean squared error loss what will happen?
- 236
2
votes
1
answer
144
views
Which form of cross-entropy loss is correct?
For classification problems with more than two classes, I've seen these two forms of cross-entropy loss:
-$\sum_k y_k \log(a_k)$
-$\sum_k y_k \log(a_k) + (1-y_k) \log(1-a_k)$
Here $y_i$ are the true ...
- 697
0
votes
0
answers
9
views
Choosing Distortion Measures for Decision Rules with Logarithmic Posteriors
I've been delving into Bayesian decision theory and specifically looking at scenarios where we work with the logarithm of the posterior distribution (log-posterior). My understanding is that in such ...
- 113
2
votes
1
answer
83
views
Loss function for volatility forecasts from GARCH
What are the options for loss functions, when trying to compare the volatility (sigma) forecasts from different GARCH models? I was thinking about the Qlike function but am not sure if this would give ...
- 703
3
votes
1
answer
126
views
Why is the regularization term multiplied by the error term in the cost function of SVM?
The cost function of the Optimal Margin Classifier(non-kernelized SVM) is given as :
$$
J(\mathbf{\vec w}, b) = \frac{1}{2}\|\mathbf{\vec w}\|_{2}^{2} + C \sum_{i=1}^{n}\max(0, 1-y ^{(i)}(\mathbf{\vec ...
- 115
0
votes
0
answers
103
views
What are the benefits of using pseudo-residuals in Gradient Boosting?
At each iteration $t$ of the Gradient Boosting algorithm, we're basically trying to add the weak learner $f_t$ that minimizes:
$$
\mathcal{L}_t = \sum\limits_{i=1}^{n} l(y_i, \hat{y}_i^{(t-1)} + f_t(\...
- 143
0
votes
0
answers
19
views
Increasing the clarity in the tasks of image generation using CNN
What methods exist to improve the quality of generated images and the clarity of contours in the tasks of image denoising/debluring (using CNN), style transfer etc? I am interested in approaches that ...
- 101
0
votes
0
answers
47
views
How to penalize disagreement between two classification loses?
I am working with a multi-head, multi-loss neural network. Each of the two heads is associated with a multi-class classification loss. The losses are combined additively. Assume loss 1 is trained to ...
4
votes
4
answers
171
views
Why do we work with factor of likelihoods instead of e.g. a sum for a batch in the negative log likelihood loss function?
In a classification task, at a certain stage of the training process, we get a likelihood of sampling proper class Y for a particular data point X. For batch, we get many independent likelihoods.
Let'...
3
votes
1
answer
250
views
Scenario where minimizing 0-1 loss is different than minimizing hinge loss
Suppose we're using linear predictors. I'm trying to conceptually understand how minimizing hinge loss and 0-1 loss aren't necessarily the same. For instance I was told that one can choose a set of ...
2
votes
1
answer
44
views
Training loss reach to zero, then suddenly increases, then decreases to zero
I get the following loss behavior when training multilayer perceptron with mean squared error loss on some synthetic data using default Adam with default learning. (I am working on 1 demention data)
I ...
0
votes
1
answer
41
views
Is there a (lower) limit/minimum for learning rate values?
I'm building a model for traffic prediction with ConvLSTM and A3T-GCN cells. Since the input data is highly complex and the model is relatively big, I can only load ...
- 101
6
votes
2
answers
165
views
Real-world example of quantile loss used for evaluation
We can use quantile loss (a.ka. tick or pinball loss) for training a model or for evaluating predictions. (It is helpful to distinguish the two clearly, e.g. as done here.) I am interested in the ...
- 69.5k
2
votes
0
answers
40
views
Solving a system of equalities using a neural network
Assume $P$ is a set of pairs $(x, y)$,
where both $x$ and $y$ are in $\mathbb{R}^n$.
Assume $P'$ is a subset of $P$.
I want to train a neural network
$N: \mathbb{R}^n \to \mathbb{R}^m$
such that, for ...
- 21
0
votes
0
answers
30
views
Forcing NN to have fixed or identity output in a region of state space
I have a transition system defined over state space $X$,
with a transition function $f: X \to X$.
Let us assume my task is to learn a function $G: X\to \mathbb{R}$ such that $G$ is decreasing, i.e. $G(...
- 21
0
votes
0
answers
40
views
How does the chain-rule look for the gradient of a loss function?
When we are computing the gradient of the loss function, $L$, of a Word2Vec model, for the context word-embedding, $w_i$, and the target word-embedding, $t$. Where the loss function, $L$, looks like:
$...
- 1
2
votes
0
answers
106
views
Modification of square loss analogous to absolute and vs pinball loss: what is elicited?
Quantile regression at quantile $\tau$ minimizes the following "pinball" loss function, $L_{\tau}$, and elicits conditional quantile $\tau$.
$$
l_{\tau}(y_i, \hat y_i) = \begin{cases}
\...
- 67k
0
votes
0
answers
165
views
Exponentially Weighted Covariance Matrix with Ledoit Wolf Shrinkage
The Ledoit Wolf paper "Honey, I Shrunk the Sample Covariance Matrix" presents the formulation for the shrinkage intensity parameter estimate in Appendix B.
The formula for a weighted ...
- 49