Questions tagged [loss-functions]
A function used to quantify the difference between observed data and predicted values according to a model. Minimization of loss functions is a way to estimate the parameters of the model.
1,143
questions
0
votes
1
answer
33
views
Does the matrix notation for OLS loss function assume that indexing a row from $X$ return a column or row vecotr?
I'm confused about the matrix notation of the loss function in ordinary least squares regression. In matrix form, the expression for a linear model is:
$$\hat{y} = Xw$$
Where
$\hat{y} \rightarrow$ ...
5
votes
1
answer
111
views
Ordinal log-loss in a multiclass classification in XGBoost?
I have a multi-class problem that which classes are simultaneously mutually exclusive and have ordering. You can think of the classes as being some score: 0 (Low), 1 (Medium), 2 (High).
What I would ...
- 221
1
vote
1
answer
51
views
Is model distillation an ill-defined problem?
Model distillation (or knowledge distillation) consist in making a student model learn from a teacher model in order to eventually use the student model as an alternative to the original teacher model....
- 11
0
votes
0
answers
11
views
cGAN: Discriminator loss going to zero while Generator's going always up but the result is very good
I have a Conditional Generative Adversarial Network for Quantum State Tomography. The metrics I am monitoring during the training process are the losses and the Fidelity (the degree of similarity ...
- 15
0
votes
0
answers
14
views
Should you average loss and accuracy across epochs or steps?
In a typical training loop what is the correct way to report and average metrics like accuracy and loss. Should you maintain a running average through the whole process or reset it each epoch. What ...
- 556
0
votes
0
answers
14
views
Proximal operator of Adaptive Elastic Net
I would like to learn how to find the proximal operator of the Adaptive Elastic Net, from DOI: 10.1214/08-AOS625 "ON THE ADAPTIVE ELASTIC-NET WITH A DIVERGING NUMBER OF PARAMETERS" by HUI ...
- 35
2
votes
0
answers
32
views
Why this error surface for this loss function has negative values>
I'm studying linear regression in a Coursera course.
The author uses the following loss function in his examples:
And he shows the following error surface when he explains the gradient descent.
In ...
- 718
2
votes
1
answer
17
views
How to Normalise a loss function containing actual values?
I defined a loss function that I am trying to optimise consisting of two elements. let's say a and b:
w1 * (a_actual - a_measured)^2 + w2* (b_actual - b_measured)^2
I am trying to simplify the ...
Peyman Shadmani
0
votes
1
answer
58
views
How (or can) you formulate the Fisher information matrix in terms of a loss function, specifically cross-entropy loss?
I recently saw the following formulation of the Fisher information matrix in a paper on Transformer pruning:
$$
\mathcal{I} := \frac{1}{|D|} \sum_{(x,y) \in D} \left( \frac{\partial \mathcal{L}(x,y;1)}...
- 1
0
votes
0
answers
59
views
Is my custom loss function differentiable?
Consider the following loss function.
loss = ( ( torch.where(d > threshold, torch.sqrt(d), 0) * t ) + ( torch.where(d <= threshold, (1 - d), 0) * (1 - t) ) )
...
- 285
0
votes
0
answers
19
views
Constrain loss function based on input variables
I'm doing a simple regression task using a neural network with two inputs (x1 and x2) and one output (...
0
votes
0
answers
20
views
Search recall optimization - what appropriate loss function to use?
I am studying machine learning and wanted to work on a project of my own so that I have better chances after graduating college. I'm studying the application of ML to improve searches using a toy ...
- 161
1
vote
2
answers
116
views
Mean squared error (MSE) vs Least squares error (LSE)
From my understanding the only difference between MSE and LSE is that with MSE you divide the sum of squared errors by the total number of values to get an average rather than just using the sum.
This ...
- 2,121
0
votes
0
answers
31
views
Unexpected distribution of scores after using class-weighted loss, when data is highly imbalanced (2%), low N and high p
I won't go into the way the data is built because I want to keep the discussion general. Relative to balancing, I couldn't find a lot of materials online about the results of cost-sensitive learning.
...
- 326
0
votes
0
answers
13
views
Loss function for CNN segmentation problem that allows for segmentations to 'shift'?
In short: I am looking for a loss function that prioritizes the size of the segmentation over the location of the segmentation.
We are using a modified U-Net to segment an event on two time based ...
- 101
0
votes
0
answers
72
views
Comparison of Squared Dice Loss vs. Standard Dice Loss
I've been diving into segmentation tasks and came across two variations of the Dice Loss that I'm considering for my neural network: the standard Dice Loss and the Squared Dice Loss.
The Standard Dice ...
- 118
1
vote
0
answers
33
views
Which parameters optimise the weighted cross-entropy loss for a pre-specified categorical distribution?
Question:
Given a categorical distribution $C_q$ with parameters $q_1, \ldots, q_K$ with $K > 2$, $\sum_k q_k = 1$, which (new) categorical distribution $C_p$ with parameters $p_1, \ldots, p_K$ ...
- 51
0
votes
1
answer
65
views
Image pre-processing for Variational Autoencoder
Setting
I am training a Variational Autoencoder (VAE) on the CIFAR10 dataset, which has RGB colors. The VAE uses convolution and transposed convolution layers as well as linear layers to encoder and ...
- 33
2
votes
1
answer
35
views
How do additional loss terms impact the parameter count in a Deep Neural Network?
I've been working on training DNNs for various tasks and recently started incorporating additional loss terms into my models to improve convergence and performance.
I'm curious to know if I need to ...
- 153
0
votes
0
answers
21
views
Effective number of samples (ENS) and evaluation metric choice
I have a question regarding the appropriate evaluation metric for my problem. I'm working on a classification problem with highly imbalanced classes. I've decided to employ ENS (effective number of ...
- 1
1
vote
0
answers
28
views
Why I get spikes during training with vanilla gradient descent? [closed]
I developed my own NN toolbox, and it seems it works fine.
But I am not sure why I get these spikes in my loss during training:
I a training for a classification task of 2 inputs and 2 classes, ...
- 119
1
vote
1
answer
22
views
Ideal loss for a multi-label problem with soft targets
Given an input X, my goal is to predict a list of probabilities for n factors, where the factors could be attributes like ...
- 123
1
vote
1
answer
56
views
What exactly is the problem with overconfident predictions?
Say I have a neural network that classifies images by training to minimise cross-entropy loss with one-hot encoded training labels. It is often seen that such neural networks are 'overconfident', with ...
- 111
1
vote
0
answers
89
views
When training word2vec, why is a new negative sampling process formulated instead of simply downsampling?
(For background, see The Skip-Gram Model.1 This question does not exactly use their notation, but you should be able to follow along.)
The original skip-gram log-likelihood of a single word, $w$, ...
- 1,260
0
votes
1
answer
38
views
Is categorical cross entropy loss wasting training data?
Given this categorical cross entropy loss function $-\text{sum}(y * \text{log}(\hat{y}))$, where $y$ is a one-hot word vector and $\hat{y}$ is a vector of class probabilities, this loss function will ...
- 632
1
vote
1
answer
43
views
What regression problem this closed-form solution solves (if any)?
In my company, I stumbled upon a piece of software whose purpose is to correct with fresher data some future forecasts made by a complex system.
To formalize a little bit: there's a forecasting model $...
- 1,566
0
votes
0
answers
58
views
Is the equation of the derivative of a loss function relative to the input to the Sigmoid (z) the same whether computed backward or forward?
I am referring to the derivative of the binary cross-entropy loss function for logistic regression. Using back-propagation, the derivative of the loss function L ...
- 133
0
votes
0
answers
31
views
Custom loss function optimized for correlation for gradient boosted trees algorithms like XGBRegressor or LGBMRegressor
I have a rather classical tabular data prediction problem and more or less successfully using XGBRegressor or LGBMRegressor both using MSE as their loss function.
A slight deviation from the standard ...
0
votes
0
answers
34
views
Is it valid to let a neural network predict the actual value y without denormalizing through the normalized data x?
Let's assume I have a regression task. My data is numerical with x as the input data and y as the target variable.
What I usually do is that I normalize my data with both x and y. Then I feed it into ...
- 13
0
votes
0
answers
21
views
Loss drops sharply at the beginning
I use a transformer model to make some predictions similar to time series classification. The model consists of some embedding layers, multihead attention and dense layers. Why does the model loss ...
- 55
1
vote
1
answer
62
views
Multiclass Classification using Binary Representation and Sigmoid Activation in Neural Network
I am currently working on a multiclass classification problem where I have categorical variables that I've encoded using binary representations as follows:
...
- 13
0
votes
0
answers
66
views
How are the generalization error and the test set error related? Are they?
Let us say that we have a set of input data $x \in X$ with labels $y \in Y$. Given a suitable loss function $R(f(x), y)$, we can define the generalization error of a learnt function $f_{n}$, call it $...
- 9
8
votes
1
answer
622
views
Loss function that penalizes wrong sign predictions
Consider the following loss function:
$$\mathcal L (y, \hat y) = |y| \left[\log (1 + |y - \hat y|^2) \mathbf 1 _{\{y\hat y \geq 0\}} + |y - \hat y|^2\mathbf 1 _{\{y\hat y < 0\}} \right ]$$
The idea ...
- 691
8
votes
1
answer
942
views
Sign aware loss function
I want to create a regression model with the following properties:
prediction should be close to target
target and prediction should have the same sign
small penalty if either target or prediction ...
- 691
1
vote
0
answers
14
views
How to optimize on primary and secondary objective functions in machine learning?
I have developed a probability of loan default logistic regression model and it rank orders the risk across the deciles well, however, it doesn't rank order losses (which occur much later) well. Is ...
- 11
0
votes
0
answers
19
views
Was approaching this as a classification problem a mistake and should I have to use regression instead?
So I am training a model to predict baseball plate appearance outcomes, which I have been modelling as a single multi-class output problem, namely because single, mutually exclusive outcomes is what ...
3
votes
2
answers
240
views
Understanding the difference between MLE and MSE
When using MSE in linear regression, I understand that we aim to minimize the average of the square errors between the predicted value $f(x)$ and the actual label y. My understanding is that we aim to ...
- 53
0
votes
0
answers
117
views
How do we calculate the loss (and adjust the weights) in reinforcement learning if we don't know what the full correct output should look like?
I think my question would be best explained using an example.
Let's say I want to train a neural network to play Tic-Tac-Toe. Since I'm using reinforcement learning I initialise the network with ...
- 1
1
vote
1
answer
66
views
Machine learning model for matching records
I have an example, where I want to automate matching up records in two datasets. I'm wondering what kind of machine learning model would potentially be able to deal with this kind of issue. I'm ...
- 30.4k
0
votes
0
answers
62
views
Optimization of a noisy loss function
I'm trying to optimize a noisy loss function (experimental) where the absolute value of the gradient changes significantly depending on the direction taken. In other words, some parameters have a ...
0
votes
0
answers
12
views
Custom Weighs on Errors while Training
I have a linear regression model, a XGBoost model, and a MLP model that I've developed for a dataset that predicts a binary match outcome using sckit.
I want to set a rule on my model where certain ...
- 25
0
votes
0
answers
22
views
XGboost validation immediately drops and becomes stationary
I'm attempting to fit an xgboost model to some data. During training I'm seeing the RMSE for the validation set very quickly decrease, and then become basically static. The Validation performance is ...
- 1
2
votes
1
answer
36
views
Is it legit to use a point estimate along with a conformal predictive interval from a quantile regressor?
I have a quantile regression model that gives me prediction intervals (PI), and I also need to have a point estimate for all sorts of reasons (or at least something as close to a point estimate in a ...
- 21
1
vote
2
answers
37
views
why does a model with a larger val loss produce higher accuracy than a model with a smaller val loss?
I did ANN classification on training data with oversampling and without oversampling. For each data, the smallest validation loss is sought with trial and error of 18 models. In the data without ...
- 117
2
votes
2
answers
45
views
Implementing loss function for training GAN
I am trying to understand the implementation of the following loss function in training Discriminator part.
Loss function maximize log(D(x)) + log(1 - D(G(z)))
...
- 483
0
votes
1
answer
65
views
Why does the sign of the loss in VAEs appear to be backwards?
I'm trying to fully understand Variational Autoencoders (VAEs) 1 and their math but one part keeps confusing me and I hope someone can give me in an intuitive explanation what I am missing.
Here is ...
- 1
1
vote
1
answer
77
views
Minimizing the expected loss (PRML)
In Bishop's PRML in section 1.5.2, the author introduces a loss function for classification, which is the expected loss,
$$
E[L]=\sum_k \sum_j \int_{R_j} L_{kj}p(\textbf{x},C_k)d\text{x}
$$
where Lkj ...
1
vote
0
answers
68
views
Average time in which a product random variable becomes zero
Im looking for the optimal time in which a process should be cancelled before it results on financial losses.
Say M_n=X_n*Y_n-c(n) for for n =1 to 12 which is the number of hours the process gets ...
- 11
5
votes
2
answers
363
views
When dealing with data imbalance, shouldn't we never compare models based on validation loss, or at least weight it?
I know that when validating we are interested in knowing how the model performs in real-world scenarios, so we want the class ratios during validation/test to be the original ones.
Say, however, that ...
- 172
0
votes
0
answers
50
views
XGBoost custom objective/loss: when is a Reverse (inverse) Link Function required?
I'm trying to implement a custom objective function in XGBoost. I read the docs on this topic. I am not sure if I need to define a "reverse link function" (aka inverse link function) to ...
- 187