Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [loss-functions]

A function used to quantify the difference between observed data and predicted values according to a model. Minimization of loss functions is a way to estimate the parameters of the model.

0
votes
0answers
6 views

Optimal NN architecture for regression task that benefits from classification

I am aiming to build a NN that would be optimally combining classification and regression. I have reformulated the task such that it would be less abstract and would like to know if the proposed ...
0
votes
0answers
10 views

Autoencoder loss function - why minimise MSE?

Why are most loss functions used in autoencoder learning algorithms the mean squared error?
0
votes
0answers
7 views

How to interpret when using hinge loss performs a lot better than cross-entropy loss in a multi-class clasification problem?"

Given that hinge loss is based on the marginal loss in SVM, is there any reasonable assumption / interpretation one can make on the topology of the dataset, when using multi-class hinge loss ...
0
votes
1answer
16 views

Loss function for ordered categorical data [on hold]

I am using pytorch and I am working on a test problem where I have 10 output categories, but they are ordered (in an image segmentation problem). That is, 9 is closer to 10 than it is to 1. I want to ...
1
vote
1answer
21 views

Theory on custom loss functions for GBDT and other ML

I'm looking for resources on the theory behind choosing a loss function for ML---I'm interested in GBDT but for deep learning would work as well. I'd like to get a better understanding of how the loss ...
1
vote
0answers
28 views

Requirements of a loss function for an NN [closed]

Which requirements has to accomplish a loss function? Is differentiation the single requirement?
0
votes
0answers
7 views

creation of an objection function for an NN by means of an objective from an model which is above that NN [duplicate]

As the title above says, it is about an objective function of an NN. My issue is about a task where NN is combined with another model and has to be updated, but there are no target values. The NN is ...
2
votes
0answers
33 views

Maximizing Sum of Upper Triangle Matrix Elements with Respect to Column and Row Swapping

So, I wanna make a ranking method for teams in the EPL, there are 20 teams in EPL, therefore there are $20!$ configurations of ranking assignment, my final ranking assignment would be the one that ...
0
votes
0answers
19 views

Reason of validation Loss Fluctuation [closed]

I am running a deep learning model. But the validation loss is fluctuating too much. I am using an initial learning rate of 0.000001 and I am dividing it by 10 in every 10 epoch. So far, my ...
0
votes
1answer
13 views

svm loss function gradient

I was taking Stanford's cs231n class and was unable to understand the gradient calculated using the SVM loss function. You should go here to check the notes which I am talking about. This is the SVM ...
3
votes
2answers
37 views

Are loss functions necessarily additive in observations?

In all of the contexts I've seen loss functions in statistics/machine learning so far, loss functions are additive in observations. i.e.: loss $Q_D$ of dataset $D$ is an additive aggregation of losses ...
2
votes
0answers
34 views

Choosing error function for regression

I have a dataset with ~100K samples and non-negative continuous target variable. 99% of target values are zeros and the remaining 1% are right-skewed. Here are the deciles (0 and 1 correspond to min ...
1
vote
0answers
12 views

Linear regression: How to demand similar MSE across different subgroups?

In typical least square regression, we want to minimize $||y-\hat{y}||$ where $\hat{y}=B*x$ I am now working on a car fleet management problem, $y$ can be split into several groups (in my case, ...
0
votes
0answers
8 views

Loss function for one-step-ahead volatility forcasts

I'm trying to perform the MCS test using the R-package "MCS" to compare GARCH-MIDAS Models. The loss function requires as inputs a vector with some realized volatility measure ˜ σt+1 (I chose the ...
0
votes
0answers
22 views

Terminology: “L1 regularization” even if I'm using mean instead of sum? [duplicate]

In my loss function I'm using the mean of the log-cosh error between the predictions and targets, as well as an additional regularization term that scales as the mean of the absolute value of another ...
1
vote
0answers
46 views

Why using RMSE as loss function in logistic regression takes non convex form but doesn't in linear regression? [duplicate]

I am taking this deep learning course from Andrew NG. In the 3rd lecture of 2nd week of the first course, he mentions that we can use RMSE for logistic regression as well but it will take a nonconvex ...
0
votes
1answer
22 views

Penalization term for unfairness

I am reading [1], where the researchers do a logistic regression, but add to the loss function the following penalization term for fairness $ R^{AVD}_{FP}(\theta; S) = \left\lvert \dfrac{\sum\limits_{...
5
votes
0answers
54 views

Asymmetric or unequal misclassification costs in random forest

I have a general question about asymmetric costs. In machine learning problems, there are times when the cost of a false positive is different from the cost of a false negative. Accordingly, models ...
1
vote
1answer
29 views

Enforcing Dirac delta-like Activations on a Neural Network

I am working on a custom neural network model including convolutional and dense layers. I intend to enforce outputs a certain dense layer to approximate a Dirac delta function (or any localized pulse)....
0
votes
0answers
51 views

why is VAE reconstruction loss equal to MSE loss

At which situations does reconstruction loss of VAE equals MSE loss between input and reconstructed output? Other answers where not complete!
1
vote
0answers
22 views

What is the relationship between minimizing prediciton error versus parameter estimation error?

With the advent of statistical learning techniques, people are talking a lot about prediction error, while in classical statistics, one is focusing on parameter estimation error. What is the ...
1
vote
0answers
33 views

What is the relation between a loss function and an energy function?

A loss function is a function that measures the distance between the expected value and the actual value of a model (an example of a loss function is the cross entropy). An energy function can be ...
0
votes
0answers
5 views

Ranking criterion vs. entropy criterion

Problem In a classical NLP paper (Natural language processing (almost) from scratch) I am reading now, the authors claim that The entropy criterion lacks dynamical range because its numerical ...
0
votes
0answers
31 views

relation among loss function / MLE / Bayesian estimation

I have read a lot of stuff on the relation between minimizing a loss function / maximizing the likelihood / choose a centrality measure of the posterior (Bayesian estimation); but I cannot see a clear ...
0
votes
0answers
6 views

Probability of Incurring Maximum Loss

In online classification one can use mistake bound learning, where one assumes that all $y$ are generated by some target mapping $h^*: \mathcal{X} \rightarrow \mathcal{Y},\,\, h^* \in \mathcal{H}$. ...
1
vote
0answers
17 views

What could cause a flat loss function to suddenly decrease in a u-net used for denoising?

So I am trying to understand U-Nets better, and I built a very shallow U-Net and trained it to denoise MNIST images (training set is 90% of the whole dataset). The loss function evolution I obtained ...
2
votes
1answer
29 views

Constant validation loss and increasing validation accuracy

I am training a fully convolutional network. The loss is decreasing whilst the validation loss stays mostly where it is. There is some variance in the validation loss. I thought it might overfits, ...
1
vote
0answers
11 views

Different classification loss for K-nearest neighbours

Suppose we have a general classification loss instead of a 0-1 loss. How can we modify k-NN to accommodate such a loss function? I thought about using a weighted loss matrix where $L(i,j)=0$ when $i=...
0
votes
0answers
10 views

Finding Euclidian Distance of multi dimensional features

I am trying to implement a loss function, which takes an input Image, puts it through a VGG19 encoder and returns an output from one of the convolutional layers. Let's $\phi(y)$ to be the output of <...
0
votes
0answers
17 views

Admissible and Inadmissible actions

Consider the following loss matrix. $\begin{array}{|c|c|c|c|} \hline & \alpha_1 & \alpha_2 & \alpha_3 \\ \hline \theta_1 & 1000& -300& 4000\\ \hline \theta_2 & -1000&...
0
votes
1answer
22 views

What are the applications of different cost functions and which one to choose? [duplicate]

I have just read about different cost functions for training neural networks. How to determine which cost function is applicable in any given situation?
1
vote
0answers
18 views

Optimizing a piecewise-linear, convex function [closed]

I want to find the matrix $U$ that minimizes the following loss function: $$ \min_U( \max_{0 \le j \le t} (\operatorname{tr}(W_jU) + b_j) + \operatorname{tr}(U^TU)) $$ The given loss function is ...
0
votes
0answers
36 views

Logistic Regression For Classification

The origin of logistic regression is actually logistic curve which varies from the value 0 to the value 1. It looks like the letter S, and it specifies the growth of species. If our data distribution ...
0
votes
0answers
7 views

Training and validation loss: consistency and interpretation

I have following training & validation loss for my LSTM network. I was wondering what I could deduce from this data. The validation loss seems to start where the training loss ends, is this a ...
1
vote
0answers
31 views

Equivalence of AIC and LOOCV under mismatched loss functions

Under certain conditions, AIC and LOOCV (leave-one-out cross validation) are asymptotically equivalent (Stone, 1977). Stone's paper is less than 4 pages long, but quite mathy, so I turn here for some ...
0
votes
0answers
15 views

Training a Semantic Segmentation Model with Partially Labeled Data

I have recently been tasked with a two-class semantic segmentation problem on aerial imagery. From what I can tell, off-the-shelf archetectures like U-Net seem to do well in this domain, so I plan on ...
2
votes
0answers
14 views

Name for aggregate and component cost functions?

I was really thrilled to find a clear answer to Word for loss function except weight regularization? But I'm in a situation where we have multiple costs functions in the objective function. We ...
1
vote
1answer
33 views

OLS loss function 3-d surface plot

I was trying to plot the OLS loss function as a function of coefficients $\beta_0$, $\beta_1$. As far as I know it should be a convex function with one local minimum which is also a global minimum. I'...
0
votes
0answers
10 views

How to derive a formula based on hierarchy

I have a hierarchy as follows. I want to rank the leaf nodes (i.e. 2, 3, 4) by comparing its position relativly to the two nodes 1 and 5. For instance, node 4 should be ranked highest as it is ...
0
votes
0answers
16 views

Label smoothing formula

I recently came across this paper in section 3.2 it talks about label smoothing loss and how it's equivalent to s equivalent to adding the KL divergence between the uniform distribution u and the ...
0
votes
0answers
21 views

Pytorch : Loss function is not decreasing [duplicate]

Using Pytorch framework for a 3 layer Neural Net. I,ve a 1.7 M data points, 1.36 Train & 350 K Test , to perform binary classification.Initially i didn’t use batch size and directly fed the ...
10
votes
1answer
645 views

Second order approximation of the loss function (Deep learning book, 7.33)

In Goodfellow's (2016) book on deep learning, he talked about equivalence of early stopping to L2 regularisation (https://www.deeplearningbook.org/contents/regularization.html page 247). Quadratic ...
1
vote
0answers
171 views

Understanding the GAN Loss function from the original paper

I've been reading the paper Generative Adversarial Nets by Ian J. Goodfellow et al., to have a more deeper understanding about the concepts from the author's perspective (I do understand the basics of ...
2
votes
1answer
128 views

Learning a quadratic function using TensorFlow/Keras

Heads up: I'm not sure if this is the best place to post this question, so let me know if there is somewhere better suited. I am trying to train a simple neural network to learn a simple quadratic ...
1
vote
1answer
28 views

Having trouble figuring out how loss was calculated for SQuAD task in BERT paper

The BERT Paper https://arxiv.org/pdf/1810.04805.pdf Section 4.2 covers the SQuAD training. So from my understanding, there are two extra parameters trained, they are two vectors with the same ...
0
votes
0answers
24 views

Minimizing the expected loss or the mean risk?

Is there a reason why one should choose to pick his Bayesian decision minimizing the expected loss or the mean value of the risk function? The expected loss function \begin{gather} \int \mathscr{L}(\...
0
votes
0answers
9 views

What should the form of error be on CrossEntropy or KL-divergence loss function across samples of distributions?

Suppose your model produces (discrete) probability distributions and you have some truth distributions you want to compare to. For each sample $i$, you can compute the loss as the KL divergence or ...
4
votes
1answer
23 views

Continuous loss function that can measure one-side error

I am predicting a target $y$ using regression. In my application, the prediction $\hat{y}$ should be always no less than $y$. If $y>\hat{y}$, it is definitely a wrong prediction. On the $y<\hat{...
1
vote
1answer
198 views

Is a GAN's discriminator loss expected to be twice the generator's?

If a GAN generator has the same (but reversed) hidden layer architecture as the discriminator, is a the discriminator's loss expected to be approximately double the generator's? In the examples I'm ...