Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [loss-functions]

A function used to quantify the difference between observed data and predicted values according to a model. Minimization of loss functions is a way to estimate the parameters of the model.

2
votes
1answer
55 views
+50

Why are rewards scaled when using Reinforcement Learning (RL) algorithms in practice?

I was going through this tutorial in pytorch and saw the following code: ...
1
vote
1answer
19 views

cross entropy loss max value

The cross entropy loss function for multiclass can be computed as: $$-\sum\limits_{i=1}^N y_i log \hat{y}_i$$ where $y_i$ is a class and $\hat{y}_i$ the estimated probability. The minimum value is $0$ ...
0
votes
1answer
43 views

Loss function for binary classification with ordered subclasses [closed]

I have a set of companies that I want to classify as having either >= 250, or < 250, employees (so it's a binary classification), however my training dataset contains more detailed information ...
3
votes
1answer
95 views

What would be the output distribution of ReLu activation?

Suppose my data has a normal distribution and I am using an NN as a model, wherein I am applying ReLu, non-linearity to it. I am curious to know how the output distribution of the ReLu looks like? ...
0
votes
0answers
5 views

Shape of the validation loss

I am trying to understand if the validation loss should decrease constantly or can have the shape I am having in this case. I wonder because the validation accuracy does grow constantly as expected. ...
1
vote
0answers
18 views

Learning Manifolds using Gradient Descent

I have a feedforward neural network $F(W): \mathbb R^d \rightarrow \mathbb R^k$ with $Relu$ activation, $m$ neurones per layer, $L$ layers and softmax on the output layer. $W$ denotes the weight ...
-1
votes
0answers
18 views

Can you please explain expression of loss function?

Following is loss function for conditional GAN. I tried hard to understand intuition behind this expression.  Can you please dissect and help me understand this expression? If it is too much of ...
1
vote
1answer
25 views

What are some good robust loss functions for binary classification using LDA?

I am doing a project where I use LDA for binary classification. I want to know how it performs when there are outliers. What are some good robust loss functions for binary classification?
0
votes
1answer
26 views

how does the loss function work in word2vec?

I was watching CS224n and I Came across this equation for word2vec loss function. As in the blue box, "for each document\training example t we are calculating the probability of context words given ...
1
vote
1answer
25 views

variance of weights in a loss function

I would like to use variance of weights from a NN layer in my loss function. I mean: $L=\frac{1}{2}\sum(y-\hat{y})^2 - \alpha var(W)$ And the question: Is it possible to have a gradient from ...
0
votes
0answers
16 views

how to consider some miss classifications “half correct” in categorical_crossentropy - for a trading system

I have a trading system where the model receives 9 time-series and predict : A - strong down B - week down C - neutral D - week up E - strong up (these classes ...
0
votes
0answers
14 views

What's different between hinge loss and squared hinge loss in SVC?

Sorry for the bad english. I'm an Asian. As the title says, What's different between hinge loss and squared hinge loss in SVC and how they effect on the decision boundary? And could you tell me when ...
1
vote
0answers
35 views

Does every loss function correspond to MLE/MAP

Many of the losses used in regression/classification tasks correspond to maximum likelihood estimation (MLE) or maximum aposteriori (MAP) under a specific data likelihood distribution $p(\mathbf{y}|X,\...
0
votes
0answers
10 views

loss='categorical_crossentropy' VS loss=K.categorical_crossentropy

Why I have very different loss values in training using these two lines code to define the loss function? ...
0
votes
0answers
10 views

Why does fully convolutional network plateau first and then learns?

Im training a fully convolutional network to classify handwriting Chinese characters. The dev dataset I am using has 250 classes with 200 - 300 samples in each class. And I found out no matter how I ...
1
vote
0answers
15 views

Constraints on low dimensional representations of data

Is there literature discussing introduction of constraints to loss functions in order to specify certain structures on low dimensional representations? If so, how do they compare the efficacy of the ...
0
votes
2answers
68 views

What loss function to use if I try to minimize $\frac{1}{n} \sum_{i=1}^n (y'_i - y_i)^2$

So my goal is to minimize $$\frac{1}{n} \sum_{i=1}^n (y'_i - y_i)^2$$ Where $y'$ is output of network and $y_i$ is a target label. I have two questions: What is the name of this minimization ...
0
votes
0answers
23 views

customized loss function

I am trying to solve a regression problem where I have to predict for how long a machine will be out of order given its status when it breaks. The goal is to fix first machines that are predicted to ...
0
votes
0answers
5 views

Why does the YOLO output tensor has only one set of label predictions, while there may be more than 1 bounding boxes?

There is some controversy I see in different internet articles describing YOLO approach to object detection. This slide is from the presentation from the author of YOLO. So basically it says that we ...
0
votes
1answer
39 views

What is the difference between SSIM and MS-SSIM?

I would like to know what is the difference between SSIM and MS-SSIM? Also, there is a built-in function in Tensorflow for both of them, I am curious to know when should I use SSIM and when MS-SSIM? ...
0
votes
0answers
30 views

Softmax Loss vs Binary Loss for classification?

I was trying to understand the final section of the paper "Revisiting Baselines for Visual Question Answering". Authors state that their model performs better with a binary loss in comparison to a ...
1
vote
1answer
47 views

Hard attention loss function

I am referring to paper: Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (page 4). I wished to know, why we look to maximize the lower bound of the log likelihood ...
1
vote
1answer
47 views

How can I create a meaningful weighting for RMSE?

Background I should start off by saying I am not a mathematician and please excuse simple/stupid mistakes! The goal of my exercise is to find the “best-fitting” model for the purpose of prediction. ...
1
vote
1answer
35 views

Strange batch loss in keras

Im training a Bidirectional RNN with keras.losses.MSE and have my dataset shuffled before training. I manually split it into validation and train data. However when ...
0
votes
0answers
33 views

Ranking/Sorting Star Ratings - Bayesian Credible Interval

I recently started analyzing episode polling data from a website that uses a 1-10 rating system. I've been reading about ranking star rating systems using Bayesian Credible Interviews as explained by ...
3
votes
0answers
24 views

Bounds for the expected value of the Kolmogorov-Smirnoff loss function

Let $$ \mathcal{F}=\{F:\mathbb{R}\longrightarrow\mathbb{R}: \text{$F$ is the CDF of some probability measure on $\mathbb{R}$}\}. $$ Consider the loss function, $L:\mathcal F\times\mathcal F\to\mathbb ...
0
votes
0answers
28 views

How to “use” Yolo Loss Function

I am dealing with Yolo Loss Function (the following). $$\begin{align} &\lambda_{coord} \sum_{i=0}^{S^2}\sum_{j=0}^B \mathbb{1}_{ij}^{obj}[(x_i-\hat{x}_i)^2 + (y_i-\hat{y}_i)^2 ] \\&+ \lambda_{...
0
votes
0answers
25 views

Does every commonly used loss function has an probabilistic explanation?

We know that quadratic loss can be deduced using maximum likelihood of Gaussian distribution; cross-entropy loss can be deduced using maximum likelihood of Bernoulli distribution. In such sense I mean ...
0
votes
0answers
11 views

How to choose the best parameter for the LINEX loss function?

I am using a LINEX loss function to evaluate my forecast. What procedure should I follow to find the best $\alpha$ parameter? LINEX function: $$L(e) = \exp(\alpha e) - \alpha e - 1$$ Where $e$ is the ...
2
votes
1answer
91 views

Custom Loss Function - Inducing sparsity

From the comments, I realized that my question wasn't clear enough, so I'll start with a short background. I am trying to construct an attention model that performs classification based on just a ...
0
votes
0answers
25 views

CNN can't learn - “oscillating” loss function and classifier that marks every sample with same label

I would be really thankful for any hint as I have no idea what to do next. I am trying to create a working convolutional neural network for an image classification task. There is a 96x96 RGB image as ...
0
votes
0answers
21 views

Adam optimizer's escape from local minima [duplicate]

I noticed some interesting behaviour in my loss history while training my model. Please note the sudden change in test loss at around epoch 106. A similar drop will appear around epoch ~1000. It ...
0
votes
0answers
26 views

Binary Loss Function

Currently I am working on Lasso Logistic Regression and in the book by Bulhmann et.al (2011) page 48, it is claimed that the binary loss classification function is given by, $$ \rho(f,y) = log (1+exp(...
0
votes
1answer
18 views

Is there an advantage to normalizing labels when using MSE loss?

I am designing a NN that uses MSE as a loss regressor. Its a big network and when I train, the loss/gradients are HUGE. I have to clip my gradients our else the loss just goes to NaNs. The differences ...
2
votes
1answer
69 views

Bayes estimator with weighted Loss

I have been working through a wide variety of problems involving Bayes risk and loss functions and I couldn't immediately solve the following From "The Bayesian Choice", Consider $x \sim N(\...
0
votes
0answers
15 views

What are the shortcomings of calculating the loss in pixel space vs. feature space

While training (Variational)-Autoencoder networks, I came along the paper by Higgins et al. "DARLA" where she stated: The shortcomings of calculating the log-likelihood term [...] on a per-pixel ...
1
vote
1answer
16 views

Word for loss function except weight regularization?

A typical loss function in machine learning is: $$L(\theta,x) = \mathcal L(\theta,x) +\sum_{\theta} |\theta|$$ I typically use the word “loss function” both for $L(\theta,x)$ and for $\mathcal L(\...
0
votes
1answer
56 views

Approach to prevent bias/racism in neural network fitting?

I have a dataset comprised of different ethnic groups and I want to build a classification model on this data. When I do this I find that the performance of the algorithm is better on some groups than ...
0
votes
1answer
53 views

Is `sigmoid` required for binary cross entropy?

I have a DNN that has to predict whether an input belongs to a class or not. During training, I use binary cross entropy as a loss function. I noticed that if my ...
1
vote
0answers
21 views

Binary cross etropy loss with non binary ground truth data [closed]

Is it possible to use binary cross etropy loss with non binary ground truth data, i.e. not [0,1] values, but [0,0.1,0.5,1.0] ...
1
vote
1answer
381 views

YOLOv3 loss function

Follow-up to stats.stackexchange.com/questions/373266/yolo-v3-loss-function: In trying to finalize the development of my training labels and loss function I'm confused by the part in bold in the ...
0
votes
0answers
13 views

Modelling my own cost function for skewed distributions

A while ago I worked in a model used for fraud detection, as that involved a very asymmetric distribution, I used F1-score as my loss function. However, I cannot stop thinking about it, and I feel F1-...
0
votes
0answers
19 views

For loss function (training error), how are $X,Y$ defined in $E(T(Y,\hat{y}(X)|(y,x)))$

For loss function (training error), how are $X,Y$ defined in $E(T(Y,\hat{y}(X)|(y,x)))$ $T(y, \hat{y}(x))=\frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i(x_i))^2$ So it read that one somehow takes ...
0
votes
0answers
15 views

Proving a hinge-loss hypothesis has 0 loss with respect to 0-1-loss

In the case of multi-class classification of the form $$f(x;w_1,\ldots,w_n)= {\arg \max}_{y\in\{1,\ldots,L\}} w_y\cdot x$$ Denote $W^{opt}$ is an hypothes that minimize the hinge loss. $W^*$ is an ...
1
vote
0answers
106 views

loss function in CRF keras-contrib returns Nan in join mode

I use a BiLSTM-CRF architecture to assign some labels to a sequence of the sentences in a paper. We have 150 papers each of which contains 380 sentences and each sentence is represented by a double ...
2
votes
2answers
200 views

Loss function and evaluation metric

When building a learning algorithm we are looking to maximize a given evaluation metric (say accuracy), but the algorithm will try to optimize a different loss function during learning (say MSE/...
0
votes
0answers
248 views

Gradient of the cross entropy loss function

I have been puzzled by how to calculate the derivative of the following cross entropy loss function underlying my neural network: CEloss = $\frac{-1}{N} \sum_{n=1}^{N} \sum_{k=1}^{K} t_{n,k} \log y_{...
1
vote
0answers
76 views

How to construct a cross-entropy loss for general regression targets?

It's common short-hand in neural networks literature to refer to categorical cross-entropy loss as "cross-entropy," even though there are a number of loss functions which could properly be described ...
1
vote
1answer
515 views

Cross Entropy Loss for One Hot Encoding

CE-loss sums up the loss over all output nodes $\sum_i[ - target_i*\log(output_i) ]$. The derivative of CE-loss is: $- \frac{target_i}{output_i}$. Since for a target=0 the loss and derivative of ...
0
votes
1answer
152 views

How can I combine a particular loss function into a DNN with another loss/objective?

I'm training a fully-connected layer with a custom loss $L_1$ to perform dimensionality reduction. This loss is in function of the weights, which pushes the network to a solution which has some ...