Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

A function used to quantify the difference between observed data and predicted values according to a model. Minimization of loss functions is a way to estimate the parameters of the model.

1
vote
1answer
30 views

Cross Entropy Loss for One Hot Encoding

CE-loss sums up the loss over all output nodes $\sum_i[ - target_i*\log(output_i) ]$. The derivative of CE-loss is: $- \frac{target_i}{output_i}$. Since for a target=0 the loss and derivative of ...
0
votes
0answers
56 views

How can I combine a particular loss function into a DNN with another loss/objective?

I'm training a fully-connected layer with a custom loss $L_1$ to perform dimensionality reduction. This loss is in function of the weights, which pushes the network to a solution which has some ...
2
votes
2answers
611 views

Asymmetric cost function in neural networks

I am trying to build a deep neural network based on asymmetric loss functions that penalizes underestimation of a time series. Preferably, by the use of the LINEX loss function (Varian 1975): $ \...
0
votes
0answers
21 views

Finding the type II error given the type I error for a minimax decision rule with 0-1 loss

Assume a two world state ($\Omega=\left\{ \omega_{0},\omega_{1}\right\}$ ) scenario and that we are given the [continuous] ROC curve $\left\{ \left(\alpha\left(\theta\right),1-\beta\left(\theta\right)\...
0
votes
1answer
24 views

Is the loss is the same as the error in deep learning?

Is the loss is the same as the error in deep learning? I feel it's the same but I'm maybe wong...
0
votes
0answers
9 views

Measuring predicitve accurateness relative to mean squared error

Imagine that $N$ agents try to predict t $k$ values $e_1, ..., e_K$, that differ in their 'predictability', i.e. some values are much easier to predict than other values. I am trying to define a ...
0
votes
0answers
7 views

strange loss trajectory for vgg11

I trained a vgg network with 11 layers from scratch and am observing some strange behavior with the loss plot (see below). The loss is binary cross entropy. My learning rate is 1e-4 and I decreased ...
0
votes
0answers
27 views

Loss function for comparing high-dimensional joint distributions

I'm synthesizing data trained from a source dataset, and am looking for a loss function to compare different data synthesis methods*. I have some ideas below, but each has drawbacks and none is very ...
0
votes
0answers
8 views

Loss decreases but mAP is dropping

I'm using the RetinaNet model for object detection in images. When I train the model, I all ways see the same behavior: For the first three - four epochs the mAP increases and then it decreases again. ...
3
votes
0answers
51 views

Why without regularization, the asymptotic nature of logistic regression would keep driving loss towards 0 in high dimensions?

While understanding the Logistic regression, I didn't completely get the behavior of its asymptotic nature which says: Without regularization, the asymptotic nature of logistic regression i.e (it ...
1
vote
0answers
27 views

Influence of RMSE versus MSE on Convergence of Gradient Descent

I am working in TensorFlow and was wondering if I choosing an MSE loss function would cause different convergence behaviour when compared to a RMSE loss function. The square root will influence the ...
1
vote
0answers
22 views

How to optimize ratiometric loss function with variance term in it?

I'm training a neural network (or any ML model with non-convex gradient-based optimization) to predict a continuous outcome variable. Currently, I use the mean squared error loss function, i.e., if $y$...
1
vote
1answer
30 views

different trends in loss and AUC ROC metric

I am training a deep neural network for a binary classification I am using binary_crossentropy as loss and area under the roc curve as performance metric as ...
0
votes
0answers
25 views

Bayes risk with explicit wegihted average for $\mu$

I have the following problem to solve: $Y_i\overset{i.i.d.}{\sim} N(\mu,1) \text{ for } i = 1,\dots,n$. Let $\hat{\mu}(Y_{1:n})$ denote an estimator of $\mu$. The loss is quadratic and the ...
1
vote
1answer
241 views

Yolo v3 loss function

The original loss function can be seen here and is more or less explained in Yolo Loss function explanation: \begin{align} &\lambda_{coord} \sum_{i=0}^{S^2}\sum_{j=0}^B \mathbb{1}_{ij}^{obj}[(x_i-...
0
votes
1answer
33 views

back propagation in neurons with zero weight and some specific conditions

I have read a lot of articles to understand what is happening behind the scene in backpropagation like Ive gone through this and many other like that. I think I understand how the backpropagation ...
3
votes
2answers
106 views

Confusion on hinge loss and SVM

I'm reading a book on data science and get confused about how the book describes the hinge loss of SVM. Here is a figure from the book on Page 94: This figure shows the loss function of a NEGATIVE ...
0
votes
0answers
31 views

Generalization Of Cross Entropy Derivative

I'm trying to find the derivative of the cross entropy loss function (without applying any function like softmax) in context of neural network. I have the following item: Where R(θ) is my cross ...
0
votes
0answers
14 views

Subtract-everything-to-one-side loss function

For any given regression problem, there are literally an infinite number of possible loss functions that, when minimized, give something pretty close to an OLS or TLS solution. For example, when ...
0
votes
0answers
6 views

How do tensorflow canned estimators compute loss?

This might seem like a weird question. But for some reason, the documentation lacks any clarification about the subject. For example: https://www.tensorflow.org/api_docs/python/tf/estimator/...
0
votes
0answers
8 views

Bounding Classification Loss Function

I am having trouble to establish some bounds. An exercise asks to me to bound: $$ L(\theta) - L(\tilde{\theta}) \leq C E[\min((X'(\theta-\tilde{\theta})^2,1)] $$ where $L(\theta) = E[(Y-\phi(\theta'...
1
vote
0answers
27 views

Model fitting vs minimizing expected risk

I'm confused about the mechanics of model fitting vs minimizing risk in decision theory. There's numerous resources online, but I can't seem to find a straight answer regarding what I'm confused about....
0
votes
0answers
8 views

Loss Function Asymptotic Distribution

I am trying to find the asymptotic distribution of a maximum estimator loss function of the type: $$ \hat{\theta} = \arg \max_\tilde{\theta} M_n(\tilde{\theta},x) $$ $$ \theta = \arg \max_\tilde{\...
1
vote
1answer
30 views

regression algorithms for asymmetric losses

I have a regression problem where over-predictions are better than under-predictions. I am now aware of Quantile Regression. Just wondering, which machine learning algorithms exist and/or can be ...
0
votes
2answers
33 views

How to calculate the derivative of crossentropy error function?

I'm reading this tutorial (presented below) on computing derivative of crossentropy. The author used the loss function of logistic regression I think. https://www.dropbox.com/s/rxrtz3auu845fuy/...
2
votes
1answer
41 views

Constant Accuracy with decreasing loss

I am fairly new to Cross Validated section so I apologize if my question structure is incorrect. I am currently working on Fully Convolutional Networks for Semantic Segmentation. I am first trying ...
0
votes
0answers
30 views

Image reconstruction with deep learning preserving speckle noise

My goal is to reconstruct images that have a lot of speckle noise on them, using a deep fully convolutional neural network. So far I have tried using the obvious choices of L1 and L2 losses for the ...
0
votes
0answers
24 views

How to understand what is going on based on the loss curves?

I have been implementing a many-to-one LSTM and have been searching for the best hyperparameters. However, sometimes I am a little confused on the reason the loss acts as it does. Here is a screenshot ...
2
votes
1answer
68 views

Why binary crossentropy can be used as the loss function in autoencoders?

I was wondering why binary crossentropy can be used as the loss function in autoencoders trained on (normalized) images, e.g. here or this paper? I know that binary crossentropy can be used in binray ...
2
votes
0answers
23 views

Could a mismatch between loss functions used for fitting vs. tuning parameter selection be justified?

Could it make sense (and if so, under what circumstances) to define a penalized estimator based on one loss function but then select its tuning parameter (say, via cross validation) based on another ...
0
votes
0answers
100 views

Loss decreasing but highly oscillating over all training examples

I'm training a sequence model. I have 40 sequences of length 70,000. I have divided these into minibatches of size $[40, 200]$. Since I want to utilise the stateful nature of the GRU I'm training, I ...
0
votes
1answer
40 views

Keras backend (tensorflow) vs tensorflow [closed]

I do not really understand the difference between Keras backend (when you use tensorflow as backend) and tensorflow. I saw some posts where people were trying to modify a Keras loss function and to do ...
0
votes
0answers
23 views

Neural network not learning with custom loss function

I have implemented a custom loss function that is supposed to enhance a binary cross entropy loss function, by weighting incorrect decisions depending on an opportunity cost. The whole code is listed ...
2
votes
1answer
55 views

Does it make sense to use `logit` or `softplus` loss for binary classification problem?

With $z$ is the logit, $p \in \{1, 0\}$ is the class. Usually binary classification problem use sigmoid and cross-entropy to compute loss: $$\mathcal{L_1} = - \sum{p \log \sigma(z) + (1-p) \log \sigma(...
1
vote
1answer
42 views

Expected Loss Calculation

How do you solve the integral $$E(L(\theta_A,\theta_B)) = \int_0^1\int_{\theta_B}^1(\theta_A - \theta_B)f(\theta_A)f(\theta_B)d\theta_Ad\theta_B$$ where $\theta_A \sim Beta(\alpha_1, \beta_1)$ and $\...
1
vote
1answer
73 views

Is there an alternative to categorical cross-entropy with a notion of “class distance”?

I have a signal $ x \in \mathbb{R}^{t \times l} $ which is discretized into $ l = 32 $ levels for $ t = 100000 $ time points. This enables me to turn a regression problem into a classification problem,...
0
votes
0answers
17 views

Review of distance measures

Is there work(s) on what effect different meaures of distance have on optimal values for a loss function? E.g., the best solution for L1 distance may well be different to L2 distance, and L2 distance ...
0
votes
0answers
25 views

Comparison of methods in the batch learning algorithm

In the standard batch learning algorithm, we process examples from the batch one by one, and then we average the gradients of weight changes for each neuron. I was wondering if we could change the ...
0
votes
0answers
16 views

Radial Kernel in SVM computes with Hinge Loss?

It is known that the constrained optimization problem of a SVM is to maximize M (the margin), or the same problem can be seen as: $minimize{\substack{\beta_{0},\beta_{1}, .., \beta{p} }} \{ L(X, y, \...
2
votes
0answers
53 views

GANs for image translation

I am training a generative adversarial network to perform style transfer from two different image domains (source S and target T)...
0
votes
0answers
8 views

Does robustness of L1 happens frequently

It is generally said that L2 is not as robust as L1, in the sense that big error (>1) is penalized more heavily, or the model weights these error more. However, if error is smaller than 1, would it ...
1
vote
0answers
16 views

What can be implied from loss function that its regularizer needs large coefficient

I run loss function with l1-norm as regularizer for source separation. $min\sum_{i=1}^{n} V(f(x_{i}), y_{i}) + \lambda R(f)$ I varied the coefficient ($\lambda$) from 0 to 1e14. The results ($\frac{\...
1
vote
0answers
84 views

How to implement custom loss function on keras for VAE

I have implemented a custom loss function. While training the model, I want this loss function to be calculated per batch. ...
1
vote
1answer
47 views

Can all neural network cost functions be written as an average of individual cost and as a function of the activations at the output?

In chapter 2 of Michael Nielsen's Neural Networks and Deep Learning it says backpropagation relies on The first assumption we need is that the cost function can be written as an average $C = \frac{...
11
votes
2answers
1k views

Why 0-1 loss function is intractable

In Ian Goodfellow's deep learning book, it is written that Sometimes, the loss function we actually care about (say, classification error) is not one that can be optimized efficiently. For example,...
4
votes
1answer
70 views

When training a neural network to determine appropriate number of epochs should you look at validation loss or validation objective loss?

I'm currently training a basic feedforward neural network for a binary classification task. For the loss function I'm currently using logloss, but the actual objective that I'm looking to improve on ...
1
vote
0answers
15 views

Custom Loss Lunction MXNet for Neural Network

I'm new here and was wondering if anyone would help me out with creating a custom loss function in R using the MXNet package. I was hoping to improve on my current model by trying to make it cost ...
1
vote
1answer
143 views

How can change in cost function be positive?

In chapter 1 of Nielsen's Neural Networks and Deep Learning it says To make gradient descent work correctly, we need to choose the learning rate η to be small enough that Equation (9) is a good ...
0
votes
0answers
14 views

How to combine loss values for multi-class predictions over a batch?

Let's say I have a semantic segmentation neural network and I am doing multi-class predictions for 3 classes. So, that means my loss function will be applied for each class separately and I now have 3 ...
0
votes
2answers
31 views

Is a loss function computed after each step of gradient descent or after a whole epoch?

In neural networks with mini-batch or stochastic gradient descent, is a loss function computed after each step of gradient descent or after a whole epoch?