Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

A function used to quantify the difference between observed data and predicted values according to a model. Minimization of loss functions is a way to estimate the parameters of the model.

2
votes
0answers
20 views

Does it make sense to use `logit` or `softplus` loss for binary classification problem?

With $z$ is the logit, $y \in \{1, 0\}$ is the class. Usually binary classification problem use sigmoid and cross-entropy to compute loss: $$\mathcal{L_1} = - \sum{y \log \sigma(z) + (1-y) \log \sigma(...
1
vote
1answer
28 views

Expected Loss Calculation

How do you solve the integral $$E(L(\theta_A,\theta_B)) = \int_0^1\int_{\theta_B}^1(\theta_A - \theta_B)f(\theta_A)f(\theta_B)d\theta_Ad\theta_B$$ where $\theta_A \sim Beta(\alpha_1, \beta_1)$ and $\...
0
votes
1answer
53 views

Training a Neural Network to maximize a function of its output [on hold]

I have a Neural network which proposes a multinomial distribution: a softmax output layer with $n$ neurons. I want want to train the network to maximize a function of this distribution. This function ...
1
vote
1answer
20 views

Is there an alternative to categorical cross-entropy with a notion of “class distance”?

I have a signal $ x \in \mathbb{R}^{t \times l} $ which is discretized into $ l = 32 $ levels for $ t = 100000 $ time points. This enables me to turn a regression problem into a classification problem,...
0
votes
0answers
7 views
0
votes
0answers
15 views

Review of distance measures

Is there work(s) on what effect different meaures of distance have on optimal values for a loss function? E.g., the best solution for L1 distance may well be different to L2 distance, and L2 distance ...
0
votes
0answers
21 views

Comparison of methods in the batch learning algorithm

In the standard batch learning algorithm, we process examples from the batch one by one, and then we average the gradients of weight changes for each neuron. I was wondering if we could change the ...
0
votes
0answers
14 views

Radial Kernel in SVM computes with Hinge Loss?

It is known that the constrained optimization problem of a SVM is to maximize M (the margin), or the same problem can be seen as: $minimize{\substack{\beta_{0},\beta_{1}, .., \beta{p} }} \{ L(X, y, \...
2
votes
0answers
50 views

GANs for image translation

I am training a generative adversarial network to perform style transfer from two different image domains (source S and target T)...
0
votes
0answers
8 views

Does robustness of L1 happens frequently

It is generally said that L2 is not as robust as L1, in the sense that big error (>1) is penalized more heavily, or the model weights these error more. However, if error is smaller than 1, would it ...
1
vote
0answers
14 views

What can be implied from loss function that its regularizer needs large coefficient

I run loss function with l1-norm as regularizer for source separation. $min\sum_{i=1}^{n} V(f(x_{i}), y_{i}) + \lambda R(f)$ I varied the coefficient ($\lambda$) from 0 to 1e14. The results ($\frac{\...
1
vote
0answers
37 views

How to implement custom loss function on keras for VAE

I have implemented a custom loss function. While training the model, I want this loss function to be calculated per batch. ...
1
vote
1answer
31 views

Can all neural network cost functions be written as an average of individual cost and as a function of the activations at the output?

In chapter 2 of Michael Nielsen's Neural Networks and Deep Learning it says backpropagation relies on The first assumption we need is that the cost function can be written as an average $C = \frac{...
9
votes
2answers
1k views

Why 0-1 loss function is intractable

In Ian Goodfellow's deep learning book, it is written that Sometimes, the loss function we actually care about (say, classification error) is not one that can be optimized efficiently. For example,...
4
votes
1answer
46 views

When training a neural network to determine appropriate number of epochs should you look at validation loss or validation objective loss?

I'm currently training a basic feedforward neural network for a binary classification task. For the loss function I'm currently using logloss, but the actual objective that I'm looking to improve on ...
1
vote
0answers
7 views

Custom Loss Lunction MXNet for Neural Network

I'm new here and was wondering if anyone would help me out with creating a custom loss function in R using the MXNet package. I was hoping to improve on my current model by trying to make it cost ...
1
vote
1answer
104 views

How can change in cost function be positive?

In chapter 1 of Nielsen's Neural Networks and Deep Learning it says To make gradient descent work correctly, we need to choose the learning rate η to be small enough that Equation (9) is a good ...
0
votes
0answers
10 views

How to combine loss values for multi-class predictions over a batch?

Let's say I have a semantic segmentation neural network and I am doing multi-class predictions for 3 classes. So, that means my loss function will be applied for each class separately and I now have 3 ...
0
votes
2answers
27 views

Is a loss function computed after each step of gradient descent or after a whole epoch?

In neural networks with mini-batch or stochastic gradient descent, is a loss function computed after each step of gradient descent or after a whole epoch?
1
vote
1answer
106 views

What model should I use to fit an arbitrary continuous distribution for each data point?

My problem is, for each input data $x$ I have some corresponding output samples $y\sim p(y|x)$, where $p(y|x)$ can be an arbitrary continuous distribution. I don't think a least squares model will ...
0
votes
0answers
24 views

Cross entropy for image segmentation

I'm running a FCN in Keras that uses the binary cross-entropy as the loss function. However, im not sure how the losses are accumulated. I know that the loss gets applied at the pixel level, but then ...
2
votes
1answer
16 views

Some general questions regarding an implementation of VAE

I'm a biology student, and on my spare time trying to learn a little bit about ML, DL and math.Right now I'm working on a project in which I need to learn how a Variational Auto-Encoder (VAE) works. ...
2
votes
1answer
121 views

Improving spam classification with tensorflow logistic regression

I would like to classify a mail (spam = 1/ham = 0), using logistic regression. My implementation is similar to this implementation and using tensorflow. A mail is represented as a bag-of-words ...
0
votes
0answers
33 views

How to minimize this loss function?

I have a loss function $L = \sum_i I(\text{log_loss}_i > 0.6)$, where $I$ is the indicator function (equals 0 when argument is false, 1 otherwise) and log_loss$_i$ is the log loss of the $i^{th}$ ...
0
votes
0answers
22 views

reconstruction loss for disentangled variational autoencoders

I am using disentangled variational autoencoders which is a variant of VAE. You will find the github code in this link. I want to quantify the difference or the loss between the ground truth (...
0
votes
1answer
27 views

is there any diffrence to use nonlinear or liner activation function in single hidden layer network

I am working in a classification problem in which I use RBF with a single hidden layer. I want to use SoftMax activation function for the output layer. I already read some documents about the ...
0
votes
0answers
12 views

Discontinuity metric for class prediction

I have a classifier that classifies input variable $\vec x$ into a few classes $0, 1, 2, \dots, n$, predicting the probability of each class. I used approaches typical in multi-class problems, now I ...
1
vote
0answers
35 views

Keras NN - loss gets stuck at 8.6791 [duplicate]

What does it mean when my neural network always gets stuck at the exact number 8.6791 when I use binary-crossentropy loss? Some strange local minimum? It happens regardless of my learning rate, ...
1
vote
0answers
38 views

which loss function is better for multi-classification

I training the multilabel classifier using the RBF network. I want to choose between different loss functions. I looked at a bunch of documentation about the different loss functions but can't get an ...
1
vote
0answers
17 views

Benefit of adding policy and value loss in actor-critic?

Is there an benefit of adding the value and policy loss together in an actor-critic setting? In the original A3C paper in Algorithm S3 it says to apply both gradients separate but I have seen people ...
0
votes
0answers
36 views

Keras accuracy not affected by any change in model [duplicate]

I am trying to construct a model for single-label multiclass classification using Keras in a Jupyter notebook. Here's my model (or see full jupyter notebook): ...
0
votes
0answers
28 views

GAN training : spikes in loss?

I came across the following situation during the training of a DCGAN on quite a complicated dataset : In many other cases of complicated datasets, loss G would slowly grow until at some point failure ...
23
votes
1answer
2k views

What is the difference between a loss function and an error function?

Is the term "loss" synonymous with "error"? Is there a difference in definition? Also, what is the origin of the term "loss"? NB: The error function mentioned here is not to be confused with normal ...
0
votes
0answers
10 views

can focal loss function works for text classification problem?

I am working on a relation extraction and classification problem. The data is in the form of text files. The data is imbalanced. I want to use focal loss function to address class imbalance problem in ...
2
votes
1answer
33 views

Bayes estimate with weighted square error loss

First, let $T(x)$ be an estimator of $g(\theta)$ and assume we have a square error loss function defined as $$L[g(\theta),T(x)]=[g(\theta)-T(x)]^2$$ Then the posterior expected risk of $T$ is $$\...
0
votes
0answers
30 views

Is the regularization term necessary when classifying one feature?

I'm using the Stochastic Gradient Descent linear classifier (implemented in Scikit-learn) to classify an image pixel by pixel. So my dataset has only one feature, ...
2
votes
1answer
37 views

Two large decreses in loss function with ADAM optimizer

I am training an RNN with tensorflow using the ADAM optimizer. The loss with respect to iteration has a sort of "two stage" decrease. As shown in the image below. The task itself if just trying to ...
0
votes
0answers
13 views

Can cross entropy be employed for measuring effective mapping of cross-modal data?

I am working on a new metric for cross-modal retrieval which can measure the effectiveness of mapping two modalities on a manifold. However, the usual approach is to employ a distance metric and not ...
2
votes
0answers
40 views

When to stop training of neural network when validation loss is still decreasing but gap with training loss is increasing?

During training of CNNs, I often come across this case for training and validation loss : X axis is epochs, Y axis is cross entropy loss. I would like to keep the "best model", meaning the one which ...
1
vote
1answer
27 views

Cross-entropy yields strange results when neural network gets too sure about his outputs

I'm using a classical CNN for image binary classification. Output is composed of two neurons, each giving the network's "raw output" for the 2 classes. So for an image, it would be eg $(0.62, -0.52)$. ...
1
vote
0answers
22 views

Impact of Data Distribution on Loss Functions' Suitability

I read at a place that mean squared loss doesn't work well with multi-modal data. I had heard that it is also not robust to outliers and Mean absolute error is more robust. How do I choose a loss ...
0
votes
0answers
56 views

custom loss function to optimize payoff via binary decision

I have written a custom loss function that is supposed to optimize a payoff via a binary decision. However, the neural networks is struggling to convert, and I'm suspecting that there's something ...
1
vote
1answer
44 views

How does gbm for classification work?

I have a got a fair idea about how it works in regression where each successive decision tree tries to predict the residual (negative gradient for loss function) and the predicted value gets added to ...
4
votes
1answer
26 views

Why isn't mean integrated square error (MISE) probability-weighted?

We often try to minimize the MISE of a KDE: $\text{E}_{\mathbb{P}^n}[\int (\hat{p}(x) - p(x))^2 dx]$. Why don't we instead try to minimize $\text{E}_{\mathbb{P}^n}[\int (\hat{p}(x) - p(x))^2 p(x) dx]$,...
1
vote
1answer
22 views

How do I pick a good value of num_epochs?

I'm running a program that uses an RNN to build a language model of some sample text: https://github.com/crazydonkey200/tensorflow-char-rnn When training it, I currently use the default value of ...
0
votes
1answer
118 views

What is a good loss function for object localisation and classification using a cnn?

Context: Using a CNN to localise a object in an image. There are two kinds of objects present represented by classes C1 and C2. The output of the CNN is 6 nodes i.e. C1, C2, x, y, w, h. Where [C1,C2] =...
3
votes
1answer
213 views

On Yolo, and its loss function

I'm having a hard time understanding some on the inner-working of YOLO, especially the loss function depicted in this seminal paper. Bear in mind that I'm nowhere closed to being a specialist in deep ...
4
votes
1answer
37 views

Show, Attend, Tell why monte carlo sampling?

In the paper Show, Attend and Tell the authors derive the Loss function as $$ L_s = \sum_s p(s|a)\log(p(y|s,a)) , $$ with $s_i \sim \text{Multinoulli}(\alpha_i)$ so that $p(s_i|a) = \alpha_i$, which ...
5
votes
1answer
56 views

Evaluate approximation of PCA from randomized algorithms

I have been comparing different PCA implementations (some via explicit calculation of the covariance matrix, some with randomized/truncated SVD) in terms of speed, and now wanted to compare how good ...
5
votes
2answers
198 views

Does a density forecast add value beyond a point forecast when the loss function is given?

Density forecasts are more universal than point forecasts; they provide information on the whole predicted distribution of a random variable rather than on a concrete function thereof (such as ...