Questions tagged [loss-functions]

A function used to quantify the difference between observed data and predicted values according to a model. Minimization of loss functions is a way to estimate the parameters of the model.

Filter by
Sorted by
Tagged with
7
votes
1answer
415 views

Why does the likelihood function of a binomial distribution not include the combinatorics term? [duplicate]

So the likelihood function for a binomial distribution is: Why is the likelihood function above not multiplied by a combinatorics term: n! / (x! * (n - x)!) If the likelihood function is ...
0
votes
0answers
23 views

Compute the Risk function

Suppose we are given $(X_1,...,X_n)$ random variables which are iid. from $\mathcal{N}(\mu,\theta)$ and finite variance. Let $Y=\frac{1}{n}\sum_{i=1}^n(X_i-\overline X)^2$ and define a loss function $...
0
votes
0answers
11 views

How to compute the loss function using Mc Fadden pseudo r-squared

I have currently developed an optimised deep learning model (DNN) using cross-entropy loss function. To objectively compare my DNN model with conditional logistics I need to obtain McFadden pseudo r2 ...
0
votes
1answer
24 views

Similar loss, different results

I have trained multiple CNNs for image classification. I suspect there is something wrong with my training pipeline, since many of my experiments get very similar training loss at the end of training, ...
0
votes
0answers
5 views

One-to-many custom loss function Keras [closed]

Is there a way to write a penalty based loss function? Say for example my training example is a tuple x = [1,0,1,0] It can be mapped to more than one output, in ...
0
votes
0answers
11 views

Why does Dice loss neglect to predict a random subset of classes?

I implemented Dice loss for a semantic segmentation problem (with a severe class imbalance in my dataset) as follows: ...
2
votes
1answer
18 views

Is exponential loss function the only reason for AdaBoost being adaptive algorithm?

Main concept of AdaBoost is that on each iteration algorithm learns what samples were difficult to classify and increases weights of these samples, while decreasing weights of those that were easy to ...
0
votes
0answers
7 views

How to interpret plot that compares log loss, hinge loss, and squared error loss

In books and articles that compare different loss functions, authors very often make the following plot. The following comes from Bishop's PRML book, with the caption Plot of the ‘hinge’ error ...
0
votes
0answers
29 views

Custom cross entropy loss function

I want to define custom cross entropy loss penalizing different class errors. Categorical cross entropy loss = $\sum_{i=1}^K y_i log(p_i)$ I want to give different weights to different prediction ...
0
votes
0answers
16 views

upper bound the error for a classification problem on axis aligned rectangles

Greeeting to the members of the community, I wrote a solution for problem 3 question 2 in section 2 from the book understanding machine learning: My solution: Let's consider the hypothesis space ...
2
votes
0answers
41 views

Evaluating classification results when importance of correct classification varies with class

Suppose we have a categorical variable $Y$ and we are trying to classify it. Our decision (the predicted class for $Y$) is $\hat Y$. We are facing a loss function which can be represented by a loss ...
0
votes
0answers
12 views

Conditional multilabel loss

Is there any limitation to train a neural network with the following loss: $-\sum_{c=0}^{C}\left[y_c \log(p_c) + \left(1-y_c \right )\log(1-p_c)\right ] + \alpha \sum_{c=1}^{C}y_c ||\hat{\theta_c}-\...
0
votes
2answers
76 views

What does it mean L1 loss is not differentiable?

I was looking through this lecture https://davidrosenberg.github.io/ml2015/docs/3a.loss-functions.pdf Slide 3: Absolute or Laplace or L1 loss not differentiable What does it mean ...
0
votes
0answers
9 views

Negative Log Likelihood for Censored Data

I'd like to use negative log likelihood as an objective function to model roughly gaussian data with right censoring. My objective function will look something like the following, but I'm not sure how ...
1
vote
0answers
36 views

Neural Net Regression SSE Loss

Notation $y_i$ is observation $i$ of some response variable $Y$. $\hat{y}_i$ is the value of $y_i$ predicted by the regression. $\bar{y}$ is the average of all observations of the response variable....
0
votes
0answers
13 views

Loss Function Algebra

Note. This is a crosspost from the math stackexchange site. That question was not receiving any answers/comments. Suppose that $$ \hat{y}_{i}^{0} = 0 \\ \hat{y}_{i}^{1} = f_{1}(x_i) = \hat{y}_{i}^{0} ...
1
vote
1answer
14 views

Regression with constrained output variable

I want to predict the labels of images using a neural network. The labels all lie in the range [-1,1] (Ratio scaled). Values with an absolute value greater than one are meaningless and do not occur in ...
0
votes
1answer
30 views

Optimality of AIC w.r.t. loss functions used for evaluation

Under certain conditions, AIC is an efficient model selection criterion. I understand this roughly as if AIC will tend to select the model that will yield the largest expected likelihood of a new data ...
5
votes
1answer
34 views

In model selection, what to do if expected prediction loss of all models is infinity?

Consider choosing a model for prediction. The criterion is expected prediction loss: the lower the expected loss, the better the model. Suppose the distribution of prediction errors has relatively ...
1
vote
0answers
32 views

Why Massive Random Spikes of Validation Loss?

My problem is to estimate the length of a straight line in an image, in pixel. My training size is 6000 images, validation is 1000 images. Each image has 200 x 200 pixels. My data is generated using ...
0
votes
0answers
12 views

Creating a regressional model that predicts tha annual awards of a sport league. How to?

I am writing an essay for my high school diploma program and my topic of choice was Machine Learning and NNs. After some thinking I decided to create a model that tries to predict the annual awards ...
0
votes
1answer
23 views

Training loss low, but testing loss is high [duplicate]

I am trying to classify a particular image. My labels are PatternA and NotPatternA. NotPatternA is any image that I have not classified as PatternA. There are very few PatternAs compared to ...
3
votes
1answer
105 views

Loss function (and encoding?) for angles

I'm training a network to predict the angle of arrival of a signal. Labels are single values in the [-180, 180) interval. I'm seeing a discontinuity in predictions around ±180 degrees, which makes ...
0
votes
0answers
14 views

How to interpret intermitent decrease of loss?

I am new to NN. I am trying to train a network where the loss is defined by the Covariance between the first 10 samples and the last 10 samples. I am able to achieve a working model, but this is how ...
0
votes
0answers
13 views

Why my binary classification neural network performance oscillates a lot through epochs? [duplicate]

I am training a CNN with Keras, vgg16-like model and i don't understand the results. For example, in epoch 15 i have good results but in 14 and 16 it's horrible (you can see it in the loss). What ...
0
votes
1answer
52 views

Disadvantages of Mean Squared Error?

I'm using mean squared error as reconstruction error for my autoencoder. The dataset is ECG (time series) and model is conv1d. I assumed MSE as the best option for reconstruction error, but it's ...
0
votes
1answer
27 views

Regarding loss function

Does loss function need to be (at least locally) differentiable in order to be used in a model that support definition of loss function (e.g., xgboost, lstm, etc.)?
0
votes
0answers
11 views

Does increasing the margin value/ delta in a SVM loss function decrease the frequency of coming across kinks when evaluating the gradient?

According to http://cs231n.github.io/optimization-1/, kinks refer to non-differentiable points of a function. Even if the analytical gradient would be zero at such a point, the numerical gradient ...
0
votes
0answers
14 views

AutoGrad Hessian in custom loss function in XGboost takes very long time

I am using the autograd package and generating the hessian of my loss function using that package as part of my custom loss function XGB model. However, it takes an extremely long time to iterate ...
0
votes
0answers
14 views

How to measure the quality of fakes produced by a digital twin?

I have asked this on stackoverflow, but it was considered too broad, perhaps because there is no specific code involved. I hope this question is more appropriate for crossvalidated. I am developing a ...
3
votes
1answer
69 views

Are there any mathematical features that an evaluation metric must have?

I'm trying to optimize the hyperparameters of my model using the Bayesian approach with the hyperopt library. I have to code a ...
0
votes
0answers
9 views

Bounding neuronal network outputs to avoid vanishing gradient

In some publications like this one, the neuronal network output $o$ is bounded such that it can never reach a given target $t$. They chose $o\in[0.1,1)$, such that any output which tries to reach the ...
0
votes
0answers
8 views

How would I use Evan Miller's sort criterion to modify the ranks of Bayesian average ratings?

Evan Miller wrote these guidelines for constructing a Bayesian average ratings and then sorting them using a multi-linear loss function: http://www.evanmiller.org/bayesian-average-ratings.html I ...
0
votes
0answers
12 views

entropy coefficient in A3C

I see that my policy entropy is decreasing very fast and converges to zero in no time, causing the policy to sample the same action again and again (which results in a sub-optimal behavior). I think a ...
0
votes
0answers
10 views

Implementing a specific GAN loss

In this paper in section 3.4 they define a GAN loss as (I simplify to the important part): $$\mathcal L_{GAN} = \max_D \min_G E_{x_{1} \sim P_1}[ \Vert D(G(x_{1})) \Vert] + \max_D E_{x_2 \sim P_2}[\...
0
votes
1answer
20 views

How to modify RMSE loss function to adopt for integer valued predictions, using a Neural Network?

Context: Prediction of dependent variables like Age, Siblings, Children, etc (which are not categorical, but bounded, and integer-valued) from a dataset using Neural Network. I'm experimenting with a ...
1
vote
0answers
9 views

Cross-entropy loss when some categories are broader than others

Let's say I want to write a classifier for pictures of dogs. Most importantly, I want to know whether something is a picture of a dog or not. Secondarily, it'd be nice to know what breed the dog is. ...
1
vote
1answer
61 views

Stability with L1 vs L2 norms

I've been looking over http://www.chioka.in/differences-between-l1-and-l2-as-loss-function-and-regularization/ and trying to get a deeper understanding of a stable vs unstable solution for L1 vs L2. ...
0
votes
0answers
12 views

Increasing Mean Intersection Over Union with Increasing Validation Loss - Semantic Segmentation

Firstly I'm new to Cross Validated so I apologize if this is structured incorrectly or I didn't find some related post or missed out something. I'm training deep networks for semantic segmentation ...
1
vote
1answer
49 views

Why does Perceptron use L1 norm as its error function?

Usually, people use L2 norm as a machine learning error function. per wiki the error function for a Perceptron model is ${\displaystyle {\frac {1}{s}}\sum _{j=1}^{s}|d_{j}-y_{j}(t)|}$ why is that?
0
votes
0answers
8 views

Are there some guidelines to follow while combining different types of losses to make a cost function?

I'm training an Autoencoder to reproduce the input, and the architecture is a simple fully connected neural network. The initial phase of the implementation was using float/integer dataset, and ...
0
votes
0answers
9 views

Similarity of perceptron criterion and SVM

In the book "Neural Networks and Deep Learning" by Aggarwal there is an exercise 2.10.1: Consider the following loss function for training pair $(\overline{X},y)$: $$L=max(0, a -y(\overline{W} \...
0
votes
1answer
90 views

Tensorflow 2.0/Keras - loss function with multiple inputs

I have a case where my model has multiple outputs, and I want to backpropagate the loss on one of the outputs based on a different label. Basically, the model should detect whether an object is in an ...
3
votes
1answer
26 views

Loss function to widen distribution of predictions

I'm working on a project where my business counterparts want my model's distribution of continuous predictions to more closely represent the distribution of actual values. Below is an example of what ...
0
votes
1answer
35 views

Hinge Loss: what does hinge mean?

This is terminology question, I know what is hinge loss mathematically; but I can't grok what hinge mean in hinge loss. Is it rod, or door hinge, or something else?
0
votes
0answers
43 views

How to find the optimal classifier for a given loss function?

For a binary classification problem (labels being 0 and 1) and a classifier $g$ we consider the loss function $L(g)=P[Y\neq g(X)]$. It is known that the optimal classifier $g^*$ that minimizes $L$ is $...
2
votes
1answer
63 views

how to interpret the sharp decline in loss in seq2seq models

I have a seq2seq model. I have applied this data over 20_newsgroup data set. My problem is that I face with exploding gradient ...
1
vote
1answer
44 views

What is a correct loss for a model predicting angles from images?

Backgrounds I am working with a dataset, where compass-like images are labeled with their corresponding angles from horizontal line ($0$ degrees). I am trying to make a CNN model to predict the ...
0
votes
0answers
9 views

Proving that an SVM problem with a complex loss function is convex [duplicate]

The end goal, along with proving that the problem is convex, is to be able to get the problem into a form that can be coded in CVX. I have m positively labeled data points $x_i$ $\in$ $\mathbb{R}^n, ...
1
vote
0answers
35 views

Proving that an SVM problem with a complex loss function is convex

The end goal, along with proving that the problem is convex, is to be able to get the problem into a form that can be coded in CVX. I have m positively labeled data points $x_i$ $\in$ $\mathbb{R}^n, ...