Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [loss-functions]

A function used to quantify the difference between observed data and predicted values according to a model. Minimization of loss functions is a way to estimate the parameters of the model.

0
votes
0answers
5 views

Having trouble figuring out how loss was calculated for SQuAD task in BERT paper

The BERT Paper https://arxiv.org/pdf/1810.04805.pdf Section 4.2 covers the SQuAD training. So from my understanding, there are two extra parameters trained, they are two vectors with the same ...
0
votes
0answers
7 views

Properties of Proper Loss Functions for Continuous Prediction

It is clear to me what a proper loss function is in the context of binary classification or emitting a probability. I am trying to find a full distribution counterpart. Suppose that I would like to ...
-2
votes
2answers
105 views

How to prove mathematically that SBAF(Saha-Bora Activation Function) is mathematically linked to binary logistic function?

How to prove mathematically that SBAF (Saha-Bora Activation Function) is mathematically linked to binary logistic function ? The activation function is as follows: $$y=\frac{1}{1+kx^\alpha(1-x)^{1-\...
0
votes
0answers
11 views

Bayesian point estimation: minimizing the expected loss or the mean risk?

Is there a reason why one should choose to pick his Bayesian decision minimizing the expected loss or the mean value of the risk function? The expected loss function \begin{gather} \int \mathscr{L}(\...
0
votes
0answers
27 views

Perceptron learning algorithm. Gradient descent

Linear classifier: $a(x,w) = sign\langle\,x,w\rangle$, if sign is positive object belongs to $+1$ class and if it is negative to $-1$. When we use gradient descent to train the perceptron, we are ...
0
votes
0answers
6 views

What should the form of error be on CrossEntropy or KL-divergence loss function across samples of distributions?

Suppose your model produces (discrete) probability distributions and you have some truth distributions you want to compare to. For each sample $i$, you can compute the loss as the KL divergence or ...
4
votes
1answer
20 views

Continuous loss function that can measure one-side error

I am predicting a target $y$ using regression. In my application, the prediction $\hat{y}$ should be always no less than $y$. If $y>\hat{y}$, it is definitely a wrong prediction. On the $y<\hat{...
1
vote
1answer
22 views

Is a GAN's discriminator loss expected to be twice the generator's?

If a GAN generator has the same (but reversed) hidden layer architecture as the discriminator, is a the discriminator's loss expected to be approximately double the generator's? In the examples I'm ...
0
votes
0answers
23 views

How is the L2 regularization derived? [duplicate]

I just proved to myself why the regularization is added rather than multiplied to loss function. I did so by taking the MLE formula... $$\operatorname{argmax}\sum \log(P(x_i\mid\Theta ))$$ and ...
1
vote
0answers
15 views

How does one go about evaluating a new loss function? [closed]

Suppose a new loss function similar to log loss is proposed. Now we have to check whether it'll work in practical scenarios in general and tractable. What experiments one should run get to some ...
2
votes
0answers
46 views

Neural nets with custom loss function depending on inputs?

I'm trying to make a custom loss function for my neural network. Each of my data point $i$ has a set of input parameters and an output coefficient $p_i$. My goal is to have a neural net which ...
1
vote
0answers
23 views

Substitution for unknown true density in 'Density Estimation Trees'

I'm having a hard time understanding parts of the derivation of the objective function for Density Estimation Trees (reference below) regarding the loss function. Taken from the article (Sec. 3.1): ...
1
vote
1answer
42 views

The most appropriate model for dataset [closed]

For example, we have a dataset, and we want to find best representative hyperplane for this dataset. In other words, we aim to perform regression operation. This hyperplane can be in linear, ...
0
votes
0answers
39 views

Multiclass hinge loss gradient

I am trying to compute the gradient of multi class hinge loss function but i am kinda confused. First things first, I have a W matrix [10xD] (10 classes) that contains the weights. The loss ...
0
votes
0answers
27 views

train_accuracy and train_loss are not consistent in binary classification [duplicate]

I am training a binary classification algorithm in Keras, the loss is cross-entropy ...
1
vote
1answer
25 views

Should loss function be defined over output or parameters?

In machine learning loss is usually defined over the actual output and the predicted output $L(Y,\hat{Y}(X))$, while in statistics it's defined in the parameter space $L(\theta,\hat{\theta}(X))$. Why? ...
1
vote
1answer
25 views

How to compute the loss normal function (not standard normal distribution)

I am struggling with computing the expression of the following term: $E[x-Q]^+$ where $x$ is a normal r.v with mean $\mu$ and variance $V^2$. I want to express it as a function of $\mu$, $V^2$ ...
0
votes
0answers
27 views

Training loss does not decrease beneath a very specific value

I am training a U-Net for the semantic segmentation of images. I have 3 classes (circles, squares, background) that I want to distinguish. No matter what changes I do to my network architecture, or ...
0
votes
0answers
21 views

Conditional Density Estimate Loss. Why the double integral?

I read RFCDE: Random Forests for Conditional Density Estimation. Just like it sounds, these folks trained random forests for making conditional density estimates. At inference time, density estimates ...
1
vote
1answer
28 views

Loss function for KNN Regressor

What is the Loss function for KNN Regressor? Would it be similar to OLS? If so what would be the main difference?
1
vote
0answers
48 views

The question of Taylor expansion of loss function in XGBoost [duplicate]

I am learning XGBoost from documentation, but there are a few questions in the derivation of it. In the part of ...
0
votes
1answer
32 views

Why do we use “Sum of Squared Errors” as loss function in linear regression? [duplicate]

What is a loss function? How can we relate the slope of Linear Regression with Sum of Squared Errors?
0
votes
3answers
53 views

Weights not converging while cost function has converged in neural networks [closed]

I'm talking in an ideal scenario where a validation set isn't used. Without validation, as many epochs as possible are calculated. Training stops and finishes only when the loss function is minimized ...
0
votes
1answer
39 views

Does minimizing expected squared loss (MSE) result in an unbiased estimator?

I have heard that the estimator with the lowest expected squared loss (mean squared error) is not always unbiased, but I have also heard that the constant that minimizes the expected squared loss vs. ...
0
votes
2answers
67 views

Variance/bias trade off regularisation penalty - why does it take this form?

In machine learning, if we estimate weights using a loss function $$L(W) = ||Y-F_W(X)||^2$$ (where $W$ is a weight matrix) we may add a "regularisation penalty" to control for the "variance/bias ...
1
vote
0answers
13 views

Sudden drop in training error despite smoothly-varying learning rate?

I trained my model for 10 epochs using the FastAI library, and observed sharp drops in the error at the start of each epoch (first plot). However when I plot the learning rate, it's shown to be ...
1
vote
1answer
49 views

Loss function for generalized linear models

What is the loss function in the GLMs. We only deal with the mean posterior of response given input $E[Y|X]$, therefore I assume underneath we assume $L_2$ loss. Is that correct? What about other loss ...
0
votes
0answers
9 views

How to validate your loss function when it is not a simple regression or classification?

Assuming I have loss function f(y_pred,y_target) that I will use to train my neural network. In this case the loss function is a regression, and let's say it should ...
0
votes
0answers
10 views

Loss function for regression for bin prediction?

Say I want to predict the weight of somebody. I know that the weight of person A is something between 85 - 90 kg, but there is no exact value. One way to treat this problem is indeed just ...
1
vote
0answers
16 views

Averaging over loss values in training of ANNs

I am training my neural network with simple SGD and I am wondering, what is the correct way to plot the loss function? I have this result The averaged loss (over 5 epochs) is less spikey and easier ...
0
votes
0answers
18 views

Is the difference between categorical cross entropy and binary cross entropy the activation function used over the output layer?

what is the difference between binary cross entropy and categorical cross entropy? Machine Learning: Should I use a categorical cross entropy or binary cross entropy loss for binary predictions? In ...
1
vote
1answer
52 views

why are there loss functions AND auxiliary loss functions?

I am a beginner reading neural networks. I wonder why there are loss functions, and then there are auxiliary loss functions. (By definition, 'auxiliary' would mean supplementary, help, support). My ...
2
votes
3answers
148 views

Why is binary cross entropy (or log loss) used in autoencoders for non-binary data

I am working on an autoencoder for non-binary data ranging in [0,1] and while I was exploring existing solutions I noticed that in many people (e.g., the keras ...
1
vote
0answers
22 views

What is the difference between the validation loss on a regression task and the mean squared error?

The validation loss on regression task using mean squared error loss function is different from the mean squared error value directly calculated on the validation set. What is the difference between ...
0
votes
0answers
56 views

Weighted binary crossentropy in U-Net has no effect on accuracy (dice coefficient)

I am currently working on implementing a weighted binary crossentropy loss function as described in the U-Net paper ...
0
votes
1answer
11 views

Loss function gets bigger on reducing data features

I have data set with 200 features and I am running it through nn with 3 hidden layers. I get 0 loss on training data set, but here comes the interesting part. If I choose to select only 100 features ...
1
vote
0answers
29 views

Bias-Variance decomposition for non-squared loss

While the Bias-Variance decomposition of the squared loss is part of any introductory ML class, I am curious to know if similar decompositions can be done for other loss functions, e.g., cross entropy?...
0
votes
1answer
204 views

How to achieve variational autoencoder (VAE) with unrestricted input?

For a normal VAE an input and a reconstruction with values in the range of $[0, 1]$ are expected. This is necessary since the log loss only makes sense for this range. If the input is not within $[0, ...
1
vote
0answers
56 views

Reinforcement Learning partial derivative of loss function w.r.t. input of softmax

In the paper "Self-critical sequence training for image captioning" (link) on page 3 they define the loss function (of the parameters $\theta$) of an image captioning system as the negative expected ...
0
votes
0answers
24 views

Loss function for a risk neutral binary classification

For binary classification task, with samples labeled $y=0$ and $y=1$, a neural network has one output node with sigmoid activation function, producing predictions $\hat{y}\in(0;1)$. Is the following ...
5
votes
3answers
854 views

Why there is square in MSE (mean squared error)?

Please forgive me for such a beginner question, since I'm learning stats . & machine learning. I'm trying to understand Mean Squared Error. I understand the "Mean Error", the Mean of Errors ...
3
votes
1answer
137 views

Why are rewards scaled when using Reinforcement Learning (RL) algorithms in practice?

I was going through this tutorial in pytorch and saw the following code: ...
1
vote
1answer
83 views

cross entropy loss max value

The cross entropy loss function for multiclass can be computed as: $$-\sum\limits_{i=1}^N y_i log \hat{y}_i$$ where $y_i$ is a class and $\hat{y}_i$ the estimated probability. The minimum value is $0$ ...
4
votes
1answer
230 views

What would be the output distribution of ReLu activation?

Suppose my data has a normal distribution and I am using an NN as a model, wherein I am applying ReLu, non-linearity to it. I am curious to know how the output distribution of the ReLu looks like? ...
0
votes
0answers
6 views

Shape of the validation loss

I am trying to understand if the validation loss should decrease constantly or can have the shape I am having in this case. I wonder because the validation accuracy does grow constantly as expected. ...
2
votes
0answers
26 views

Learning Manifolds using Gradient Descent

I have a feedforward neural network $F(W): \mathbb R^d \rightarrow \mathbb R^k$ with $Relu$ activation, $m$ neurones per layer, $L$ layers and softmax on the output layer. $W$ denotes the weight ...
1
vote
1answer
44 views

What are some good robust loss functions for binary classification using LDA?

I am doing a project where I use LDA for binary classification. I want to know how it performs when there are outliers. What are some good robust loss functions for binary classification?
0
votes
1answer
107 views

how does the loss function work in word2vec?

I was watching CS224n and I Came across this equation for word2vec loss function. As in the blue box, "for each document\training example t we are calculating the probability of context words given ...
1
vote
1answer
29 views

variance of weights in a loss function

I would like to use variance of weights from a NN layer in my loss function. I mean: $L=\frac{1}{2}\sum(y-\hat{y})^2 - \alpha var(W)$ And the question: Is it possible to have a gradient from ...
0
votes
0answers
25 views

how to consider some miss classifications “half correct” in categorical_crossentropy - for a trading system

I have a trading system where the model receives 9 time-series and predict : A - strong down B - week down C - neutral D - week up E - strong up (these classes ...