Questions tagged [loss-functions]

A function used to quantify the difference between observed data and predicted values according to a model. Minimization of loss functions is a way to estimate the parameters of the model.

Filter by
Sorted by
Tagged with
1
vote
1answer
17 views

Regarding 0-1 loss and hinge loss functions of SVM

I would like to ask about SVM. I want to compare between 0-1 loss and hinge loss functions. My question is how to compare between them?!. Should we construct different SVM models, which each for ...
0
votes
0answers
38 views

What happens if I use categorical cross-entropy to classify just one class

I have used categorical cross-entropy instead of binary cross-entropy to classify just one class (and my data is filled with only one class). If the results are wrong, does the proportion maintain? ...
1
vote
0answers
21 views

Deriving a loss function properly accounting for different error type costs

Consider a classification setting like a medical test: Not finding an existing health issue might be much worse (by a factor of 50) than assuming an issue when there is none. I.e. a setting in which ...
0
votes
0answers
11 views

Regression with Physical Loss

Imagine you want to use a DNN to solve this simple problem: estimate the frequency of (noisy) sine wave, similarly to what they did in this question: Neural network: estimating sine wave frequency. ...
0
votes
0answers
20 views

Variance estimator that is optimal under absolute loss

Given a random i.i.d. sample from a population with a finite variance $\sigma^2<\infty$, what estimator of $\sigma^2$ is optimal under absolute loss? $$ \arg\min_{\hat\sigma^{2}\in F}\mathbb{E}(|\...
1
vote
1answer
12 views

Find weight distribution in multiple term loss

I have a question if it is possible to find/learn the weight distribution in a multiple term loss where each weight models the importance of each term on the total loss. ...
0
votes
0answers
21 views

keras mse adam val_loss much lower than nrmse?

I'm trying to make sense of the keras.models.Sequential reported val_loss. It is a much better ...
0
votes
0answers
17 views

Training stuck at certain loss

I am new to NN and particularly to TensorFlow. I have implemented a simple network with 1 hidden layer and the loss output as below in the figure. I have read many blog posts, and forums to figure ...
1
vote
1answer
24 views

Choosing activation and loss functions in autoencoder

I am following this keras tutorial to create an autoencoder using the MNIST dataset. Here is the tutorial: https://blog.keras.io/building-autoencoders-in-keras.html. However, I am confused with the ...
0
votes
0answers
14 views

How to understand my CNN's training results?

I created a multi-label classification CNN to classify chest X-ray images into zero or more possible lung diseases. I've been doing some configuration tests on it and analyzing its results and I'm ...
0
votes
1answer
52 views

Bayesian estimator $\theta(x)$

Given a training set of $(X, Y )$'s where the $X$'s are the source variables and the $Y$'s are the targets, derive an estimator that minimizes the mean squared error between target values and ...
0
votes
1answer
42 views

imbalanced dataset - class weight vs weighted loss function

I'm working on a classification problem with a very imbalanced dataset. Many papers mention a "weighted cross-entropy loss function" or "focal loss with balancing weights". I can't find any of those ...
4
votes
1answer
152 views

Inconsistent Empirical Risk Minimization procedure, but why?

Given a random variable $Y$ and the typical squared loss function: $$L(Y,\hat{Y}) = (Y-\hat{Y})^2$$ the minimizer for expected loss $E[L(Y,\hat{Y})]$ is know to be the mean, $\hat{Y} = E[Y] = \mu$. ...
1
vote
0answers
11 views

Interpret learning curve results

I leveraged a pre-trained VGG network for a binary classification problem and am tuning with its hyper-parameters. I have generated curves for each setting (both accuracy and loss) below. I am unsure ...
1
vote
1answer
24 views

Symmetric Mean Absolute Error

I'm currently working in a ML problem, in which I have to predict the amount of returned quantities of some sold products. I'm working on some sort of weighted error metric which not only takes into ...
0
votes
0answers
8 views

Gradient of a convex loss for linear classifiers

Let L(w; x; y) be a convex loss function for a linear classifier w. Can you always express the gradient of ...
1
vote
0answers
50 views

Log of cost function converges to 0 and is extremely “noisy”

I'm writing my very first NN algorithm to solve a regression problem. I have a signal I would like to use a NN to find its correlation with eight other signals (along the line as f = f(x1, x2,..., x8))...
0
votes
1answer
37 views

derivation of objective function in linear regression

In linear regression, we have a very simple task. This is to measure a distance between Y and y_hat, where y_hat for sake of simplicity is multiplication of X and w. So we can say: Error = Y-y_hat = ...
0
votes
0answers
4 views

MLP: Validation loss and accuracy increases together

The graphs above are from an MLP Ensemble of 20 networks using the ADAM optimizer with an initial learning rate of 0.1 and one dropout layer dropping 10% of the neurons. From my understanding, the ...
2
votes
0answers
51 views

Different loss functions for estimation and cross-validation

Assume that the goal is to estimate some parameter $\theta$ (finite or infinite dimensional) based on some data available. Also, assume that there are other nuisance parameters are present in the ...
0
votes
0answers
10 views

Derivation of Optimal Classifier Under Hinge Loss

Let $Y \in \{-1, 1\}$ and $\mathbb{X} \in \mathbb{R}^p$ be a random variable and random vector, respectively. Consider the "hinge" loss function $L(a, b) = \max\{0, 1 - ab\}$. I am seeking the ...
0
votes
1answer
54 views

Quantile loss 50th is MAE, is it? [duplicate]

I'm not sure the above sentence is true, but I read it here, here and here that quantile loss function percentile 0.5 is MAE(mean absolute error), Is it true(Yes or No)? and How?
0
votes
0answers
16 views

Is it wrong to use categorical_crossentropy instead of binary_crossentropy for binary classification? [duplicate]

I was trying to build a CNN model. Data: 1) Consists of time series data of minute-wise water temperature to predict if there is high level of bacteria growth(label Y) in the water or not(label N). ...
3
votes
2answers
58 views

Terminology question: distinguishing two meanings of “loss function”?

I've heard people use "loss function" to refer to 2 different functions, with different type signatures: 1) A real-valued function of a label, $y$, and a prediction $\hat{y}$. 2) A real-valued ...
0
votes
0answers
8 views

Proper loss function to map latent feature space of two Autoencoders

I have two autoencoders, I would like to train them jointly to a point that their latent feature space represent same feature vectors. I currently keep deepest layer of both networks, compute ...
1
vote
1answer
62 views

Why does gradient descent HAVE to find the minimum as oppose to a change in the opposite direction

I have a question about the gradient descent step in neural networks. I fully understand the derivative step and taking the steps required to move in the direction that reduces the loss (finding the ...
1
vote
0answers
18 views

What loss function should be optimized so that the model error does not go above a certain MAPE threshold?

For the resulting regression model it is critically important not to go above the 10% threshold of MAPE (MAPE = abs((pred - fact)/fact). At the same time it is important to minimize the median of ...
0
votes
0answers
33 views

Training a GAN and loss curves

I am training a GAN with the discriminator and the generator being trained alternately for 2 epochs each. As expected I end up with oscillating loss curves, but they aren't damping as the training ...
1
vote
1answer
42 views

How to calculate pinball loss for quantiles and for point forecasts?

I have a few general questions about pinball loss: Is a pinball loss typically calculated for each point in the forecast horizon or is it calculated across all points in the forecast horizon? How is ...
1
vote
0answers
10 views

Modified Loss Function(s) for decorrelating neurons within a layer?

I'm looking for previous references on a specific topic. Does anyone know of any modified loss functions that incentivize a network to produce a diagonal neuron-to-neuron covariance matrix (averaging ...
2
votes
1answer
129 views

What is the intuition behind what makes dice coefficient handle imbalanced data?

I am writing my master thesis right now doing a project in deep learning doing semantic segmentation of MRI-images. Me and my partner have been looking at using dice loss instead of categorical cross-...
0
votes
1answer
24 views

How do filters affect the training loss in a convolutional neural network?

I am training a model, I am trying to lower the training loss. While testing different architectures I increased the number of filters to 128 from 64 - this reduced the training loss. I do not ...
0
votes
1answer
61 views

Behaviour of Keras BinaryCrossentropy() loss function

From the Keras source code, this is the definition of the BinaryCrossentropy() for the Numpy backend and the plot of the loss function for the values around logit 0 in both directions (appoaching to ...
0
votes
0answers
18 views

Fisher Information Matrix for regression problem

In DeepMind's paper [Overcoming catastrophic forgetting in neural networks]( https://arxiv.org/abs/1612.00796), elastic weight consolidation with Fisher Information Matrix is used to tackle the ...
2
votes
1answer
139 views

Cross Entropy with Log Softmax Activation

My question is about how is log softmax implemented in practice with the cross-entropy loss. Softmax gives values between 0 and 1, which means log softmax will give values between -infinity and 0. ...
0
votes
0answers
34 views

Custom Loss function Keras gradient

0 I am having trouble implementing a rather simple custom loss function, which calculates the output of the network times y_true, like so: ...
0
votes
0answers
10 views

Weight scheduling for combining multiple losses in Keras

In a multiple output network built with Keras, I have two loss functions which are combined with loss_weights option. Now I need to set the dynamic weight where the value would increase from zero to ...
0
votes
1answer
67 views

batch-training LSTM with pretrained & out-of-vocabulary word embeddings in keras

My goal is to batch-train an RNN LSTM mode using Stochastic Gradient Descent to predict named entities from labeled text in keras. The input to my model are word-sized units. I chose to represent ...
0
votes
0answers
6 views

loss not decreasing with convolutional nn [duplicate]

I have data with 1414 columns and 21 features (1414, 21). It is a regression problem and I am trying to use CNN as a model. In order to use a cnn, I reshaped my ...
0
votes
0answers
35 views

Derive the Bayes estimator of this loss function with indicator. bayesian inference statistic?

I need derive the Bayes estimator of this loss function with indicator. I have a sample of n distribution Bernoulli($\theta$) , and a priori distribution Beta(a,b). So, the a posteriori ...
1
vote
0answers
23 views

Softmax + CE vs Sigmoid + BCE for batched training with negative sampling, for training similarity properties

This is a follow up to this question Machine Learning: Should I use a categorical cross entropy or binary cross entropy loss for binary predictions? I am training cos similarity properties for ...
4
votes
1answer
50 views

Loss function for regression with uncertain labels

I have a regression task, for which I'm training a model with MSE loss. So for label $y$ and estimation $\hat{y}$ the loss is $$\ell(y,\hat{y})=(y-\hat{y})^2$$ However, there is an uncertainty in ...
0
votes
0answers
12 views

One-hot encoded fixed-size DNA sequence as output, what loss should I use?

Suppose I have some (deep learning) model that predicts $4 \times n$ (where $n$ fixed) matrices supposedly "unnormalized" one-hot encoded DNA sequences of length $n$, and that I know the ground truth ...
0
votes
0answers
79 views

Spearman correlation as an objective function for ranking

I am working on a machine learning problem, where the goal is to rank a list of objects. Importantly, in my problem ranking of all objects matters equally as opposed to a problem, where the ultimate ...
0
votes
0answers
17 views

$L_1$ vs $L_2$ loses in cross-validation

In which case one would estimate the performance of, let's say, regression, using $L_1$ loss function and in which case $L_{2}$ is better? I know that $L_{1}$ is more robust in a sense that it is ...
0
votes
0answers
21 views

Bounded Loss Function Proof (Hoeffding's Inequalities)

Let $ h \in \{ -1, 1\}^X $ be a fixed hypothesis and $ S \sim D^m $, where: $h$ = predictor $S$ = training sample $D$ = probability distribution $m$ = sample size $X$ = domain / feature space $L_D$ ...
1
vote
1answer
29 views

Log-Log regression and cost function

I have made a very siple linear regression model having used log-log tranformation for the y and one of the independent variables: log(y)=B0+log(X1)B1+X2B2 where B0 is the intercept and B1,B2 the ...
0
votes
0answers
18 views

Custom loss function for an unsupervised regression problem

I have recently started using neural networks in my research work and I am required to write a specific loss function for my problem. My network produces a parameter Q which should have a relationship ...
0
votes
0answers
24 views

Bounded loss function whose minimizer is the mean

Let $$ y = \operatorname*{argmin}_\hat{x} \operatorname*{E}_x L(|\hat{x} - x|) $$ where $L$ is a loss function. As noted here, if $f(s) = s^p$ then $p \rightarrow 0$ implies $y \rightarrow$ the ...
2
votes
0answers
51 views

Realistic/intuitive example where a nonadditive loss function is preferred over additive ones

This thread asks, Are loss functions necessarily additive in observations? As of now, one answer is in the negative. However, I am not aware of any practical examples of nonadditive loss functions ...