Questions tagged [cross-entropy]
A measure of the difference between two probability distributions for a given random variable or set of events.
232
questions
0
votes
0
answers
17
views
1
vote
0
answers
24
views
Is there an empirical rule for selecting the value of label smoothing?
I am wondering if there is any emperical rule for selecting the value of label smoothing when training a neural network. Let's define smoothed prediction targets in relation to a value $\epsilon$ to ...
- 11
1
vote
0
answers
18
views
Final Layer and Inference with CE vs BCE
I have read a similar question here: 1 neuron BCE loss VS 2 neurons CE loss that suggests there is no difference between softmax cross entropy loss and binary cross entropy loss, when choosing between ...
- 171
0
votes
0
answers
20
views
Why most works on Cityscapes don't use weighted cross-entropy?
Weight Cross-Entroy (WCE) helps to handle an imbalanced dataset, and Cityscapes is quite imbalanced as seen below:
If we check the best benchmarks on this dataset, most of the works use bare CE as a ...
- 130
0
votes
0
answers
37
views
Impact of L1 and L2 regularisation with cross-entropy loss
When we are dealing with Mean Square Error (MSE) loss function in optimization problems, we often add $L_1$ or $L_2$ penalty terms (or a combination of both) to the MSE loss function while training. ...
- 101
0
votes
0
answers
63
views
How can I get the Binary Cross Entropy from the Cross Entropy function for GANs
I got the definition of log-likelihood by Goodfellow's Deep Learning book:
\begin{equation}
\label{eq:loglikelihood}
\theta_{ML} = {argmax}\sum_{i=1}^{m} \log p_{model}(x_i; \theta).
\end{...
1
vote
1
answer
39
views
XGBoost Objective Derivation Problem
This is the loss function of XGBoost.
This is the Second-order approximation of the loss function.
Note:
\begin{equation} L^{(t)} \text{: cross entropy loss function.} \end{equation}
\begin{equation}...
- 13
0
votes
0
answers
46
views
Likelihood and cross-entropy: continuous case
I think it's pretty clear to me that average log-likelihood is equivalent to negative cross-entropy for discrete distributions, as shown here:
$$\frac{1}{N}\log\mathcal{L}(\theta) = \frac{1}{N}\log \...
8
votes
2
answers
345
views
Disadvantages of using a regression loss function in multi-class classification
Given $k > 2$ classes, consider the following loss function
$$
\sum_i||y^{(i)} - \hat y^{(i)}||^2
$$
Here $y^{(i)} \in \{0,1\}^k$ is the $i^{th}$ one-hot encoded true label and $\hat y^{(i)} \in [0,...
- 255
1
vote
1
answer
591
views
Confused with binary cross-entropy vs categorical cross-entropy
I have a dataset with 10 input categorical features and one output categorical feature with class 0 and 1. X_train follows a 3D array so I have done label encoding beforehand on the dataset.
I have ...
- 13
0
votes
0
answers
27
views
Is there an explanation for a classifier achieving high F1 scores, but having still high CrossEntropyLoss?
I am training a CNN classifier on a balanced dataset (around 35k examples for each label) with 13 classes. The model seems to achieve high F1 scores from the first batches; The F1 score for each class ...
- 3
0
votes
0
answers
51
views
Why do GANs use binary cross entropy in practice instead of the Jensen-Shannon divergence?
I believe I understand the JSD formula which is derived from the loss of an optimal discriminator. However, I do not understand why the discriminator is usually implemented with a binary cross entropy ...
- 13
0
votes
1
answer
54
views
Intuition behind Energy function in Restricted Boltzmann Machines
What does the Energy function in Restricted Boltzmann Machines represent intuitively?
I explain what I mean by the following example. If we look at cross-entropy $H(p,q_{\theta})=-\sum_{x}p(x)\log(q_{\...
- 3
2
votes
0
answers
161
views
Disadvantages of cross entropy loss comparing to SVM loss [closed]
What are some disadvantages and limitations of the cross entropy loss, especially compared with SVM loss/hinge loss? I am just looking for a general idea of when would one use SVM loss over cross ...
- 21
0
votes
1
answer
56
views
Is it possible to use softmax for anomaly detection?
Suppose we have a model for classification.
Normally the head of the model is a softmax over all the label/categories. Is it a good idea (that is being used in ...
- 165
2
votes
1
answer
91
views
What is a good binary_crossentropy or categorical_crossentropy?
I am training a binary classification model using LSTM and the training binary_crossentropy loss went from 0.84 to 0.83. I want to know what is a good binary_crossentropy loss value? There seems to be ...
- 65
2
votes
0
answers
56
views
Higher derivative of cross-entropy loss?
Has anyone worked out a formula for the higher derivatives of cross-entropy loss $J(z)$ with $z$ referring to logits? Loss is defined as follows:
$$J(p(z)) = -\sum_i q_i\log p(z)_i$$
$$p(z)_i=\frac{\...
- 5,281
1
vote
0
answers
401
views
Compute Gradient of Cross Entropy Loss with respect to its logits
I am in the freshman year of my master degree and I have been asked to compute the gradient of Cross Entropy Loss with respect to its logits. I should base the computation on Stanford notes page 4 ...
- 123
2
votes
2
answers
138
views
Minimizing KL-divergence and log-likelihood for generative machine learning models
I am reading a paper on quantum ML: A generative modeling approach for benchmarking and training shallow quantum circuits, where it is claimed that:
Following a standard approach from generative ...
- 133
0
votes
0
answers
69
views
Cross Entropy for sigmoid/tanh regression
My neural network has a tanh activation function for the output layer. It would be no problem to change this to sigmoid. The labels are values in the same range. By this I mean that the target value ...
0
votes
0
answers
110
views
Cross-Entropy VS sum of Binary-Cross-Entropies for multiclass
Most of the classification models that I've encountered so far perform classification using CE loss.
For example, if we have 2 possible classes and the GT class is 1, then:
the CE loss will be $-\log{\...
2
votes
1
answer
183
views
How to prove a loss function is classification-calibrated?
Assuming that I have a custom cross-entropy-like loss function defined as below, how can I prove that the loss function is classification-calibrated?
$$
L=-\frac{1}{n}\sum_{i=1}^n\sum_{j=1}^c w_j^{...
- 75
1
vote
1
answer
109
views
Why KL divergence close to zero when Q close to P?
I was understanding cross-entropy and ended up understanding KL divergence. I learnt Cross entropy is Entropy + KL Divergence:
H(P, Q) = H(P) + D_KL(P||Q)
Minimizing Cross-entropy means minimizing ...
0
votes
0
answers
23
views
How to modify cross entropy loss for soft labels? [duplicate]
I am trying to train a neural network on a soft target problem (note that my neural network has a softmax activation at the end). My labels are in the form
$[x_1, x_2, x_3, x_4, x_5, x_6]$ (each $x_i$ ...
1
vote
0
answers
66
views
Categorical Cross Entropy Loss Derivation
I understand the categorical cross entropy based loss function to be the following.
$$J(w) = \sum_{i=1}^ny_i\ln[P(y_i|x_i,w)]$$
where $$\ln\left[P(y_i|x_i,w)\right] = \sum_{i=k}^Kr_{ik}P(y_i=k|x_i,w)$$...
2
votes
0
answers
132
views
Why do we use cross entropy instead of Kullback-Leibler divergence as loss function? Why do we use forward KL divergence and not the reverse?
Was just having a discussion with a colleague, and realize I have the following questions about cross-entropy that is typically used in classification problems.
We know that cross entropy contains ...
- 445
1
vote
0
answers
15
views
Multi-label classification where predicting any one label is fine
I am working on a problem with muti-label classification, where, in contrast to the conventional requirement that the correct prediction of each label is expected, we just need to predict ANY ONE of ...
- 131
0
votes
0
answers
57
views
Classification on ordinal data
I have classification task, so I need to assign label from 5 classes to object. Because classes are ordinal $[0, 1, 2, 3, 4]$ a special architecture can be used rathen then ...
2
votes
1
answer
36
views
Splitting criterion of classification tree: Does the growth process come naturally to a stop?
With respect to growing a classification tree: Does growing with Gini or Cross-entropy (CE) imply we would grow the tree until every leaf is pure (in case of no other stopping criteria)? Put ...
- 635
2
votes
1
answer
176
views
Cross Entropy in PyTorch is different from what I learnt (Not about logit input, but about the loss for every node)
I know that the CrossEntropyLoss in Pytorch expects logits. I also know that the reduction argument in CrossEntropyLoss is to ...
0
votes
0
answers
19
views
Finding the best weights for sparse categorical cross entropy loss
In semantic segmentation and similar applications, sparse categorical cross entropy is often used as a loss function. Now it usually happens that samples are imbalanced. In my case, I have one class ...
- 121
0
votes
1
answer
320
views
Loss values above 1.0
I have a convolutional neural network for tensors classification in Pytorch. I am using Cross-Entropy Loss. My optimizer is Stochastic Gradient Descent and the learning rate is 0.0001. The accuracy of ...
- 41
4
votes
0
answers
508
views
Categorical cross-entropy vs Binary cross-entropy for multi-class classification with mixup
I understand that for multi-class classification the correct loss to use is categorical cross-entropy. However, when performing mixup as a regularisation technique two samples $(X_1, y_1)$ and $(X_2, ...
- 934
3
votes
1
answer
422
views
What value of predictions minimizes the binary cross entropy loss function? Is it 0.5?
I saw some examples of Autoencoders (on images) which use sigmoid as output layer and ...
- 195
5
votes
1
answer
335
views
Can proportions over variables be learned/predicted with Neural Networks using multiple-outputs?
I am interested in understanding how Neural Networks could be used to both learn from and predict proportions.
That is, say matrix $X$ is the training features data with $N$ cases and $k$ features. ...
- 51
1
vote
1
answer
27
views
cross entropy logistic test set
In logistic regression, the binary cross-entropy (logistic loss function) is defined as $$\ell (\boldsymbol{y}, \boldsymbol{\hat{y}}) = - \sum_{i=1}^n y_i \log \hat{y}_i + (1-y_i) \log (1-\hat{y}_i).$...
- 283
2
votes
1
answer
100
views
Is $-\sum_{n=1}^{N} \log(1+\exp(-t_ny_n)) $ the same loss as $\sum_{n=1}^{N}\{ t_n\log(y_n) + (1 - t_n)\log (1-y_n) \}$?
I am trying to understand different forms of loss functions. I get confused with the terms cross entropy-loss and negative log-likelihood losses. I have seen the two following definitions of cross ...
- 23
2
votes
1
answer
581
views
Relationship between cross entropy and average negative log likelihood
I'm trying to understand some machine learning theory background: specifically, the relationship between cross entropy loss and "negative log likelihood".
To start, I already fully ...
- 3,270
2
votes
1
answer
192
views
Why would a cross-entropy approach negative infinity?
I'm studying Deep Learning by Ian Goodfellow. In section 6.2.1.1 it says
For real-valued output variables, if the model can control the density of the output distribution (for example, by learning ...
- 23
1
vote
0
answers
96
views
Is label smoothing equivalent to adding a KL divergence term or a cross entropy term?
In the context of cross-entropy loss objectives for neural networks, I tend to think of label smoothing from the standpoint of directly manipulating the labels. This blog post explains how doing so is ...
- 549
0
votes
0
answers
40
views
cross entropy equivalence to to maximising log likelihood [duplicate]
I was wondering how can we show that cross-entropy is equivalent to maximising log likelihood of a training assuming the data can be modeled by this distribution:
- 1
1
vote
0
answers
90
views
Training a neural network to optimize parameters for a black box loss function
I have a black box loss function that is evaluated by an external stimulator. It accepts two vectors $x$ and $y$ , $L(x,y)$. I have the freedom to choose $y$ for a given $x$. Therefore, I would like ...
- 143
0
votes
0
answers
29
views
Cross entropy error: Poor modelling giving too much weight to unlikely events
I was reading this paper. link (page 5)
In this paper, there is a statement that goes like this:
To begin, cross entropy error is just one among many possible distance
measures between probability ...
- 279
1
vote
1
answer
325
views
Quantifying overlap between two categorical distributions with some non-identical categories
I'm looking for a way to measure the overlap (or general similarity) between two categorical distributions in which some of the categories are shared between each and some are not. For example, if the ...
- 365
0
votes
0
answers
30
views
Is MLE a theoretically sound method for uncertainty estimations?
I'm reading Evidential Deep Learning to Quantify Classification Uncertainty and it mentions that MLE is a flawed metric for evaluation uncertainty as it is a "frequentist" algorithm, but in ...
- 640
1
vote
0
answers
136
views
Why Binary Cross Entropy is more suitable than Categorical Cross Entropy in multi label classification?
I found this answers. But, I don't get fully. If I have three labels in multi label classification task, did BCE produce 3 separate outputs? Why we shouldn't use CCE?
In this Facebook work they claim ...
- 11
1
vote
1
answer
1k
views
Why is sparse categorical cross-entropy "sparse"?
This is just a conceptual question... It seems to me that what is called categorical cross-entropy should be called sparse because with the one hot encoding it creates a sparse matrix/tensor (whereas ...
- 113
2
votes
1
answer
2k
views
Use of ignore_index on CrossEntropyLoss() for text models
I have been using PyTorch's CrossEntropyLoss() on a Language Autoencoder. I noticed that most people use ignore_index for ignoring the pad token in loss calculation eg this.
From what I understand ...
- 23
1
vote
0
answers
42
views
How is the input to the encoder noted mathematically in a variational autoencoder?
We’re calculating VAE loss (the reconstruction part) as: $$E_q[\log \space p_\theta(x|z)]$$
I don’t know what the exact breakdown is, and how it became the cross entropy between the encoder’s input ...
- 205
1
vote
0
answers
29
views
Why optimizing the difference between $q(z|x)$, and $p(z|x)$ makes the latent variables "complete"?
In the past month I've spent most of my time digging deep into how neural networks work, from the basic idea (to estimate the true posterior $p(z|x)$, we create a variational posterior $q(z|x)$ - the ...
- 205