# Questions tagged [cross-entropy]

A measure of the difference between two probability distributions for a given random variable or set of events.

232 questions
Filter by
Sorted by
Tagged with
17 views

...
1 vote
24 views

### Is there an empirical rule for selecting the value of label smoothing?

I am wondering if there is any emperical rule for selecting the value of label smoothing when training a neural network. Let's define smoothed prediction targets in relation to a value $\epsilon$ to ...
• 11
1 vote
18 views

### Final Layer and Inference with CE vs BCE

I have read a similar question here: 1 neuron BCE loss VS 2 neurons CE loss that suggests there is no difference between softmax cross entropy loss and binary cross entropy loss, when choosing between ...
• 171
20 views

### Why most works on Cityscapes don't use weighted cross-entropy?

Weight Cross-Entroy (WCE) helps to handle an imbalanced dataset, and Cityscapes is quite imbalanced as seen below: If we check the best benchmarks on this dataset, most of the works use bare CE as a ...
37 views

### Impact of L1 and L2 regularisation with cross-entropy loss

When we are dealing with Mean Square Error (MSE) loss function in optimization problems, we often add $L_1$ or $L_2$ penalty terms (or a combination of both) to the MSE loss function while training. ...
• 101
63 views

### How can I get the Binary Cross Entropy from the Cross Entropy function for GANs

I got the definition of log-likelihood by Goodfellow's Deep Learning book: \label{eq:loglikelihood} \theta_{ML} = {argmax}\sum_{i=1}^{m} \log p_{model}(x_i; \theta). \end{...
1 vote
39 views

### XGBoost Objective Derivation Problem

This is the loss function of XGBoost. This is the Second-order approximation of the loss function. Note: $$L^{(t)} \text{: cross entropy loss function.}$$ ...
• 13
46 views

• 5,281
1 vote
401 views

### Compute Gradient of Cross Entropy Loss with respect to its logits

I am in the freshman year of my master degree and I have been asked to compute the gradient of Cross Entropy Loss with respect to its logits. I should base the computation on Stanford notes page 4 ...
• 123
138 views

### Minimizing KL-divergence and log-likelihood for generative machine learning models

I am reading a paper on quantum ML: A generative modeling approach for benchmarking and training shallow quantum circuits, where it is claimed that: Following a standard approach from generative ...
• 133
69 views

### Cross Entropy for sigmoid/tanh regression

My neural network has a tanh activation function for the output layer. It would be no problem to change this to sigmoid. The labels are values in the same range. By this I mean that the target value ...
110 views

• 283
100 views

### Is $-\sum_{n=1}^{N} \log(1+\exp(-t_ny_n))$ the same loss as $\sum_{n=1}^{N}\{ t_n\log(y_n) + (1 - t_n)\log (1-y_n) \}$?

I am trying to understand different forms of loss functions. I get confused with the terms cross entropy-loss and negative log-likelihood losses. I have seen the two following definitions of cross ...
• 23
581 views

### Relationship between cross entropy and average negative log likelihood

I'm trying to understand some machine learning theory background: specifically, the relationship between cross entropy loss and "negative log likelihood". To start, I already fully ...
192 views

### Why would a cross-entropy approach negative infinity?

I'm studying Deep Learning by Ian Goodfellow. In section 6.2.1.1 it says For real-valued output variables, if the model can control the density of the output distribution (for example, by learning ...
• 23
1 vote
96 views

### Is label smoothing equivalent to adding a KL divergence term or a cross entropy term?

In the context of cross-entropy loss objectives for neural networks, I tend to think of label smoothing from the standpoint of directly manipulating the labels. This blog post explains how doing so is ...
40 views

### cross entropy equivalence to to maximising log likelihood [duplicate]

I was wondering how can we show that cross-entropy is equivalent to maximising log likelihood of a training assuming the data can be modeled by this distribution:
1 vote
90 views

### Training a neural network to optimize parameters for a black box loss function

I have a black box loss function that is evaluated by an external stimulator. It accepts two vectors $x$ and $y$ , $L(x,y)$. I have the freedom to choose $y$ for a given $x$. Therefore, I would like ...
• 143
29 views

### Cross entropy error: Poor modelling giving too much weight to unlikely events

I was reading this paper. link (page 5) In this paper, there is a statement that goes like this: To begin, cross entropy error is just one among many possible distance measures between probability ...
• 279
1 vote
325 views

### Quantifying overlap between two categorical distributions with some non-identical categories

I'm looking for a way to measure the overlap (or general similarity) between two categorical distributions in which some of the categories are shared between each and some are not. For example, if the ...
30 views

### Is MLE a theoretically sound method for uncertainty estimations?

I'm reading Evidential Deep Learning to Quantify Classification Uncertainty and it mentions that MLE is a flawed metric for evaluation uncertainty as it is a "frequentist" algorithm, but in ...
• 640
1 vote
136 views

### Why Binary Cross Entropy is more suitable than Categorical Cross Entropy in multi label classification?

I found this answers. But, I don't get fully. If I have three labels in multi label classification task, did BCE produce 3 separate outputs? Why we shouldn't use CCE? In this Facebook work they claim ...
1 vote
1k views

### Why is sparse categorical cross-entropy "sparse"?

This is just a conceptual question... It seems to me that what is called categorical cross-entropy should be called sparse because with the one hot encoding it creates a sparse matrix/tensor (whereas ...
• 113
2k views

### Use of ignore_index on CrossEntropyLoss() for text models

I have been using PyTorch's CrossEntropyLoss() on a Language Autoencoder. I noticed that most people use ignore_index for ignoring the pad token in loss calculation eg this. From what I understand ...
• 23
1 vote
We’re calculating VAE loss (the reconstruction part) as: $$E_q[\log \space p_\theta(x|z)]$$ I don’t know what the exact breakdown is, and how it became the cross entropy between the encoder’s input ...
### Why optimizing the difference between $q(z|x)$, and $p(z|x)$ makes the latent variables "complete"?
In the past month I've spent most of my time digging deep into how neural networks work, from the basic idea (to estimate the true posterior $p(z|x)$, we create a variational posterior $q(z|x)$ - the ...