Questions tagged [cross-entropy]

A measure of the difference between two probability distributions for a given random variable or set of events.

Filter by
Sorted by
Tagged with
0 votes
0 answers
17 views

GoogleNet-LSTM, cross entropy loss does not decrease [duplicate]

...
user avatar
  • 1
1 vote
0 answers
24 views

Is there an empirical rule for selecting the value of label smoothing?

I am wondering if there is any emperical rule for selecting the value of label smoothing when training a neural network. Let's define smoothed prediction targets in relation to a value $\epsilon$ to ...
user avatar
1 vote
0 answers
18 views

Final Layer and Inference with CE vs BCE

I have read a similar question here: 1 neuron BCE loss VS 2 neurons CE loss that suggests there is no difference between softmax cross entropy loss and binary cross entropy loss, when choosing between ...
user avatar
  • 171
0 votes
0 answers
20 views

Why most works on Cityscapes don't use weighted cross-entropy?

Weight Cross-Entroy (WCE) helps to handle an imbalanced dataset, and Cityscapes is quite imbalanced as seen below: If we check the best benchmarks on this dataset, most of the works use bare CE as a ...
user avatar
0 votes
0 answers
37 views

Impact of L1 and L2 regularisation with cross-entropy loss

When we are dealing with Mean Square Error (MSE) loss function in optimization problems, we often add $L_1$ or $L_2$ penalty terms (or a combination of both) to the MSE loss function while training. ...
user avatar
0 votes
0 answers
63 views

How can I get the Binary Cross Entropy from the Cross Entropy function for GANs

I got the definition of log-likelihood by Goodfellow's Deep Learning book: \begin{equation} \label{eq:loglikelihood} \theta_{ML} = {argmax}\sum_{i=1}^{m} \log p_{model}(x_i; \theta). \end{...
user avatar
1 vote
1 answer
39 views

XGBoost Objective Derivation Problem

This is the loss function of XGBoost. This is the Second-order approximation of the loss function. Note: \begin{equation} L^{(t)} \text{: cross entropy loss function.} \end{equation} \begin{equation}...
user avatar
0 votes
0 answers
46 views

Likelihood and cross-entropy: continuous case

I think it's pretty clear to me that average log-likelihood is equivalent to negative cross-entropy for discrete distributions, as shown here: $$\frac{1}{N}\log\mathcal{L}(\theta) = \frac{1}{N}\log \...
user avatar
8 votes
2 answers
345 views

Disadvantages of using a regression loss function in multi-class classification

Given $k > 2$ classes, consider the following loss function $$ \sum_i||y^{(i)} - \hat y^{(i)}||^2 $$ Here $y^{(i)} \in \{0,1\}^k$ is the $i^{th}$ one-hot encoded true label and $\hat y^{(i)} \in [0,...
user avatar
1 vote
1 answer
591 views

Confused with binary cross-entropy vs categorical cross-entropy

I have a dataset with 10 input categorical features and one output categorical feature with class 0 and 1. X_train follows a 3D array so I have done label encoding beforehand on the dataset. I have ...
user avatar
  • 13
0 votes
0 answers
27 views

Is there an explanation for a classifier achieving high F1 scores, but having still high CrossEntropyLoss?

I am training a CNN classifier on a balanced dataset (around 35k examples for each label) with 13 classes. The model seems to achieve high F1 scores from the first batches; The F1 score for each class ...
user avatar
  • 3
0 votes
0 answers
51 views

Why do GANs use binary cross entropy in practice instead of the Jensen-Shannon divergence?

I believe I understand the JSD formula which is derived from the loss of an optimal discriminator. However, I do not understand why the discriminator is usually implemented with a binary cross entropy ...
user avatar
0 votes
1 answer
54 views

Intuition behind Energy function in Restricted Boltzmann Machines

What does the Energy function in Restricted Boltzmann Machines represent intuitively? I explain what I mean by the following example. If we look at cross-entropy $H(p,q_{\theta})=-\sum_{x}p(x)\log(q_{\...
user avatar
  • 3
2 votes
0 answers
161 views

Disadvantages of cross entropy loss comparing to SVM loss [closed]

What are some disadvantages and limitations of the cross entropy loss, especially compared with SVM loss/hinge loss? I am just looking for a general idea of when would one use SVM loss over cross ...
user avatar
0 votes
1 answer
56 views

Is it possible to use softmax for anomaly detection?

Suppose we have a model for classification. Normally the head of the model is a softmax over all the label/categories. Is it a good idea (that is being used in ...
user avatar
2 votes
1 answer
91 views

What is a good binary_crossentropy or categorical_crossentropy?

I am training a binary classification model using LSTM and the training binary_crossentropy loss went from 0.84 to 0.83. I want to know what is a good binary_crossentropy loss value? There seems to be ...
user avatar
2 votes
0 answers
56 views

Higher derivative of cross-entropy loss?

Has anyone worked out a formula for the higher derivatives of cross-entropy loss $J(z)$ with $z$ referring to logits? Loss is defined as follows: $$J(p(z)) = -\sum_i q_i\log p(z)_i$$ $$p(z)_i=\frac{\...
user avatar
1 vote
0 answers
401 views

Compute Gradient of Cross Entropy Loss with respect to its logits

I am in the freshman year of my master degree and I have been asked to compute the gradient of Cross Entropy Loss with respect to its logits. I should base the computation on Stanford notes page 4 ...
user avatar
2 votes
2 answers
138 views

Minimizing KL-divergence and log-likelihood for generative machine learning models

I am reading a paper on quantum ML: A generative modeling approach for benchmarking and training shallow quantum circuits, where it is claimed that: Following a standard approach from generative ...
user avatar
  • 133
0 votes
0 answers
69 views

Cross Entropy for sigmoid/tanh regression

My neural network has a tanh activation function for the output layer. It would be no problem to change this to sigmoid. The labels are values in the same range. By this I mean that the target value ...
user avatar
0 votes
0 answers
110 views

Cross-Entropy VS sum of Binary-Cross-Entropies for multiclass

Most of the classification models that I've encountered so far perform classification using CE loss. For example, if we have 2 possible classes and the GT class is 1, then: the CE loss will be $-\log{\...
user avatar
2 votes
1 answer
183 views

How to prove a loss function is classification-calibrated?

Assuming that I have a custom cross-entropy-like loss function defined as below, how can I prove that the loss function is classification-calibrated? $$ L=-\frac{1}{n}\sum_{i=1}^n\sum_{j=1}^c w_j^{...
user avatar
1 vote
1 answer
109 views

Why KL divergence close to zero when Q close to P?

I was understanding cross-entropy and ended up understanding KL divergence. I learnt Cross entropy is Entropy + KL Divergence: H(P, Q) = H(P) + D_KL(P||Q) Minimizing Cross-entropy means minimizing ...
user avatar
0 votes
0 answers
23 views

How to modify cross entropy loss for soft labels? [duplicate]

I am trying to train a neural network on a soft target problem (note that my neural network has a softmax activation at the end). My labels are in the form $[x_1, x_2, x_3, x_4, x_5, x_6]$ (each $x_i$ ...
user avatar
1 vote
0 answers
66 views

Categorical Cross Entropy Loss Derivation

I understand the categorical cross entropy based loss function to be the following. $$J(w) = \sum_{i=1}^ny_i\ln[P(y_i|x_i,w)]$$ where $$\ln\left[P(y_i|x_i,w)\right] = \sum_{i=k}^Kr_{ik}P(y_i=k|x_i,w)$$...
user avatar
2 votes
0 answers
132 views

Why do we use cross entropy instead of Kullback-Leibler divergence as loss function? Why do we use forward KL divergence and not the reverse?

Was just having a discussion with a colleague, and realize I have the following questions about cross-entropy that is typically used in classification problems. We know that cross entropy contains ...
user avatar
  • 445
1 vote
0 answers
15 views

Multi-label classification where predicting any one label is fine

I am working on a problem with muti-label classification, where, in contrast to the conventional requirement that the correct prediction of each label is expected, we just need to predict ANY ONE of ...
user avatar
  • 131
0 votes
0 answers
57 views

Classification on ordinal data

I have classification task, so I need to assign label from 5 classes to object. Because classes are ordinal $[0, 1, 2, 3, 4]$ a special architecture can be used rathen then ...
user avatar
2 votes
1 answer
36 views

Splitting criterion of classification tree: Does the growth process come naturally to a stop?

With respect to growing a classification tree: Does growing with Gini or Cross-entropy (CE) imply we would grow the tree until every leaf is pure (in case of no other stopping criteria)? Put ...
user avatar
  • 635
2 votes
1 answer
176 views

Cross Entropy in PyTorch is different from what I learnt (Not about logit input, but about the loss for every node)

I know that the CrossEntropyLoss in Pytorch expects logits. I also know that the reduction argument in CrossEntropyLoss is to ...
user avatar
0 votes
0 answers
19 views

Finding the best weights for sparse categorical cross entropy loss

In semantic segmentation and similar applications, sparse categorical cross entropy is often used as a loss function. Now it usually happens that samples are imbalanced. In my case, I have one class ...
user avatar
0 votes
1 answer
320 views

Loss values above 1.0

I have a convolutional neural network for tensors classification in Pytorch. I am using Cross-Entropy Loss. My optimizer is Stochastic Gradient Descent and the learning rate is 0.0001. The accuracy of ...
user avatar
  • 41
4 votes
0 answers
508 views

Categorical cross-entropy vs Binary cross-entropy for multi-class classification with mixup

I understand that for multi-class classification the correct loss to use is categorical cross-entropy. However, when performing mixup as a regularisation technique two samples $(X_1, y_1)$ and $(X_2, ...
user avatar
  • 934
3 votes
1 answer
422 views

What value of predictions minimizes the binary cross entropy loss function? Is it 0.5?

I saw some examples of Autoencoders (on images) which use sigmoid as output layer and ...
user avatar
  • 195
5 votes
1 answer
335 views

Can proportions over variables be learned/predicted with Neural Networks using multiple-outputs?

I am interested in understanding how Neural Networks could be used to both learn from and predict proportions. That is, say matrix $X$ is the training features data with $N$ cases and $k$ features. ...
user avatar
  • 51
1 vote
1 answer
27 views

cross entropy logistic test set

In logistic regression, the binary cross-entropy (logistic loss function) is defined as $$\ell (\boldsymbol{y}, \boldsymbol{\hat{y}}) = - \sum_{i=1}^n y_i \log \hat{y}_i + (1-y_i) \log (1-\hat{y}_i).$...
user avatar
  • 283
2 votes
1 answer
100 views

Is $-\sum_{n=1}^{N} \log(1+\exp(-t_ny_n)) $ the same loss as $\sum_{n=1}^{N}\{ t_n\log(y_n) + (1 - t_n)\log (1-y_n) \}$?

I am trying to understand different forms of loss functions. I get confused with the terms cross entropy-loss and negative log-likelihood losses. I have seen the two following definitions of cross ...
user avatar
  • 23
2 votes
1 answer
581 views

Relationship between cross entropy and average negative log likelihood

I'm trying to understand some machine learning theory background: specifically, the relationship between cross entropy loss and "negative log likelihood". To start, I already fully ...
user avatar
2 votes
1 answer
192 views

Why would a cross-entropy approach negative infinity?

I'm studying Deep Learning by Ian Goodfellow. In section 6.2.1.1 it says For real-valued output variables, if the model can control the density of the output distribution (for example, by learning ...
user avatar
  • 23
1 vote
0 answers
96 views

Is label smoothing equivalent to adding a KL divergence term or a cross entropy term?

In the context of cross-entropy loss objectives for neural networks, I tend to think of label smoothing from the standpoint of directly manipulating the labels. This blog post explains how doing so is ...
user avatar
0 votes
0 answers
40 views

cross entropy equivalence to to maximising log likelihood [duplicate]

I was wondering how can we show that cross-entropy is equivalent to maximising log likelihood of a training assuming the data can be modeled by this distribution:
user avatar
  • 1
1 vote
0 answers
90 views

Training a neural network to optimize parameters for a black box loss function

I have a black box loss function that is evaluated by an external stimulator. It accepts two vectors $x$ and $y$ , $L(x,y)$. I have the freedom to choose $y$ for a given $x$. Therefore, I would like ...
user avatar
  • 143
0 votes
0 answers
29 views

Cross entropy error: Poor modelling giving too much weight to unlikely events

I was reading this paper. link (page 5) In this paper, there is a statement that goes like this: To begin, cross entropy error is just one among many possible distance measures between probability ...
user avatar
  • 279
1 vote
1 answer
325 views

Quantifying overlap between two categorical distributions with some non-identical categories

I'm looking for a way to measure the overlap (or general similarity) between two categorical distributions in which some of the categories are shared between each and some are not. For example, if the ...
user avatar
0 votes
0 answers
30 views

Is MLE a theoretically sound method for uncertainty estimations?

I'm reading Evidential Deep Learning to Quantify Classification Uncertainty and it mentions that MLE is a flawed metric for evaluation uncertainty as it is a "frequentist" algorithm, but in ...
user avatar
1 vote
0 answers
136 views

Why Binary Cross Entropy is more suitable than Categorical Cross Entropy in multi label classification?

I found this answers. But, I don't get fully. If I have three labels in multi label classification task, did BCE produce 3 separate outputs? Why we shouldn't use CCE? In this Facebook work they claim ...
user avatar
1 vote
1 answer
1k views

Why is sparse categorical cross-entropy "sparse"?

This is just a conceptual question... It seems to me that what is called categorical cross-entropy should be called sparse because with the one hot encoding it creates a sparse matrix/tensor (whereas ...
user avatar
2 votes
1 answer
2k views

Use of ignore_index on CrossEntropyLoss() for text models

I have been using PyTorch's CrossEntropyLoss() on a Language Autoencoder. I noticed that most people use ignore_index for ignoring the pad token in loss calculation eg this. From what I understand ...
user avatar
  • 23
1 vote
0 answers
42 views

How is the input to the encoder noted mathematically in a variational autoencoder?

We’re calculating VAE loss (the reconstruction part) as: $$E_q[\log \space p_\theta(x|z)]$$ I don’t know what the exact breakdown is, and how it became the cross entropy between the encoder’s input ...
user avatar
1 vote
0 answers
29 views

Why optimizing the difference between $q(z|x)$, and $p(z|x)$ makes the latent variables "complete"?

In the past month I've spent most of my time digging deep into how neural networks work, from the basic idea (to estimate the true posterior $p(z|x)$, we create a variational posterior $q(z|x)$ - the ...
user avatar

1
2 3 4 5