Skip to main content

Questions tagged [cross-entropy]

A measure of the difference between two probability distributions for a given random variable or set of events.

Filter by
Sorted by
Tagged with
2 votes
0 answers
92 views

Link between Cross-entropy and MLE

There are numerous material that show the relationship between MLE and cross-entropy. Typically, these are the steps taken to show the relationship for a I.I.D data generating process $D = (X,Y)$: $$ ...
spie227's user avatar
  • 65
0 votes
0 answers
17 views

Prove Decreasing Cross Entropy of outputs with Decreasing KL divergence of inputs

I am trying to prove the inequality $H(gt, y) > H(gt, y_1) > H(gt, y_2)$, given that $D_{KL}(x, x_1) > D_{KL}(x_1, x_2)$, where $y = f(x)$, $gt$ - ground truth, $D_{KL}$ - KL divergence, $H$ ...
user412867's user avatar
0 votes
0 answers
52 views

I'm struggling to connect cross-entropy minimization with likelihood maximization

I am struggling to understand the equivalence between cross-entropy loss and maximum likelihood estimation. I'll present the standard derivation of this equivalence for the simple case of binary ...
Alessandro Power's user avatar
2 votes
1 answer
141 views

Which form of cross-entropy loss is correct?

For classification problems with more than two classes, I've seen these two forms of cross-entropy loss: -$\sum_k y_k \log(a_k)$ -$\sum_k y_k \log(a_k) + (1-y_k) \log(1-a_k)$ Here $y_i$ are the true ...
theQman's user avatar
  • 677
0 votes
0 answers
151 views

Is it better to use KL Divergence for soft labels instead of cross entropy?

So I was going through this paper: Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2017) by Junnan Li et al., in this they perform contrastive loss using KL ...
mutli-arm-bandit's user avatar
2 votes
0 answers
154 views

Why is cross-entropy increasing with accuracy? [closed]

I'm making an implementation of the softmax regression and I'm struggling to understand the nature behind the problem of increasing value of Cross-Entropy: $H(y_i, p_i)=-\sum_{i=1}^C y_i log(p_i)$, ...
JoshJohnson's user avatar
1 vote
1 answer
301 views

Understanding shannon entropy and computation with scipy.stats.entropy

I am trying to understand the shannon entropy better. By definition, the shannon entropy is calculated as H = -sum(pk * log(pk)). I am using the scipy.stats.entropy formula and I am running the ...
GGChe's user avatar
  • 165
0 votes
1 answer
139 views

How (or can) you formulate the Fisher information matrix in terms of a loss function, specifically cross-entropy loss?

I recently saw the following formulation of the Fisher information matrix in a paper on Transformer pruning: $$ \mathcal{I} := \frac{1}{|D|} \sum_{(x,y) \in D} \left( \frac{\partial \mathcal{L}(x,y;1)}...
premed's user avatar
  • 1
0 votes
0 answers
43 views

Concentration Inequality for cross-entropy

I am currently trying to estimate the cross-entropy between two distributions with densities $p$ and $q$. $$ \ell = -\mathbb{E}_{x\sim p(x) }[\log q(x)] $$ I am using a Monte-Carlo estimate: $$ \hat{\...
Nick Bishop's user avatar
0 votes
0 answers
57 views

Minimizing cross entropy over a restricted domain?

Suppose $f(x;q)$ is the true distribution. The support of the random variable $X$ is $\Omega$. Suppose, I am interested in a particular subset of $\Xi \subset \Omega$. I would like to minimize the ...
entropy's user avatar
  • 19
1 vote
1 answer
141 views

What exactly is the problem with overconfident predictions?

Say I have a neural network that classifies images by training to minimise cross-entropy loss with one-hot encoded training labels. It is often seen that such neural networks are 'overconfident', with ...
Danny Duberstein's user avatar
0 votes
1 answer
66 views

Is CE(X, Y) equivalent to H(X) + H(Y)?

From my understanding mutual information can be defined in the following ways: [1]: $I(X;Y)=H(X)+H(Y)-H(X,Y)$ where $H(X), H(Y)$ are marginal entropies and $H(X,Y)$ is the joint entropy. [2]: $I(X;Y)=...
Rui's user avatar
  • 3
1 vote
0 answers
34 views

Decision boundary for Cross entropy loss and Least square loss

We can see the source in this paper. My question is that why cross entropy loss has a boundary line in slope but least square loss has horizontal boundary. Can somebody explain?
batuman's user avatar
  • 483
0 votes
1 answer
61 views

Derivation of cross entropy loss in machine learning

Given a dataset $\mathcal{D} = \{ (x_1, y_1),\cdots, (x_n, y_n)\}$, let's say we want to approximate the conditional probability $p(y|x)$, and we parameterized it as $p_{\theta}(y|x)$. So,for a ...
UESTCfresh's user avatar
1 vote
1 answer
338 views

can we use binary cross entropy with labels -1 and 1?

Binary cross entropy is written as follows: \begin{equation} \mathcal{L} = -y\log\left(\hat{y}\right)-(1-y)\log\left(1-\hat{y}\right) \end{equation} In every reference that I read, when using binary ...
andryan86's user avatar
  • 127
2 votes
1 answer
671 views

Meaning of non-{0,1} labels in binary cross entropy?

Binary cross entropy is normally used in situations where the "true" result or label is one of two values (hence "binary"), typically encoded as 0 and 1. However, the documentation ...
R.M.'s user avatar
  • 1,015
1 vote
1 answer
121 views

Calculating KL divergence with entropy and cross entropy for VAEs

When looking at implementations of VAE's online, specifically the KL divergence loss, the formula used is: $$ KL\hspace{1mm} Loss = -\frac{1}{2}(1+\log{\sigma^2}-\mu^2-\sigma^2) $$ or some variation ...
pyrrosk's user avatar
  • 33
1 vote
1 answer
75 views

Very balanced dataset and a multiclass classification problem, no context behind the inputs. Which evaluation metric to use?

I have constructed a simple neural network model, for a classification problem, with 10 target classes where an input (with some number of features) is to be classified to only one of the 10 classes. ...
creamedcheese83's user avatar
1 vote
0 answers
22 views

Log base in Cross Entropy Loss [duplicate]

What is the base for the logarithm used in the cross entropy loss (while doing multiclass classification's backpropagation)? Is it e, 2, or 10?
Sachin's user avatar
  • 111
2 votes
1 answer
603 views

Why is the cross entropy of the same probability distribution not 0?

From what I've been reading, if there is no underlying difference between the 2 probabilities distributions we would have perfect entropy. I'm putting an example below. Can anybody explain why the ...
julian lagier's user avatar
1 vote
0 answers
114 views

How does the cross entropy loss function interact with the final layer of a neural network?

I am having trouble understanding how the result of categorical cross entropy loss can be used to calculate the gradient for all of the weights. The output of cross entropy function is the sum of all ...
Nick's user avatar
  • 33
2 votes
0 answers
187 views

Understanding intuitive difference between KL divergence and Cross entropy

I know there are related questions already asked, for example this one. I also know the following: KL divergence $D_{KL}(P\Vert Q)$ is given as: $$\begin{align} D_{KL}(P\Vert Q) & = -\sum_xP(x)\...
Mahesha999's user avatar
2 votes
1 answer
372 views

Derivative error with respect to bias in binary cross entropy

I will do research using NN with 1 hidden layer. To calculate loss using binary cross entropy and for the activation function using sigmoid. I found the derivative formula from Sadowski, 2016 (link: ...
Andryan's user avatar
  • 47
5 votes
1 answer
255 views

Does logistic regression try to predict the true conditional P(Y|X)?

Consider a binary classification dataset (X, Y), generated according to some unknown distribution $P(X, Y)$. I have a question about models which output probabilities by minimizing the cross-entropy ...
usual me's user avatar
  • 1,247
3 votes
2 answers
425 views

Understanding StatQuest video: why cross entropy is used over Sum Squared Error

I was watching cross entropy video from StatQuest. While explaining why to use cross entropy over SSE in multi output scenario with softmax output activation, Josh gives this graph of both losses: He ...
Mahesha999's user avatar
1 vote
0 answers
17 views

Relate cross-entropy formal definition to the cross-entropy loss [duplicate]

Cross entropy for a random variable $x \sim p$ and a distribution $q$ is defined as: $$H(p,q) = -\sum_{x\in\mathcal{X}} p(x)\log q(x) = \mathbb{E}(\log q(x))$$ $\mathcal{X}$ is all possible values ...
rando's user avatar
  • 308
0 votes
0 answers
95 views

Add Bias to classification after training

I have a dataset with classes [a, b] where during training I have made sure that the dataset is equally balanced. I have trained the network using cross-entropy loss with equal importance. I am able ...
JakobVinkas's user avatar
1 vote
0 answers
45 views

Genetic Algorithm as engine for Variational Inference?

I'm curious if anyone has used, heard of, or otherwise considered using Genetic Algorithms as an engine for Variational Inference (VI)? My understanding of VI is that it's an optimization algorithm, ...
jbuddy_13's user avatar
  • 3,372
2 votes
2 answers
149 views

Surprisal in rankings

I'm looking for some metric of surprisal when comparing ranked lists - things along the lines of (eg) the rankings in a marathon race, or the times in the race. Intuitively, in a race with 100 people, ...
Alex I's user avatar
  • 1,153
4 votes
0 answers
437 views

Cross-entropy vs dot product

Performance of classification algorithms is quantified by comparing the predicted probability distribution of the labels $q$ to the true probability $p$, which is commonly a vector of zeros for all ...
Aleksejs Fomins's user avatar
1 vote
1 answer
133 views

How do machine learning algorithms handle classification labels?

I am working on a domain adaptation problem, where the default is a classification problem. I have worked exclusively with regression problems until now, so I am kind of thrown for a loop when it ...
Scott's user avatar
  • 13
1 vote
0 answers
209 views

How to explain the high accuracy and F1 score on the test set with a huge binary crossentropy loss?

I'll provide a little of introduction based on my example. I have a small collection of RGB (but 'gray-looking') brain MRI photos, divided into 2 classes: healthy and tumor. My data split looks like ...
Karolina Świergała's user avatar
2 votes
1 answer
151 views

Why is it called the cross-entropy of q relative to p, not p relative to q?

I'm looking into the definition of cross entropy from wikipedia. https://en.wikipedia.org/wiki/Cross_entropy Cross entropy is not symmetric, so I think for sure it shouldn't be called cross entropy ...
user900476's user avatar
1 vote
1 answer
600 views

Calculating the variance of softmax

I'm working through Dive Into Deep Learning right now and am struggling with the following question: We can explore the connection between exponential families and the softmax in some more depth. ...
jimac82's user avatar
  • 21
2 votes
1 answer
298 views

Cross entropy of a random variable or a probability distribution function? [duplicate]

I'm looking into the wikipedia page of cross entropy. https://en.wikipedia.org/wiki/Cross_entropy $$H(p,q)=-\sum_{x\in \mathcal{X}} p(x)\log q(x)$$ It can be written as $$H(p,q) = H(p) + D_{KL} (p||q)$...
user900476's user avatar
0 votes
0 answers
18 views

What performance we get with same data combines with different datasets?

Suppose that we have dataset of special kind of cat. We are going to train a model on combination of the cat a the car! Suppose that in this model we will get a performance ( precision,recall or...) X ...
Mahdi Amrollahi's user avatar
0 votes
0 answers
20 views

GoogleNet-LSTM, cross entropy loss does not decrease [duplicate]

...
makala's user avatar
  • 11
3 votes
0 answers
500 views

Is there an empirical rule for selecting the value of label smoothing?

I am wondering if there is any emperical rule for selecting the value of label smoothing when training a neural network. Let's define smoothed prediction targets in relation to a value $\epsilon$ to ...
thiaamak's user avatar
1 vote
0 answers
575 views

Final Layer and Inference with CE vs BCE

I have read a similar question here: 1 neuron BCE loss VS 2 neurons CE loss that suggests there is no difference between softmax cross entropy loss and binary cross entropy loss, when choosing between ...
Anonymous's user avatar
  • 181
0 votes
0 answers
377 views

Why most works on Cityscapes don't use weighted cross-entropy?

Weight Cross-Entroy (WCE) helps to handle an imbalanced dataset, and Cityscapes is quite imbalanced as seen below: If we check the best benchmarks on this dataset, most of the works use bare CE as a ...
Rafael Toledo's user avatar
2 votes
1 answer
974 views

Impact of L1 and L2 regularisation with cross-entropy loss

When we are dealing with Mean Square Error (MSE) loss function in optimization problems, we often add $L_1$ or $L_2$ penalty terms (or a combination of both) to the MSE loss function while training. ...
Aravind G.'s user avatar
0 votes
0 answers
274 views

How can I get the Binary Cross Entropy from the Cross Entropy function for GANs

I got the definition of log-likelihood by Goodfellow's Deep Learning book: \begin{equation} \label{eq:loglikelihood} \theta_{ML} = {argmax}\sum_{i=1}^{m} \log p_{model}(x_i; \theta). \end{...
Lucas Lima de Sousa's user avatar
1 vote
1 answer
235 views

XGBoost Objective Derivation Problem

This is the loss function of XGBoost. This is the Second-order approximation of the loss function. Note: \begin{equation} L^{(t)} \text{: cross entropy loss function.} \end{equation} \begin{equation}...
ChrisChu's user avatar
2 votes
1 answer
97 views

Likelihood and cross-entropy: continuous case

I think it's pretty clear to me that average log-likelihood is equivalent to negative cross-entropy for discrete distributions, as shown here: $$\frac{1}{N}\log\mathcal{L}(\theta) = \frac{1}{N}\log \...
Alex Zakharov's user avatar
14 votes
2 answers
2k views

Disadvantages of using a regression loss function in multi-class classification

Given $k > 2$ classes, consider the following loss function $$ \sum_i||y^{(i)} - \hat y^{(i)}||^2 $$ Here $y^{(i)} \in \{0,1\}^k$ is the $i^{th}$ one-hot encoded true label and $\hat y^{(i)} \in [0,...
helperFunction's user avatar
1 vote
1 answer
4k views

Confused with binary cross-entropy vs categorical cross-entropy

I have a dataset with 10 input categorical features and one output categorical feature with class 0 and 1. X_train follows a 3D array so I have done label encoding beforehand on the dataset. I have ...
be_real's user avatar
  • 113
0 votes
0 answers
87 views

Is there an explanation for a classifier achieving high F1 scores, but having still high CrossEntropyLoss?

I am training a CNN classifier on a balanced dataset (around 35k examples for each label) with 13 classes. The model seems to achieve high F1 scores from the first batches; The F1 score for each class ...
Andrew's user avatar
  • 13
0 votes
1 answer
270 views

Intuition behind Energy function in Restricted Boltzmann Machines

What does the Energy function in Restricted Boltzmann Machines represent intuitively? I explain what I mean by the following example. If we look at cross-entropy $H(p,q_{\theta})=-\sum_{x}p(x)\log(q_{\...
Qwerty's user avatar
  • 3
2 votes
0 answers
617 views

Disadvantages of cross entropy loss comparing to SVM loss [closed]

What are some disadvantages and limitations of the cross entropy loss, especially compared with SVM loss/hinge loss? I am just looking for a general idea of when would one use SVM loss over cross ...
Kaihua Hou's user avatar
0 votes
1 answer
306 views

Is it possible to use softmax for anomaly detection?

Suppose we have a model for classification. Normally the head of the model is a softmax over all the label/categories. Is it a good idea (that is being used in ...
IsaacLevon's user avatar

1
2 3 4 5 6