Questions tagged [cross-entropy]
A measure of the difference between two probability distributions for a given random variable or set of events.
264
questions
0
votes
0
answers
30
views
Concentration Inequality for cross-entropy
I am currently trying to estimate the cross-entropy between two distributions with densities $p$ and $q$.
$$
\ell = -\mathbb{E}_{x\sim p(x) }[\log q(x)]
$$
I am using a Monte-Carlo estimate:
$$
\hat{\...
- 101
0
votes
0
answers
16
views
Weight Sum Constraint in Loss Functions for Learning with Noisy Labels
I'm currently working on a machine learning task involving a dataset with noisy labels, and I'm considering using a reweighted loss function to address the issue of label noise. I understand that ...
0
votes
0
answers
45
views
Minimizing cross entropy over a restricted domain?
Suppose $f(x;q)$ is the true distribution. The support of the random variable $X$ is $\Omega$. Suppose, I am interested in a particular subset of $\Xi \subset \Omega$. I would like to minimize the ...
- 19
1
vote
1
answer
45
views
What exactly is the problem with overconfident predictions?
Say I have a neural network that classifies images by training to minimise cross-entropy loss with one-hot encoded training labels. It is often seen that such neural networks are 'overconfident', with ...
- 111
0
votes
1
answer
55
views
Is CE(X, Y) equivalent to H(X) + H(Y)?
From my understanding mutual information can be defined in the following ways:
[1]:
$I(X;Y)=H(X)+H(Y)-H(X,Y)$ where $H(X), H(Y)$ are marginal entropies and $H(X,Y)$ is the joint entropy.
[2]:
$I(X;Y)=...
- 3
1
vote
0
answers
21
views
Decision boundary for Cross entropy loss and Least square loss
We can see the source in this paper.
My question is that why cross entropy loss has a boundary line in slope but least square loss has horizontal boundary.
Can somebody explain?
- 483
0
votes
1
answer
47
views
Derivation of cross entropy loss in machine learning
Given a dataset $\mathcal{D} = \{ (x_1, y_1),\cdots, (x_n, y_n)\}$, let's say we want to approximate the conditional probability $p(y|x)$, and we parameterized it as $p_{\theta}(y|x)$. So,for a ...
- 101
1
vote
1
answer
87
views
can we use binary cross entropy with labels -1 and 1?
Binary cross entropy is written as follows:
\begin{equation}
\mathcal{L} = -y\log\left(\hat{y}\right)-(1-y)\log\left(1-\hat{y}\right)
\end{equation}
In every reference that I read, when using binary ...
- 117
2
votes
1
answer
243
views
Meaning of non-{0,1} labels in binary cross entropy?
Binary cross entropy is normally used in situations where the "true" result or label is one of two values (hence "binary"), typically encoded as 0 and 1.
However, the documentation ...
- 980
1
vote
1
answer
81
views
Calculating KL divergence with entropy and cross entropy for VAEs
When looking at implementations of VAE's online, specifically the KL divergence loss, the formula used is:
$$ KL\hspace{1mm} Loss = -\frac{1}{2}(1+\log{\sigma^2}-\mu^2-\sigma^2) $$
or some variation ...
- 23
0
votes
0
answers
14
views
Conditioning decreases cross-entropy
I know conditioning decreases the normal entropy: $ H(Y)>H(Y|X)$. But does it hold for the cross entropy? Do we have $H_c(Y;Y')>H_c(Y|X;Y'|X)$?
1
vote
1
answer
60
views
Very balanced dataset and a multiclass classification problem, no context behind the inputs. Which evaluation metric to use?
I have constructed a simple neural network model, for a classification problem, with 10 target classes where an input (with some number of features) is to be classified to only one of the 10 classes.
...
1
vote
0
answers
18
views
Log base in Cross Entropy Loss [duplicate]
What is the base for the logarithm used in the cross entropy loss (while doing multiclass classification's backpropagation)? Is it e, 2, or 10?
- 111
2
votes
1
answer
257
views
Why is the cross entropy of the same probability distribution not 0?
From what I've been reading, if there is no underlying difference between the 2 probabilities distributions we would have perfect entropy.
I'm putting an example below. Can anybody explain why the ...
1
vote
0
answers
80
views
How does the cross entropy loss function interact with the final layer of a neural network?
I am having trouble understanding how the result of categorical cross entropy loss can be used to calculate the gradient for all of the weights.
The output of cross entropy function is the sum of all ...
- 33
0
votes
0
answers
78
views
Perplexity and cross-entropy relationship
According to wikipedia Perplexity-
A perplexity of discrete distribution p equals to $2^{H(p)}$ where H(p) is the entropy of p.
A perplexity of a probability model M, with unknown probabiltiy p - ${\...
- 31
2
votes
0
answers
119
views
Understanding intuitive difference between KL divergence and Cross entropy
I know there are related questions already asked, for example this one.
I also know the following:
KL divergence $D_{KL}(P\Vert Q)$ is given as:
$$\begin{align}
D_{KL}(P\Vert Q) & = -\sum_xP(x)\...
- 215
1
vote
1
answer
189
views
Derivative error with respect to bias in binary cross entropy
I will do research using NN with 1 hidden layer. To calculate loss using binary cross entropy and for the activation function using sigmoid. I found the derivative formula from Sadowski, 2016 (link: ...
- 47
0
votes
0
answers
42
views
Difference between two normal distributions [duplicate]
For the discrete case (in programming) the answer is clear to me.
I would put my samples into the Kulback-Leibler-Divergence or maybe cross entropy.
In my case I have pairs of normal distributions or ...
- 103
5
votes
1
answer
219
views
Does logistic regression try to predict the true conditional P(Y|X)?
Consider a binary classification dataset (X, Y), generated according to some unknown distribution $P(X, Y)$. I have a question about models which output probabilities by minimizing the cross-entropy ...
- 1,157
3
votes
2
answers
254
views
Understanding StatQuest video: why cross entropy is used over Sum Squared Error
I was watching cross entropy video from StatQuest. While explaining why to use cross entropy over SSE in multi output scenario with softmax output activation, Josh gives this graph of both losses:
He ...
- 215
0
votes
0
answers
31
views
How to compare two very large vectors that each represent a probability mass function?
As far as I know, given two vectors that each represent a probability mass function, their difference can be measured using Euclidean distance, Kullback–Leibler divergence, cross entropy and so on. ...
- 131
0
votes
0
answers
22
views
Probability and Entropy of a Multichannel Image
I'm currently trying to implement the method covered in the paper "Hyperspectral Band Selection via Optimal Combination Strategy"
I found that point 1) of this questions was close to what I ...
- 1
0
votes
0
answers
12
views
How do entropy-based estimators relate to more conventional ML, least square, and GMM estimators
Over the years i have done a lot of analysis, mostly of parameters of linear approximations to the data or a forecast, and I have used linear and nonlinear least squares, maximum likelihood, and GMM ...
- 3,037
0
votes
0
answers
35
views
How to interpret the focal loss from simulation?
The focal loss first appeared in Focal Loss for Dense Object Detection by Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollár
In the paper , the focal loss is actually given as:
$−\...
- 327
1
vote
0
answers
16
views
Relate cross-entropy formal definition to the cross-entropy loss [duplicate]
Cross entropy for a random variable $x \sim p$ and a distribution $q$ is defined as:
$$H(p,q) = -\sum_{x\in\mathcal{X}} p(x)\log q(x) = \mathbb{E}(\log q(x))$$
$\mathcal{X}$ is all possible values ...
- 236
0
votes
0
answers
75
views
Add Bias to classification after training
I have a dataset with classes [a, b] where during training I have made sure that the dataset is equally balanced. I have trained the network using cross-entropy loss with equal importance.
I am able ...
- 61
1
vote
0
answers
34
views
Genetic Algorithm as engine for Variational Inference?
I'm curious if anyone has used, heard of, or otherwise considered using Genetic Algorithms as an engine for Variational Inference (VI)?
My understanding of VI is that it's an optimization algorithm, ...
- 2,377
2
votes
2
answers
119
views
Surprisal in rankings
I'm looking for some metric of surprisal when comparing ranked lists - things along the lines of (eg) the rankings in a marathon race, or the times in the race.
Intuitively, in a race with 100 people, ...
- 1,103
3
votes
0
answers
237
views
Cross-entropy vs dot product
Performance of classification algorithms is quantified by comparing the predicted probability distribution of the labels $q$ to the true probability $p$, which is commonly a vector of zeros for all ...
- 1,771
1
vote
1
answer
76
views
How do machine learning algorithms handle classification labels?
I am working on a domain adaptation problem, where the default is a classification problem. I have worked exclusively with regression problems until now, so I am kind of thrown for a loop when it ...
- 13
1
vote
0
answers
126
views
How to explain the high accuracy and F1 score on the test set with a huge binary crossentropy loss?
I'll provide a little of introduction based on my example. I have a small collection of RGB (but 'gray-looking') brain MRI photos, divided into 2 classes: healthy and tumor. My data split looks like ...
2
votes
1
answer
97
views
Why is it called the cross-entropy of q relative to p, not p relative to q?
I'm looking into the definition of cross entropy from wikipedia. https://en.wikipedia.org/wiki/Cross_entropy
Cross entropy is not symmetric, so I think for sure it shouldn't be called cross entropy ...
- 423
1
vote
1
answer
338
views
Calculating the variance of softmax
I'm working through Dive Into Deep Learning right now and am struggling with the following question:
We can explore the connection between exponential families and the
softmax in some more depth.
...
- 21
2
votes
1
answer
197
views
Cross entropy of a random variable or a probability distribution function? [duplicate]
I'm looking into the wikipedia page of cross entropy. https://en.wikipedia.org/wiki/Cross_entropy
$$H(p,q)=-\sum_{x\in \mathcal{X}} p(x)\log q(x)$$
It can be written as $$H(p,q) = H(p) + D_{KL} (p||q)$...
- 423
0
votes
0
answers
18
views
What performance we get with same data combines with different datasets?
Suppose that we have dataset of special kind of cat. We are going to train a model on combination of the cat a the car! Suppose that in this model we will get a performance ( precision,recall or...) X ...
- 159
0
votes
0
answers
20
views
2
votes
0
answers
153
views
Is there an empirical rule for selecting the value of label smoothing?
I am wondering if there is any emperical rule for selecting the value of label smoothing when training a neural network. Let's define smoothed prediction targets in relation to a value $\epsilon$ to ...
- 21
1
vote
0
answers
279
views
Final Layer and Inference with CE vs BCE
I have read a similar question here: 1 neuron BCE loss VS 2 neurons CE loss that suggests there is no difference between softmax cross entropy loss and binary cross entropy loss, when choosing between ...
- 181
0
votes
0
answers
348
views
Why most works on Cityscapes don't use weighted cross-entropy?
Weight Cross-Entroy (WCE) helps to handle an imbalanced dataset, and Cityscapes is quite imbalanced as seen below:
If we check the best benchmarks on this dataset, most of the works use bare CE as a ...
- 158
2
votes
0
answers
532
views
Impact of L1 and L2 regularisation with cross-entropy loss
When we are dealing with Mean Square Error (MSE) loss function in optimization problems, we often add $L_1$ or $L_2$ penalty terms (or a combination of both) to the MSE loss function while training. ...
- 121
0
votes
0
answers
220
views
How can I get the Binary Cross Entropy from the Cross Entropy function for GANs
I got the definition of log-likelihood by Goodfellow's Deep Learning book:
\begin{equation}
\label{eq:loglikelihood}
\theta_{ML} = {argmax}\sum_{i=1}^{m} \log p_{model}(x_i; \theta).
\end{...
1
vote
1
answer
186
views
XGBoost Objective Derivation Problem
This is the loss function of XGBoost.
This is the Second-order approximation of the loss function.
Note:
\begin{equation} L^{(t)} \text{: cross entropy loss function.} \end{equation}
\begin{equation}...
- 13
0
votes
0
answers
64
views
Likelihood and cross-entropy: continuous case
I think it's pretty clear to me that average log-likelihood is equivalent to negative cross-entropy for discrete distributions, as shown here:
$$\frac{1}{N}\log\mathcal{L}(\theta) = \frac{1}{N}\log \...
14
votes
2
answers
1k
views
Disadvantages of using a regression loss function in multi-class classification
Given $k > 2$ classes, consider the following loss function
$$
\sum_i||y^{(i)} - \hat y^{(i)}||^2
$$
Here $y^{(i)} \in \{0,1\}^k$ is the $i^{th}$ one-hot encoded true label and $\hat y^{(i)} \in [0,...
- 346
1
vote
1
answer
3k
views
Confused with binary cross-entropy vs categorical cross-entropy
I have a dataset with 10 input categorical features and one output categorical feature with class 0 and 1. X_train follows a 3D array so I have done label encoding beforehand on the dataset.
I have ...
- 13
0
votes
0
answers
73
views
Is there an explanation for a classifier achieving high F1 scores, but having still high CrossEntropyLoss?
I am training a CNN classifier on a balanced dataset (around 35k examples for each label) with 13 classes. The model seems to achieve high F1 scores from the first batches; The F1 score for each class ...
- 13
0
votes
1
answer
162
views
Intuition behind Energy function in Restricted Boltzmann Machines
What does the Energy function in Restricted Boltzmann Machines represent intuitively?
I explain what I mean by the following example. If we look at cross-entropy $H(p,q_{\theta})=-\sum_{x}p(x)\log(q_{\...
- 3
2
votes
0
answers
580
views
Disadvantages of cross entropy loss comparing to SVM loss [closed]
What are some disadvantages and limitations of the cross entropy loss, especially compared with SVM loss/hinge loss? I am just looking for a general idea of when would one use SVM loss over cross ...
- 121
0
votes
1
answer
244
views
Is it possible to use softmax for anomaly detection?
Suppose we have a model for classification.
Normally the head of the model is a softmax over all the label/categories. Is it a good idea (that is being used in ...
- 165