Questions tagged [cross-entropy]

A measure of the difference between two probability distributions for a given random variable or set of events.

Filter by
Sorted by
Tagged with
0 votes
0 answers
30 views

Concentration Inequality for cross-entropy

I am currently trying to estimate the cross-entropy between two distributions with densities $p$ and $q$. $$ \ell = -\mathbb{E}_{x\sim p(x) }[\log q(x)] $$ I am using a Monte-Carlo estimate: $$ \hat{\...
Nick Bishop's user avatar
0 votes
0 answers
16 views

Weight Sum Constraint in Loss Functions for Learning with Noisy Labels

I'm currently working on a machine learning task involving a dataset with noisy labels, and I'm considering using a reweighted loss function to address the issue of label noise. I understand that ...
External_Happy_77's user avatar
0 votes
0 answers
45 views

Minimizing cross entropy over a restricted domain?

Suppose $f(x;q)$ is the true distribution. The support of the random variable $X$ is $\Omega$. Suppose, I am interested in a particular subset of $\Xi \subset \Omega$. I would like to minimize the ...
entropy's user avatar
  • 19
1 vote
1 answer
45 views

What exactly is the problem with overconfident predictions?

Say I have a neural network that classifies images by training to minimise cross-entropy loss with one-hot encoded training labels. It is often seen that such neural networks are 'overconfident', with ...
Danny Duberstein's user avatar
0 votes
1 answer
55 views

Is CE(X, Y) equivalent to H(X) + H(Y)?

From my understanding mutual information can be defined in the following ways: [1]: $I(X;Y)=H(X)+H(Y)-H(X,Y)$ where $H(X), H(Y)$ are marginal entropies and $H(X,Y)$ is the joint entropy. [2]: $I(X;Y)=...
Rui's user avatar
  • 3
1 vote
0 answers
21 views

Decision boundary for Cross entropy loss and Least square loss

We can see the source in this paper. My question is that why cross entropy loss has a boundary line in slope but least square loss has horizontal boundary. Can somebody explain?
batuman's user avatar
  • 483
0 votes
1 answer
47 views

Derivation of cross entropy loss in machine learning

Given a dataset $\mathcal{D} = \{ (x_1, y_1),\cdots, (x_n, y_n)\}$, let's say we want to approximate the conditional probability $p(y|x)$, and we parameterized it as $p_{\theta}(y|x)$. So,for a ...
UESTCfresh's user avatar
1 vote
1 answer
87 views

can we use binary cross entropy with labels -1 and 1?

Binary cross entropy is written as follows: \begin{equation} \mathcal{L} = -y\log\left(\hat{y}\right)-(1-y)\log\left(1-\hat{y}\right) \end{equation} In every reference that I read, when using binary ...
andryan86's user avatar
  • 117
2 votes
1 answer
243 views

Meaning of non-{0,1} labels in binary cross entropy?

Binary cross entropy is normally used in situations where the "true" result or label is one of two values (hence "binary"), typically encoded as 0 and 1. However, the documentation ...
R.M.'s user avatar
  • 980
1 vote
1 answer
81 views

Calculating KL divergence with entropy and cross entropy for VAEs

When looking at implementations of VAE's online, specifically the KL divergence loss, the formula used is: $$ KL\hspace{1mm} Loss = -\frac{1}{2}(1+\log{\sigma^2}-\mu^2-\sigma^2) $$ or some variation ...
pyrrosk's user avatar
  • 23
0 votes
0 answers
14 views

Conditioning decreases cross-entropy

I know conditioning decreases the normal entropy: $ H(Y)>H(Y|X)$. But does it hold for the cross entropy? Do we have $H_c(Y;Y')>H_c(Y|X;Y'|X)$?
user382739's user avatar
1 vote
1 answer
60 views

Very balanced dataset and a multiclass classification problem, no context behind the inputs. Which evaluation metric to use?

I have constructed a simple neural network model, for a classification problem, with 10 target classes where an input (with some number of features) is to be classified to only one of the 10 classes. ...
creamedcheese83's user avatar
1 vote
0 answers
18 views

Log base in Cross Entropy Loss [duplicate]

What is the base for the logarithm used in the cross entropy loss (while doing multiclass classification's backpropagation)? Is it e, 2, or 10?
Sachin's user avatar
  • 111
2 votes
1 answer
257 views

Why is the cross entropy of the same probability distribution not 0?

From what I've been reading, if there is no underlying difference between the 2 probabilities distributions we would have perfect entropy. I'm putting an example below. Can anybody explain why the ...
julian lagier's user avatar
1 vote
0 answers
80 views

How does the cross entropy loss function interact with the final layer of a neural network?

I am having trouble understanding how the result of categorical cross entropy loss can be used to calculate the gradient for all of the weights. The output of cross entropy function is the sum of all ...
Nick's user avatar
  • 33
0 votes
0 answers
78 views

Perplexity and cross-entropy relationship

According to wikipedia Perplexity- A perplexity of discrete distribution p equals to $2^{H(p)}$ where H(p) is the entropy of p. A perplexity of a probability model M, with unknown probabiltiy p - ${\...
BOB123's user avatar
  • 31
2 votes
0 answers
119 views

Understanding intuitive difference between KL divergence and Cross entropy

I know there are related questions already asked, for example this one. I also know the following: KL divergence $D_{KL}(P\Vert Q)$ is given as: $$\begin{align} D_{KL}(P\Vert Q) & = -\sum_xP(x)\...
Mahesha999's user avatar
1 vote
1 answer
189 views

Derivative error with respect to bias in binary cross entropy

I will do research using NN with 1 hidden layer. To calculate loss using binary cross entropy and for the activation function using sigmoid. I found the derivative formula from Sadowski, 2016 (link: ...
Andryan's user avatar
  • 47
0 votes
0 answers
42 views

Difference between two normal distributions [duplicate]

For the discrete case (in programming) the answer is clear to me. I would put my samples into the Kulback-Leibler-Divergence or maybe cross entropy. In my case I have pairs of normal distributions or ...
Daraan's user avatar
  • 103
5 votes
1 answer
219 views

Does logistic regression try to predict the true conditional P(Y|X)?

Consider a binary classification dataset (X, Y), generated according to some unknown distribution $P(X, Y)$. I have a question about models which output probabilities by minimizing the cross-entropy ...
usual me's user avatar
  • 1,157
3 votes
2 answers
254 views

Understanding StatQuest video: why cross entropy is used over Sum Squared Error

I was watching cross entropy video from StatQuest. While explaining why to use cross entropy over SSE in multi output scenario with softmax output activation, Josh gives this graph of both losses: He ...
Mahesha999's user avatar
0 votes
0 answers
31 views

How to compare two very large vectors that each represent a probability mass function?

As far as I know, given two vectors that each represent a probability mass function, their difference can be measured using Euclidean distance, Kullback–Leibler divergence, cross entropy and so on. ...
m.taheri's user avatar
  • 131
0 votes
0 answers
22 views

Probability and Entropy of a Multichannel Image

I'm currently trying to implement the method covered in the paper "Hyperspectral Band Selection via Optimal Combination Strategy" I found that point 1) of this questions was close to what I ...
error404's user avatar
0 votes
0 answers
12 views

How do entropy-based estimators relate to more conventional ML, least square, and GMM estimators

Over the years i have done a lot of analysis, mostly of parameters of linear approximations to the data or a forecast, and I have used linear and nonlinear least squares, maximum likelihood, and GMM ...
andrewH's user avatar
  • 3,037
0 votes
0 answers
35 views

How to interpret the focal loss from simulation?

The focal loss first appeared in Focal Loss for Dense Object Detection by Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollár In the paper , the focal loss is actually given as: $−\...
ForestGump's user avatar
1 vote
0 answers
16 views

Relate cross-entropy formal definition to the cross-entropy loss [duplicate]

Cross entropy for a random variable $x \sim p$ and a distribution $q$ is defined as: $$H(p,q) = -\sum_{x\in\mathcal{X}} p(x)\log q(x) = \mathbb{E}(\log q(x))$$ $\mathcal{X}$ is all possible values ...
rando's user avatar
  • 236
0 votes
0 answers
75 views

Add Bias to classification after training

I have a dataset with classes [a, b] where during training I have made sure that the dataset is equally balanced. I have trained the network using cross-entropy loss with equal importance. I am able ...
JakobVinkas's user avatar
1 vote
0 answers
34 views

Genetic Algorithm as engine for Variational Inference?

I'm curious if anyone has used, heard of, or otherwise considered using Genetic Algorithms as an engine for Variational Inference (VI)? My understanding of VI is that it's an optimization algorithm, ...
jbuddy_13's user avatar
  • 2,377
2 votes
2 answers
119 views

Surprisal in rankings

I'm looking for some metric of surprisal when comparing ranked lists - things along the lines of (eg) the rankings in a marathon race, or the times in the race. Intuitively, in a race with 100 people, ...
Alex I's user avatar
  • 1,103
3 votes
0 answers
237 views

Cross-entropy vs dot product

Performance of classification algorithms is quantified by comparing the predicted probability distribution of the labels $q$ to the true probability $p$, which is commonly a vector of zeros for all ...
Aleksejs Fomins's user avatar
1 vote
1 answer
76 views

How do machine learning algorithms handle classification labels?

I am working on a domain adaptation problem, where the default is a classification problem. I have worked exclusively with regression problems until now, so I am kind of thrown for a loop when it ...
Scott's user avatar
  • 13
1 vote
0 answers
126 views

How to explain the high accuracy and F1 score on the test set with a huge binary crossentropy loss?

I'll provide a little of introduction based on my example. I have a small collection of RGB (but 'gray-looking') brain MRI photos, divided into 2 classes: healthy and tumor. My data split looks like ...
Karolina Świergała's user avatar
2 votes
1 answer
97 views

Why is it called the cross-entropy of q relative to p, not p relative to q?

I'm looking into the definition of cross entropy from wikipedia. https://en.wikipedia.org/wiki/Cross_entropy Cross entropy is not symmetric, so I think for sure it shouldn't be called cross entropy ...
user900476's user avatar
1 vote
1 answer
338 views

Calculating the variance of softmax

I'm working through Dive Into Deep Learning right now and am struggling with the following question: We can explore the connection between exponential families and the softmax in some more depth. ...
jimac82's user avatar
  • 21
2 votes
1 answer
197 views

Cross entropy of a random variable or a probability distribution function? [duplicate]

I'm looking into the wikipedia page of cross entropy. https://en.wikipedia.org/wiki/Cross_entropy $$H(p,q)=-\sum_{x\in \mathcal{X}} p(x)\log q(x)$$ It can be written as $$H(p,q) = H(p) + D_{KL} (p||q)$...
user900476's user avatar
0 votes
0 answers
18 views

What performance we get with same data combines with different datasets?

Suppose that we have dataset of special kind of cat. We are going to train a model on combination of the cat a the car! Suppose that in this model we will get a performance ( precision,recall or...) X ...
Mahdi Amrollahi's user avatar
0 votes
0 answers
20 views

GoogleNet-LSTM, cross entropy loss does not decrease [duplicate]

...
makala's user avatar
  • 11
2 votes
0 answers
153 views

Is there an empirical rule for selecting the value of label smoothing?

I am wondering if there is any emperical rule for selecting the value of label smoothing when training a neural network. Let's define smoothed prediction targets in relation to a value $\epsilon$ to ...
thiaamak's user avatar
1 vote
0 answers
279 views

Final Layer and Inference with CE vs BCE

I have read a similar question here: 1 neuron BCE loss VS 2 neurons CE loss that suggests there is no difference between softmax cross entropy loss and binary cross entropy loss, when choosing between ...
Anonymous's user avatar
  • 181
0 votes
0 answers
348 views

Why most works on Cityscapes don't use weighted cross-entropy?

Weight Cross-Entroy (WCE) helps to handle an imbalanced dataset, and Cityscapes is quite imbalanced as seen below: If we check the best benchmarks on this dataset, most of the works use bare CE as a ...
Rafael Toledo's user avatar
2 votes
0 answers
532 views

Impact of L1 and L2 regularisation with cross-entropy loss

When we are dealing with Mean Square Error (MSE) loss function in optimization problems, we often add $L_1$ or $L_2$ penalty terms (or a combination of both) to the MSE loss function while training. ...
Aravind G.'s user avatar
0 votes
0 answers
220 views

How can I get the Binary Cross Entropy from the Cross Entropy function for GANs

I got the definition of log-likelihood by Goodfellow's Deep Learning book: \begin{equation} \label{eq:loglikelihood} \theta_{ML} = {argmax}\sum_{i=1}^{m} \log p_{model}(x_i; \theta). \end{...
Lucas Lima de Sousa's user avatar
1 vote
1 answer
186 views

XGBoost Objective Derivation Problem

This is the loss function of XGBoost. This is the Second-order approximation of the loss function. Note: \begin{equation} L^{(t)} \text{: cross entropy loss function.} \end{equation} \begin{equation}...
ChrisChu's user avatar
0 votes
0 answers
64 views

Likelihood and cross-entropy: continuous case

I think it's pretty clear to me that average log-likelihood is equivalent to negative cross-entropy for discrete distributions, as shown here: $$\frac{1}{N}\log\mathcal{L}(\theta) = \frac{1}{N}\log \...
Alex Zakharov's user avatar
14 votes
2 answers
1k views

Disadvantages of using a regression loss function in multi-class classification

Given $k > 2$ classes, consider the following loss function $$ \sum_i||y^{(i)} - \hat y^{(i)}||^2 $$ Here $y^{(i)} \in \{0,1\}^k$ is the $i^{th}$ one-hot encoded true label and $\hat y^{(i)} \in [0,...
helperFunction's user avatar
1 vote
1 answer
3k views

Confused with binary cross-entropy vs categorical cross-entropy

I have a dataset with 10 input categorical features and one output categorical feature with class 0 and 1. X_train follows a 3D array so I have done label encoding beforehand on the dataset. I have ...
be_real's user avatar
  • 13
0 votes
0 answers
73 views

Is there an explanation for a classifier achieving high F1 scores, but having still high CrossEntropyLoss?

I am training a CNN classifier on a balanced dataset (around 35k examples for each label) with 13 classes. The model seems to achieve high F1 scores from the first batches; The F1 score for each class ...
Andrew's user avatar
  • 13
0 votes
1 answer
162 views

Intuition behind Energy function in Restricted Boltzmann Machines

What does the Energy function in Restricted Boltzmann Machines represent intuitively? I explain what I mean by the following example. If we look at cross-entropy $H(p,q_{\theta})=-\sum_{x}p(x)\log(q_{\...
Qwerty's user avatar
  • 3
2 votes
0 answers
580 views

Disadvantages of cross entropy loss comparing to SVM loss [closed]

What are some disadvantages and limitations of the cross entropy loss, especially compared with SVM loss/hinge loss? I am just looking for a general idea of when would one use SVM loss over cross ...
Kaihua Hou's user avatar
0 votes
1 answer
244 views

Is it possible to use softmax for anomaly detection?

Suppose we have a model for classification. Normally the head of the model is a softmax over all the label/categories. Is it a good idea (that is being used in ...
IsaacLevon's user avatar

1
2 3 4 5 6