# Questions tagged [cross-entropy]

A measure of the difference between two probability distributions for a given random variable or set of events.

264 questions
Filter by
Sorted by
Tagged with
92 views

43 views

• 235
372 views

### Derivative error with respect to bias in binary cross entropy

I will do research using NN with 1 hidden layer. To calculate loss using binary cross entropy and for the activation function using sigmoid. I found the derivative formula from Sadowski, 2016 (link: ...
• 47
255 views

### Does logistic regression try to predict the true conditional P(Y|X)?

Consider a binary classification dataset (X, Y), generated according to some unknown distribution $P(X, Y)$. I have a question about models which output probabilities by minimizing the cross-entropy ...
• 1,247
425 views

### Understanding StatQuest video: why cross entropy is used over Sum Squared Error

I was watching cross entropy video from StatQuest. While explaining why to use cross entropy over SSE in multi output scenario with softmax output activation, Josh gives this graph of both losses: He ...
• 235
1 vote
17 views

### Relate cross-entropy formal definition to the cross-entropy loss [duplicate]

Cross entropy for a random variable $x \sim p$ and a distribution $q$ is defined as: $$H(p,q) = -\sum_{x\in\mathcal{X}} p(x)\log q(x) = \mathbb{E}(\log q(x))$$ $\mathcal{X}$ is all possible values ...
• 308
95 views

### Add Bias to classification after training

I have a dataset with classes [a, b] where during training I have made sure that the dataset is equally balanced. I have trained the network using cross-entropy loss with equal importance. I am able ...
1 vote
45 views

### Genetic Algorithm as engine for Variational Inference?

I'm curious if anyone has used, heard of, or otherwise considered using Genetic Algorithms as an engine for Variational Inference (VI)? My understanding of VI is that it's an optimization algorithm, ...
• 3,372
149 views

### Surprisal in rankings

I'm looking for some metric of surprisal when comparing ranked lists - things along the lines of (eg) the rankings in a marathon race, or the times in the race. Intuitively, in a race with 100 people, ...
• 1,153
437 views

### Cross-entropy vs dot product

Performance of classification algorithms is quantified by comparing the predicted probability distribution of the labels $q$ to the true probability $p$, which is commonly a vector of zeros for all ...
• 2,121
1 vote
133 views

### How do machine learning algorithms handle classification labels?

I am working on a domain adaptation problem, where the default is a classification problem. I have worked exclusively with regression problems until now, so I am kind of thrown for a loop when it ...
• 13
1 vote
209 views

### How to explain the high accuracy and F1 score on the test set with a huge binary crossentropy loss?

I'll provide a little of introduction based on my example. I have a small collection of RGB (but 'gray-looking') brain MRI photos, divided into 2 classes: healthy and tumor. My data split looks like ...
151 views

### Why is it called the cross-entropy of q relative to p, not p relative to q?

I'm looking into the definition of cross entropy from wikipedia. https://en.wikipedia.org/wiki/Cross_entropy Cross entropy is not symmetric, so I think for sure it shouldn't be called cross entropy ...
• 433
1 vote
600 views

### Calculating the variance of softmax

I'm working through Dive Into Deep Learning right now and am struggling with the following question: We can explore the connection between exponential families and the softmax in some more depth. ...
• 21
298 views

2k views

617 views

### Disadvantages of cross entropy loss comparing to SVM loss [closed]

What are some disadvantages and limitations of the cross entropy loss, especially compared with SVM loss/hinge loss? I am just looking for a general idea of when would one use SVM loss over cross ...
• 121