# Questions tagged [cross-entropy]

A measure of the difference between two probability distributions for a given random variable or set of events.

264 questions
Filter by
Sorted by
Tagged with
30 views

1 vote
189 views

### Derivative error with respect to bias in binary cross entropy

I will do research using NN with 1 hidden layer. To calculate loss using binary cross entropy and for the activation function using sigmoid. I found the derivative formula from Sadowski, 2016 (link: ...
42 views

### Difference between two normal distributions [duplicate]

For the discrete case (in programming) the answer is clear to me. I would put my samples into the Kulback-Leibler-Divergence or maybe cross entropy. In my case I have pairs of normal distributions or ...
219 views

### Does logistic regression try to predict the true conditional P(Y|X)?

Consider a binary classification dataset (X, Y), generated according to some unknown distribution $P(X, Y)$. I have a question about models which output probabilities by minimizing the cross-entropy ...
254 views

### Understanding StatQuest video: why cross entropy is used over Sum Squared Error

I was watching cross entropy video from StatQuest. While explaining why to use cross entropy over SSE in multi output scenario with softmax output activation, Josh gives this graph of both losses: He ...
31 views

### How to compare two very large vectors that each represent a probability mass function?

As far as I know, given two vectors that each represent a probability mass function, their difference can be measured using Euclidean distance, Kullback–Leibler divergence, cross entropy and so on. ...
22 views

### Probability and Entropy of a Multichannel Image

I'm currently trying to implement the method covered in the paper "Hyperspectral Band Selection via Optimal Combination Strategy" I found that point 1) of this questions was close to what I ...
12 views

### How do entropy-based estimators relate to more conventional ML, least square, and GMM estimators

Over the years i have done a lot of analysis, mostly of parameters of linear approximations to the data or a forecast, and I have used linear and nonlinear least squares, maximum likelihood, and GMM ...
35 views

1 vote
3k views

### Confused with binary cross-entropy vs categorical cross-entropy

I have a dataset with 10 input categorical features and one output categorical feature with class 0 and 1. X_train follows a 3D array so I have done label encoding beforehand on the dataset. I have ...
73 views

### Is there an explanation for a classifier achieving high F1 scores, but having still high CrossEntropyLoss?

I am training a CNN classifier on a balanced dataset (around 35k examples for each label) with 13 classes. The model seems to achieve high F1 scores from the first batches; The F1 score for each class ...
162 views

### Intuition behind Energy function in Restricted Boltzmann Machines

What does the Energy function in Restricted Boltzmann Machines represent intuitively? I explain what I mean by the following example. If we look at cross-entropy \$H(p,q_{\theta})=-\sum_{x}p(x)\log(q_{\...