Cross Validated

Questions tagged [cross-entropy]

Ask Question

The tag has no usage guidance.

144 questions
Newest
Active
Bountied
Unanswered
Filter by
Sorted by
Tagged with
1
vote
0answers
17 views

Softmax + CE vs Sigmoid + BCE for batched training with negative sampling, for training similarity properties

This is a follow up to this question Machine Learning: Should I use a categorical cross entropy or binary cross entropy loss for binary predictions? I am training cos similarity properties for ...
1
vote
0answers
19 views

Why is softmax considered counter-intuitive for multi-label classification?

In the FB paper on Instagram multi-label classification (Exploring the Limits of Weakly Supervised Pretraining), the authors characterize as "counter-intuitive" their finding that softmax + ...
1
vote
0answers
23 views

Autoencoder predictions for extremely simple task does not make intuitive sense

I am training a simple autoencoder in Keras. The input is of length two, where each element can either be 0 or 1. This gives four distinct input possibilities: [0, 0], [0, 1], [1, 0], [1, 1]. Since ...
0
votes
0answers
44 views

Custom cross entropy loss function

I want to define custom cross entropy loss penalizing different class errors. Categorical cross entropy loss = $\sum_{i=1}^K y_i log(p_i)$ I want to give different weights to different prediction ...
0
votes
1answer
34 views

Not understanding backpropagation correctly

So I'm trying to build a simple NN where the layers are as follows: linear layer-> ReLu -> ...
0
votes
0answers
30 views

MLE and Cross Entropy for Conditional Probabilities

I'm trying to understand the relationship between maximum likelihood estimation for a function of the type $p(y^{(i)}|x^{(i)};\theta)$ and the related cross entropy minimization. For a single ...
0
votes
1answer
76 views

How does Cross-Entropy (log loss) work with backpropagation?

I am having some trouble understanding how Cross Entropy would work with backpropagation. For backpropagation we exploit the chain rule to find the partial derivative of the Error function in terms of ...
0
votes
1answer
218 views

Gradient descent with Binary Cross-Entropy for single layer perceptron

I'm implementing a Single Layer Perceptron for binary classification in python. I'm using binary Cross-Entropy loss function and gradient descent. The gradient descent is not converging, may be I'm ...
0
votes
0answers
25 views

Meaning of the derivative of cross entropy wrt $p(x)$

Lets define the cross entropy between 2 probability distributions $p(x)$ and $q(x)$ as $$H(p,q) = -\sum p(x) \log{q(x)} $$ What would be the meaning of derivative of $H(p,q)$ when taking the ...
1
vote
0answers
37 views

derivative of cross entropy yields log-odds, does that make sense?

I am looking for a proof how to derive the logistic regression from cross-entropy loss, i.e. derive the form of a sigmoid from cross entropy. my thoughts are these: $\ell = y_i \ln{p_i} + (1-y_i)\ln{...
1
vote
0answers
9 views

Cross-entropy loss when some categories are broader than others

Let's say I want to write a classifier for pictures of dogs. Most importantly, I want to know whether something is a picture of a dog or not. Secondarily, it'd be nice to know what breed the dog is. ...
0
votes
0answers
10 views

Are there some guidelines to follow while combining different types of losses to make a cost function?

I'm training an Autoencoder to reproduce the input, and the architecture is a simple fully connected neural network. The initial phase of the implementation was using float/integer dataset, and ...
0
votes
0answers
21 views

When was cross-entropy loss proposed/ since when have researchers started using it?

I started to be curious about when was the cross-entropy loss function(and its variations such as class-imbalanced cross-entropy) proposed. Was there any specific literature to refer to? Or does there ...
0
votes
0answers
29 views

How to interpret when using hinge loss performs significantly better than cross-entropy loss in a multi-class clasification problem?"

Given that hinge loss is based on the marginal loss in SVM, is there any reasonable assumption / interpretation one can make on the topology of the dataset, when using multi-class hinge loss ...
0
votes
0answers
18 views

How to compute weight change for hidden layers with cross-entropy loss? [duplicate]

I'm trying to train a neural net with 1 hidden layer (RELU) softmax output layer cross-entropy loss stochastic gradient descent My implementation seems to work fine when I don't use any hidden ...
1
vote
0answers
31 views

How does cross entropy loss get calculated in overconfident neural networks?

Given formula for categorical formula for cross entropy: or binary forumla: how is cross entropy calculated in overconfident neural netowrks (especially CNNs where this phenomenon is common), e.g. ...
0
votes
1answer
89 views

Softmax with Cross Entropy optimization vs Backpropagation

I am following a tutorial from Analytics Vidhya on creating a neural network to recognize handwritten digits (the classic example). The code from the tutorial states "First we need to define the ...
0
votes
2answers
62 views

Binary cross-entropy: plugging in probability 0

There is an answer on the Kaggle question board here by Dr. Fuzzy: ...
2
votes
0answers
43 views

Name/definition of $\int \log F(x) \cdot g(x)dx$?

We know that: $$-\int \log f(x) \cdot g(x)dx,$$ where $f$ and $g$ are density functions, is known as the cross entropy. Does $$-\int \log F(x) \cdot g(x)dx,$$ where $F$ is the cumulative ...
0
votes
1answer
289 views

Label smoothing formula

I recently came across this paper in section 3.2 it talks about label smoothing loss and how it's equivalent to s equivalent to adding the KL divergence between the uniform distribution u and the ...
0
votes
0answers
13 views

What should the form of error be on CrossEntropy or KL-divergence loss function across samples of distributions?

Suppose your model produces (discrete) probability distributions and you have some truth distributions you want to compare to. For each sample $i$, you can compute the loss as the KL divergence or ...
0
votes
0answers
25 views

Deriving gradient of cross entropy from the cross entropy

I was trying to derive the third equation from the cross entropy of a conditional bernoulli distribution. But I just can't seem to find out how gradient $\mathbf z^{(i)}$ came out. Can someone please ...
0
votes
0answers
36 views

Neural Networks output functions [duplicate]

Studying about neural networks as a newbie, I learned (I think) about the different types of output functions, but I have been confused when to use each. (softmax, cross-entropy, sigmoid, none) For ...
0
votes
1answer
21 views

Analytical expression of the minimizer of cross entropy loss when the predicted function is a constant fucntion?

Let $\{y_1...y_n\} \in \{0,1\}$, and let $c \in [0,1]$. Define the cross-entropy of loss of $c$ by: $$C(c): = \sum_{j=1}^{n}- y_j ln c - (1- y_j) ln (1-c) $$. Define $c*= arg min _{c} C(c)$ Is ...
2
votes
3answers
590 views

Why is binary cross entropy (or log loss) used in autoencoders for non-binary data

I am working on an autoencoder for non-binary data ranging in [0,1] and while I was exploring existing solutions I noticed that in many people (e.g., the keras ...
1
vote
2answers
270 views

Convexity of cross entropy

I am not sure if this is a better fit for this site or mathematics.stackexchange but I've seen similar questions on here before. I'd like to know if the following is true and if so, how I could go ...
0
votes
0answers
54 views

my CNN predict all 0 or all 1 in multi label classification problem

I am trying to build a CNN for classifying multiple objects in images. I'm on keras and I use the COCO dataset. my net takes in input a 256x256 image and outputs the vector of the predictions of each ...
1
vote
0answers
316 views

In multiclass classification with K classes does cross entropy loss need K outputs or K-1?

Hastie's "The Elements of Statistical Learning" textbook defines the probabilistic model of multiclass logistic regression with K classes as $\forall k \in \{1, \dots, K-1\} $ $$ \ln \frac{p(G=k \mid ...
1
vote
1answer
1k views

cross entropy loss max value

The cross entropy loss function for multiclass can be computed as: $$-\sum\limits_{i=1}^N y_i log \hat{y}_i$$ where $y_i$ is a class and $\hat{y}_i$ the estimated probability. The minimum value is $0$ ...
0
votes
0answers
31 views

how to consider some miss classifications “half correct” in categorical_crossentropy - for a trading system

I have a trading system where the model receives 9 time-series and predict : A - strong down B - week down C - neutral D - week up E - strong up (these classes ...
1
vote
1answer
52 views

Why is loss treated as 0 or infinity here?

I'm reading a beginner's book about deep learning and in the introduction, the following cautionary tale is written: Now, assume that you built a classifier and trained it to predict if a mushroom ...
0
votes
0answers
289 views

loss='categorical_crossentropy' VS loss=K.categorical_crossentropy

Why I have very different loss values in training using these two lines code to define the loss function? ...
1
vote
2answers
1k views

Softmax Loss vs Binary Loss for classification?

I was trying to understand the final section of the paper "Revisiting Baselines for Visual Question Answering". Authors state that their model performs better with a binary loss in comparison to a ...
0
votes
0answers
162 views

Cross entropy vs KL divergence and SGD

I was told recently that stochastic gradient descent can be used to minimize a cross entropy objective function but not a KL-divergence one, even if the two minimization problems are equivalent, ie $...
0
votes
0answers
81 views

Cross - entropy for two variables with different prob. distributions

Let us say that we have given two random variables with different prob. distributions: A = [0.1, 0, 0.5, ...] B = [0.3, 0.1, 0.03, ...] What should I do when I want to compute the reformulated cross-...
0
votes
1answer
65 views

Approach to prevent bias/racism in neural network fitting?

I have a dataset comprised of different ethnic groups and I want to build a classification model on this data. When I do this I find that the performance of the algorithm is better on some groups than ...
0
votes
1answer
52 views

Cross entropy loss: inconsistency in formula

I have a couple of problems trying to understand the exact formula for cross entropy loss. Depending on the source I see it written different ways. Is the log() function $\log_2()$? Is the argument ...
1
vote
0answers
57 views

Binary cross etropy loss with non binary ground truth data [closed]

Is it possible to use binary cross etropy loss with non binary ground truth data, i.e. not [0,1] values, but [0,0.1,0.5,1.0] ...
2
votes
1answer
75 views

Relative Entropy decomposition

Can the relative entropy (Kullback Leibler divergence) between multivariate distributions be decomposed into relative entropies of the different variables plus some measure of dependence between the ...
0
votes
1answer
71 views

Cross Entopy Loss for classification

Suppose I have a neural network, which classifies pictures of cats, dogs and fishes. The neural network uses Softmax as an activation function of the output layer. Let's say I feed a picture of dog ...
1
vote
1answer
239 views

Interpretation of learning curve (training & validation)

I know what a learning curve representative of an "ideal" overfitting looks like. However, I am not 100% sure how to interpret the learning curve shown below. Why does the model sometimes seem to "...
2
votes
0answers
275 views

How to construct a cross-entropy loss for general regression targets?

It's common short-hand in neural networks literature to refer to categorical cross-entropy loss as "cross-entropy," even though there are a number of loss functions which could properly be described ...
2
votes
1answer
4k views

Cross Entropy Loss for One Hot Encoding

CE-loss sums up the loss over all output nodes $\sum_i[ - target_i*\log(output_i) ]$. The derivative of CE-loss is: $- \frac{target_i}{output_i}$. Since for a target=0 the loss and derivative of ...
0
votes
1answer
225 views

Feed-forward neural network (MSE and Cross-entropy) questions

Question 1 Why do we divide by the number of data points (N)? I think it's done to minimize the error being back-propagated, but can't we just don't do that and instead decrease the learning rate to ...
2
votes
1answer
372 views

Difference of notation between cross entropy and joint entropy

Although it is clear to me, how the two concepts differs, it has been difficult for me to find a notation that would make it clear, to which type of entropy we refer. From wikipedia, we can see that ...
2
votes
0answers
198 views

How can I maximise binary cross entropy loss?

I have a multi-task learning model with two binary classification tasks. One part of the model creates a shared feature representation that is fed into two subnets in parallel. The loss function for ...
4
votes
1answer
722 views

Why is cross entropy not a common evaluation metric for model performance?

When we train a classifier, we use cross entropy as a loss function and, for example, an F-Score as an evaluation metric, but why? Why not use cross entropy on the test set to evaluate the model ...
4
votes
1answer
319 views

Why is the cross-entropy always more than the entropy?

I understand intuitively why cross-entropy is always bigger. However, could someone show that mathematically?
3
votes
1answer
2k views

Why binary crossentropy can be used as the loss function in autoencoders?

I was wondering why binary crossentropy can be used as the loss function in autoencoders trained on (normalized) images, e.g. here or this paper? I know that binary crossentropy can be used in binray ...
2
votes
1answer
454 views

Does it make sense to use `logit` or `softplus` loss for binary classification problem?

With $z$ is the logit, $p \in \{1, 0\}$ is the class. Usually binary classification problem use sigmoid and cross-entropy to compute loss: $$\mathcal{L_1} = - \sum{p \log \sigma(z) + (1-p) \log \sigma(...

1 2 3