Questions tagged [cross-entropy]
The cross-entropy tag has no usage guidance.
1
vote
1answer
18 views
cross entropy loss max value
The cross entropy loss function for multiclass can be computed as:
$$-\sum\limits_{i=1}^N y_i log \hat{y}_i$$
where $y_i$ is a class and $\hat{y}_i$ the estimated probability. The minimum value is $0$ ...
0
votes
0answers
16 views
how to consider some miss classifications “half correct” in categorical_crossentropy - for a trading system
I have a trading system where the model receives 9 time-series and predict :
A - strong down
B - week down
C - neutral
D - week up
E - strong up
(these classes ...
1
vote
1answer
27 views
Why is loss treated as 0 or infinity here?
I'm reading a beginner's book about deep learning and in the introduction, the following cautionary tale is written:
Now, assume that you built a classifier and trained it to predict if a mushroom ...
0
votes
0answers
10 views
loss='categorical_crossentropy' VS loss=K.categorical_crossentropy
Why I have very different loss values in training using these two lines code to define the loss function?
...
0
votes
0answers
29 views
Softmax Loss vs Binary Loss for classification?
I was trying to understand the final section of the paper "Revisiting Baselines for Visual Question Answering". Authors state that their model performs better with a binary loss in comparison to a ...
0
votes
0answers
32 views
Cross entropy vs KL divergence and SGD
I was told recently that stochastic gradient descent can be used to minimize a cross entropy objective function but not a KL-divergence one, even if the two minimization problems are equivalent, ie
$...
0
votes
0answers
29 views
Cross - entropy for two variables with different prob. distributions
Let us say that we have given two random variables with different prob. distributions:
A = [0.1, 0, 0.5, ...]
B = [0.3, 0.1, 0.03, ...]
What should I do when I want to compute the reformulated cross-...
0
votes
1answer
56 views
Approach to prevent bias/racism in neural network fitting?
I have a dataset comprised of different ethnic groups and I want to build a classification model on this data. When I do this I find that the performance of the algorithm is better on some groups than ...
0
votes
1answer
36 views
Cross entropy loss: inconsistency in formula
I have a couple of problems trying to understand the exact formula for cross entropy loss. Depending on the source I see it written different ways.
Is the log() function $\log_2()$?
Is the argument ...
1
vote
0answers
20 views
Binary cross etropy loss with non binary ground truth data [closed]
Is it possible to use binary cross etropy loss with non binary ground truth data, i.e. not [0,1] values, but [0,0.1,0.5,1.0] ...
2
votes
1answer
31 views
Relative Entropy decomposition
Can the relative entropy (Kullback Leibler divergence) between multivariate distributions be decomposed into relative entropies of the different variables plus some measure of dependence between the ...
0
votes
0answers
246 views
Gradient of the cross entropy loss function
I have been puzzled by how to calculate the derivative of the following cross entropy loss function underlying my neural network:
CEloss = $\frac{-1}{N} \sum_{n=1}^{N} \sum_{k=1}^{K} t_{n,k} \log y_{...
0
votes
1answer
47 views
Cross Entopy Loss for classification
Suppose I have a neural network, which classifies pictures of cats, dogs and fishes. The neural network uses Softmax as an activation function of the output layer.
Let's say I feed a picture of dog ...
1
vote
1answer
54 views
Interpretation of learning curve (training & validation)
I know what a learning curve representative of an "ideal" overfitting looks like. However, I am not 100% sure how to interpret the learning curve shown below. Why does the model sometimes seem to "...
1
vote
0answers
72 views
How to construct a cross-entropy loss for general regression targets?
It's common short-hand in neural networks literature to refer to categorical cross-entropy loss as "cross-entropy," even though there are a number of loss functions which could properly be described ...
1
vote
1answer
508 views
Cross Entropy Loss for One Hot Encoding
CE-loss sums up the loss over all output nodes
$\sum_i[ - target_i*\log(output_i) ]$.
The derivative of CE-loss is: $- \frac{target_i}{output_i}$.
Since for a target=0 the loss and derivative of ...
0
votes
1answer
48 views
Feed-forward neural network (MSE and Cross-entropy) questions
Question 1
Why do we divide by the number of data points (N)? I think it's done to minimize the error being back-propagated, but can't we just don't do that and instead decrease the learning rate to ...
0
votes
1answer
55 views
Difference of notation between cross entropy and joint entropy
Although it is clear to me, how the two concepts differs, it has been difficult for me to find a notation that would make it clear, to which type of entropy we refer.
From wikipedia, we can see that ...
1
vote
0answers
68 views
How can I maximise binary cross entropy loss?
I have a multi-task learning model with two binary classification tasks. One part of the model creates a shared feature representation that is fed into two subnets in parallel. The loss function for ...
4
votes
1answer
192 views
Why is cross entropy not a common evaluation metric for model performance?
When we train a classifier, we use cross entropy as a loss function and, for example, an F-Score as an evaluation metric, but why?
Why not use cross entropy on the test set to evaluate the model ...
1
vote
1answer
60 views
Why is the cross-entropy always more than the entropy?
I understand intuitively why cross-entropy is always bigger. However, could someone show that mathematically?
2
votes
1answer
450 views
Why binary crossentropy can be used as the loss function in autoencoders?
I was wondering why binary crossentropy can be used as the loss function in autoencoders trained on (normalized) images, e.g. here or this paper? I know that binary crossentropy can be used in binray ...
0
votes
0answers
24 views
How to calculate negative log-likelihoog on MNIST dataset? [duplicate]
This table is from Professor Forcing: A New Algorithm for Training Recurrent Networks paper. I couldn't find their code to calculate NLL. I would like to ask if it is simply the binary cross-entropy. ...
2
votes
1answer
96 views
Does it make sense to use `logit` or `softplus` loss for binary classification problem?
With $z$ is the logit, $p \in \{1, 0\}$ is the class. Usually binary classification problem use sigmoid and cross-entropy to compute loss:
$$\mathcal{L_1} = - \sum{p \log \sigma(z) + (1-p) \log \sigma(...
1
vote
1answer
244 views
Is there an alternative to categorical cross-entropy with a notion of “class distance”?
I have a signal $ x \in \mathbb{R}^{t \times l} $ which is discretized into $ l = 32 $ levels for $ t = 100000 $ time points. This enables me to turn a regression problem into a classification problem,...
0
votes
1answer
135 views
Why does cross entropy loss for validation dataset deteriorate far more than validation accuracy when a CNN is overfitting?
I have noticed that the cross-entropy loss for validation dataset deteriorates after a certain number of epochs when training CNN's or MLP's. This is, of course, the sign that the network is ...
3
votes
2answers
109 views
Why we use log function for cross entropy?
I'm learning about a binary classifier.
It uses the cross-entropy function as its loss function.
$y_i \log p_i + (1-y_i) \log(1-p_i)$
But why does it use the log function?
How about just use linear ...
1
vote
0answers
24 views
Describing relative performance of models on the basis on log-loss / cross-entropy
Suppose I train classification algorithms on a given dataset with three model specifications: A, B, and C. On validation data, model A has an average log-loss of 1.0, model B has a log-loss of 0.75, ...
1
vote
1answer
70 views
Maximum likelihood estimation and $\log(0)$
I am thinking I can't be the only one encountering this. I am trying to do maximum likelihood estimation on a probit model, i.e. trying to find the most optimal fit for three parameters in my case ...
4
votes
1answer
313 views
the relationship between maximizing the likelihood and minimizing the cross-entropy
There is a statement that maximizing the likelihood is equivalent to minimizing the cross-entropy. Are there any proof for this statement?
0
votes
0answers
64 views
Computing Gini index and cross-entropy for a dendrogram
I am trying to calculate purity of clusters obtained through hierarchical clustering. I followed this tutorial here: Decision tree learning
My question is: is computing these measures the same for ...
1
vote
1answer
84 views
Proof for the efficiency of Softmax in multi-classification
I already search for this question but I can't find any convincing explanation so I want to ask it here. my problem is with softmax activation function and cross-entropy.why they can produce a better ...
0
votes
1answer
56 views
What is the correct answer for cross entropy in this case?
I have two neural networks: NN1 and NN2, predicting Dog and Cat. The probabilities are below:
...
4
votes
1answer
1k views
What is the difference Cross-entropy and KL divergence?
Both of Cross-entropy and KL divergence are tools to measure the distance between two probability distribution. What is the difference?
$$ H(P,Q) = -\sum_x P(x)\log Q(x) $$
$$ KL(P | Q) = \sum_{x} P(...
0
votes
1answer
89 views
Difference between Empirical distribution and Bernoulli distribution
I've been studying binary cross entropy error for binary classification weight optimization.
From my knowledge, Cross entropy itself quantifies divergence between two probability distributions with ...
0
votes
0answers
20 views
Can cross entropy be employed for measuring effective mapping of cross-modal data?
I am working on a new metric for cross-modal retrieval which can measure the effectiveness of mapping two modalities on a manifold. However, the usual approach is to employ a distance metric and not ...
1
vote
1answer
62 views
Cross-entropy yields strange results when neural network gets too sure about his outputs
I'm using a classical CNN for image binary classification. Output is composed of two neurons, each giving the network's "raw output" for the 2 classes. So for an image, it would be eg $(0.62, -0.52)$.
...
1
vote
1answer
42 views
CNN with categorical response - how is “other” or “null” classes represented in output?
I have a created a Convolution Neural Network in Keras for an image classification problem. The purpose of the CNN is to identify whether either of the two classes (e.g. dog and cat) is present in ...
1
vote
0answers
83 views
Alternative measurement for Cross Entropy Loss than accuracy when using soft labeling?
Suppose one has multi-class ($n>2$) data, where class labels are soft, e.g. two samples of for 3-classed data might look like:
...
0
votes
0answers
101 views
change hinge loss error function with cross-entropy
I'm trying to implement cross-entropy as an error function in RBF neural networks instead of hinge loss error function. I need to find cross-entropy error for each output neuron, like hinge loss error ...
0
votes
0answers
48 views
Regression tree with cross-entropy loss
I need to predict the categorical distribution with the decision tree.
Formally there is a map $ f: X \rightarrow \{(p_1, ..., p_k): p_i \geq 0, \, \text{and} \sum_i p_i = 1\}$ defined on the given ...
0
votes
1answer
125 views
How is the Cross-Entropy Cost Function back-propagated?
I've looked at a few threads about this but they've not been exactly what I'm after. When back-propagating the quadratic cost function, you first find the output error from $\delta_L = \...
1
vote
1answer
752 views
Cross-entropy for comparing images
Suppose we have two greyscale images which are flattened to 1d arrays: $y=(y_1, y_2, \ldots, y_n)$ and $\hat{y} = (\hat{y}_1, \hat{y}_2, \ldots, \hat{y}_n)$ with pixel values in $[0,1]$. How exactly ...
0
votes
1answer
158 views
Optimal neural net weights when using cross entropy loss
I'm trying to understand how cross-entropy works for finding the optimal weights in neural networks. According to Eli Bendersky's website and neural networks and deeplearning tutorial, we can find the ...
1
vote
1answer
592 views
Difference between mathematical and Tensorflow implementation of Softmax Crossentropy with logit
Softmax cross entropy with logits is define as follows:
$a_i = \frac{e^{z_i}}{\sum_{\forall j} e^{z_j}}$
$l={\sum_{\forall i}}y_ilog(a_i)$
Where $l$ is the actual loss.
But when you look deep into ...
1
vote
0answers
244 views
Multi-label classification, binary loss concerns
I am solving a multi-label audio classification task with neural networks. The dataset is comprised of 10 classes, and the input data to the network are audio files where two of these classes are ...
1
vote
1answer
2k views
Deriving binary cross entropy loss function
I want to know how the equation for binary cross entropy came about. My approach is the following:
Let's say we have two ground truths: $y_1$ and $y_2$. We also have two predictions $p_1$ and $p_2$. ...
1
vote
1answer
183 views
analysing cross-entropy vs epochs curve
I am training a binary classification neural network model using matlab the graph that I got using 20 neurons in hidden layer is given below.
the confusion matrix and graph between cross entropy vs ...
2
votes
2answers
403 views
Entropy of a function of independent random variables
Suppose I have an operator (function) $f(\cdot)$ which takes three arguments $x,y,z$ all of which are independent random variables, and all of which I have access to the probability mass function (...
1
vote
0answers
81 views
Cross entropy of product of sequence for sequence tagging
So I'm training a charterer level NN to do sequence tagging, simple NER (Person, Object, ...)
Say I have a training example like (and perfect knowledge of how to tokenize the text )
...
