Questions tagged [softmax]
Normalizing exponential function which transforms a numeric vector such that all its entries become between 0 and 1 and together sum to 1. It is often used as the final layer of a neural network performing a classification task.
219
questions
0
votes
0
answers
18
views
How to calculate softmax for decreasing values?
I am using a model for a multi-classification / ranking task, however for each choice problem it associates the different options with a number that is in a range with negative numbers with the caveat ...
- 101
0
votes
0
answers
27
views
Why the Transformer model does not require negative sampling but word2vec does?
Both word2vec and transformer model compute a SOFTMAX function over the words/tokens on the output side.
For word2vec models, negative sampling is used for computational reasons:
Is negative sampling ...
- 2,313
0
votes
0
answers
15
views
Specifying priors over softmax outputs
Looking to train a simple single-layered NN with a N-dim softmax output (and a relatively small feature vector size, ~2-10) in a streaming fashion accumulating K samples in a buffer and then ...
- 123
0
votes
0
answers
19
views
How to prevent Softmax to be overconfidence/getting stuck in one solution
I have a Neural Network (Feed Forward Neural Network) with Softmax as the output. The neural network is maximizing some loss (call it $L$), with respect to the input called VAMP2 (there is not True ...
- 557
0
votes
0
answers
21
views
How does softmax work for vectors?
In skipgram we predict the context words. That is the output layer before applying the softmax function is a number $V$ of words, where $V$ is the dictionary size. But each word is represented as a ...
- 1
1
vote
0
answers
25
views
Strange high values in the probability vector in Image Classification
I have successfully trained a ResNet50V2 model. I used transfer learning on ImageNet database, and launch the inference on the unseen data. I obtain a high accuracy (>80%), but when I print the ...
- 51
1
vote
0
answers
11
views
How can we measure the "fit" between the softmax outputs and Dirichlet distribution?
For simplicity, I'll consider classification with 3 classes.
Then, softmax outputs can be considered as the set of points in 2-simplex.
I want to measure the 'fit' of this softmax output with target ...
- 11
0
votes
0
answers
25
views
Classification output from a neural network vs SVM
I have a classification problem which i have run through both a neural network and an SVM. I am interested in the probabilities of all the different classifications rather than just assigning to ...
0
votes
0
answers
22
views
Fast renormalization of distributions
Say I have a matrix probs $\in N\times4$ where each row is a categorical distribution.
However I have also a mask $\in N\times4$ that is meant to remove actions that are not available.
What I used to ...
1
vote
1
answer
48
views
What happens if we follow the gradient of a softmax activation
Given a softmax output layer, what does it mean to "follow the gradient"? Usually that would consist in "increasing the output" but obviously the softmax has no notion of "...
2
votes
1
answer
29
views
would adding redundant classes in the softmax affect the network performance?
I'm training an OCR model for a non-Latin language with 40 chars.
I'll need to combine English at some point which will require some changes to the Architecture and training from scratch.
Would it be ...
0
votes
0
answers
43
views
Derivation of (5.76) in "Pattern Recognition and Machine Learning"
The book "Pattern Recognition and Machine Learning" by Christopher M. Bishop says in page 248
... for softmax outputs we have:
$$\frac{\partial y_k}{\partial a_l}=\delta_{kl}y_k-y_ky_l.\tag{...
- 193
0
votes
0
answers
76
views
How to derive the softmax activation function that corresponds to the canonical link for multiclass classification
My question comes from page 236 of "Pattern Recognition and Machine Learning" by Christopher M. Bishop excerpted below:
Canonical link function is described in section 4.3.6 of the book. ...
- 193
2
votes
1
answer
60
views
How do you optimize multiple objective functions simultaneously?
For example, suppose we want to maximize the 3 expressions on the right, subject to some constraints.
To give some context, this is a problem about generating prototypes in unsupervised learning. In ...
0
votes
0
answers
54
views
Derivation of the sparsemax loss
While reading this interesting paper about a sparse variant of the reknown softmax activation function, I got stuck at the section about loss derivation. In particular, I'm having issues understanding ...
- 557
0
votes
0
answers
27
views
Why the quadratic term is cancelled for general K>2 classes in probabilistic generative models?
The problem comes from a paragraph in section 4.2.1 of Christopher M. Bishop's book "Pattern Recognition and Machine Learning". This is a paragraph that deals with general $K>2$ classes ...
- 193
0
votes
1
answer
47
views
Classifying time horizon of event
Im working on a classification class, where I want to predict the occurrence of a specific event. That is, does the event occur in the next 6 hours, 12 hours or 24 hours? Using softmax here seems not ...
- 1
0
votes
0
answers
29
views
Predict certain values in time series (not next time step)
I have input of numbers as -5,-4..4,5
I only want to predict which value will appear next first 5 or -5, sometime in the furure, not necessarily in the next step
How should I define my model?
- 159
1
vote
0
answers
229
views
Calculating the variance of softmax
I'm working through Dive Into Deep Learning right now and am struggling with the following question:
We can explore the connection between exponential families and the
softmax in some more depth.
...
- 21
1
vote
0
answers
16
views
How can classifiers with softmax output detect new novel/unseen items?
A classifier is said to perform online learning if it can detect novel class from the new input and register this class with classifier.
Since in most classifiers (particularly neural networks) we use ...
- 581
1
vote
0
answers
186
views
How does the full derivative of softmax + cross entropy have the correct dimensions?
The blog post the softmax function and its derivative explains the following:
Imagine that each input has $N$ features / pixels / etc.
Imagine each input can be classified into $C$ classes
Let the ...
- 237
1
vote
0
answers
205
views
Final Layer and Inference with CE vs BCE
I have read a similar question here: 1 neuron BCE loss VS 2 neurons CE loss that suggests there is no difference between softmax cross entropy loss and binary cross entropy loss, when choosing between ...
- 181
1
vote
0
answers
269
views
Why Gumbel softmax and not other types of softmax?
Ignore the method name. Is there a reason why in the Gumbel softmax trick we sample from Gumbel distribution? Since we are doing something similar to a reparameterization trick, can't we just sample ...
0
votes
2
answers
615
views
using logsumexp in softmax
I saw this equation in somebody's code which is an alternative approach to implementing the softmax in order to avoid underflow by division by large numbers.
softmax = e^(matrix - logaddexp(matrix)) = ...
- 277
0
votes
0
answers
66
views
How can I mathematically explain the convolutional neural network?
How can I give notations to the whole CNN network? Like, if I feed input 'x' to feature extractor, we will get the extracted feature embedding vector at the end that will be passed to the linear ...
1
vote
0
answers
55
views
Why is there a meaningful relationship between probabilities of non-ground-truth classes, in the context of knowledge distillation?
In the original knowledge distillation paper Distilling the Knowledge in a Neural Network,
Geoffrey Hinton, Oriol Vinyals and Jeff Dean state the following:
"much of the information about the ...
- 197
1
vote
0
answers
32
views
Hybrid Bayesian Network with Continuous Parents and Ordinal Child Node (Softmax Function)
Upon reading about hybrid BNs with discrete child nodes and multiple continuous parents, I came across the possibility of using the softmax (multinomial logit) function (below) in order to query ...
- 11
1
vote
1
answer
651
views
calculating the top k accuracy using logits vs softmax probabilites
I am working on calculating the top k accuracy of a model my model output logits (I am working on pythorch) so in order to calculate the top k accuracy using sklearn i was wondering what would be the ...
- 11
1
vote
0
answers
766
views
Formal steps for gradient boosting with softmax and cross entropy loss function
Consider some data $\{(x_i,y_i)\}^n_{i=1}$ and a differentiable loss function $\mathcal{L}(y,F(x))$ and a multiclass classification problem which should be solved by a gradient boosting algorithm.
...
- 240
1
vote
1
answer
59
views
Weird Term of Log likelihood
Recall the setup of logistic regression: We assume that the posterior probability is of the form
$p(Y=1|x) = \frac{1}{1+e^{\beta^Tx}}$
This assumes that Y|X is a Bernoulli Random variable. We now turn ...
- 13
1
vote
1
answer
278
views
Binary classification neural network - equivalent implementations with sigmoid and softmax
in order to solidify my understanding, I am doing some simple calculations with pen and paper for some very simple NN for binary classification (input vector with two entries, 1 hidden layer and just ...
- 43
1
vote
0
answers
237
views
Softmax computation in Transformers
In word embedding model word2vec, computation of softmax is expensive process and hence we use many alternative as provided here. Prominently, ...
- 101
0
votes
1
answer
72
views
Sigmoid equivalent to Softmax exercise 2 [duplicate]
This question is not the same as this one I asked previously. In the previous question I asked to prove that the sigmoid and softmax are equivalent. I found a solution here, but I think it's not ...
- 167
0
votes
1
answer
267
views
Sigmoid equivalent to Softmax exercise
I am currently studying the Sutton and Barto Intro To RL Book, and I'm trying to do exercise 2.9 (at the bottom of the following picture):
So the exercise wants me to show that the softmax is ...
- 167
2
votes
1
answer
319
views
Why we use hard targets (generated using the Softmax) but not soft targets or logits
I was reading about knowledge distillation (in student-teacher networks, here) and it is stated that:
Advantages of Soft Targets:
Soft targets contain valuable information on the rich similarity ...
- 171
3
votes
2
answers
328
views
Knowledge Distillation for sigmoid function
I am studying the usage of knowledge distilling and all the contents found on youtube and papers suggests the use of softmax layer as the last layer because of the temperature value and the ...
- 31
1
vote
1
answer
38
views
Model for percent resource allocation between groups
I have a panel data set indicating each entities percent resource allocation among a number of different options. I have been using a logistic regression model to predict the share of allocation for ...
3
votes
1
answer
177
views
exp(log_softmax) vs softmax as neural network activation
I have read about log_softmax being more numerical stable than softmax, since it circumvents the division. I need to use softmax, probabilities between 0 and 1, for my neural network loss function. So ...
1
vote
1
answer
120
views
softmax equation from Goodfellow's Deep Learning book
I'm having trouble understanding the notation in equation 6.31 Goodfellow/Bengio/Courville's Deep Learning book available here.
The equation is
$$
\text{softmax}(\pmb{z}(\pmb{x},\pmb{\theta}))_i = \...
- 155
11
votes
4
answers
2k
views
What is an intuitive interpretation for the softmax transformation?
A recent question on this site asked about the intuition of softmax regression. This has inspired me to ask a corresponding question about the intuitive meaning of the softmax transformation itself. ...
- 112k
0
votes
0
answers
36
views
Can we perform any machine learning modelling by using Neural Network?
I am trying to understand the powerful modelling capacity of neural network. As we know
Logistic regression can be performed by using neural network with single output node, no hidden layer and ...
- 71
1
vote
0
answers
95
views
Can we have an explicit representation of inverse of a softmax function?
As we know, logit is the inverse of logistic function in case of binary classification. Similar to this, I am willing to derive results for multinomial classification, for that I need to get an ...
- 71
1
vote
0
answers
138
views
Categorical Cross Entropy Loss Derivation
I understand the categorical cross entropy based loss function to be the following.
$$J(w) = \sum_{i=1}^ny_i\ln[P(y_i|x_i,w)]$$
where $$\ln\left[P(y_i|x_i,w)\right] = \sum_{i=k}^Kr_{ik}P(y_i=k|x_i,w)$$...
8
votes
1
answer
3k
views
Why does torchvision.models.resnet18 not use softmax?
I see image-classification models from torchvision package don't have a softmax layer as final layer. For instance, the following snippet easily shows that the resnet18 output doesn't have a sum = 1, ...
1
vote
1
answer
288
views
How to prove the monotonicity of Softmax when decreasing the weighted inputs?
Softmax function $a^L_j = z^L_j / sum_k(z^L_k)$. When we think about the monotonicity of the Softmax function, $∂a^L_j/∂z^L_k$ is positive if j=k, and negative if j≠k. As a consequence, increasing $z^...
- 79
2
votes
1
answer
64
views
Need help understanding inverting softmax from Michael Nielsen's book
I have a hard time understanding a softmax problem from the book:
Inverting the softmax layer Suppose we have a neural network with a softmax output layer
and the > activations $a^L_j$ are known. ...
- 79
1
vote
2
answers
715
views
For prediction problems, why cant we simply use softmax as activation for hidden layers and no activation function for output layer
I am a beginner who is learning how to use neural networks for prediction problems, and almost all tutorials I have found use relu for hidden layer and sigmoid function for output later. My question ...
- 11
17
votes
2
answers
10k
views
What is the role of temperature in Softmax?
I'm recently working on CNN and I want to know what is the function of temperature in the softmax formula? and why should we use high temperatures to see a softer norm in probability distribution?
The ...
- 273
0
votes
1
answer
314
views
Word2vec/SkipGram: Why softmax?
In Word2Vec (SkipGram version), there is a softmax layer at the end of the neural net. As this is expansive to calculate, some approximations are used instead, such as negative sampling.
But if in the ...
- 3
0
votes
0
answers
18
views
Predicting in Neural Network with Softmax or Logistic activation output [duplicate]
So I'm wanting to make a neural network from scratch for predicting classifications and labels. The activation layer for the output in these networks would be softmax for classification and logistic ...
- 111