Questions tagged [softmax]

Normalizing exponential function which transforms a numeric vector such that all its entries become between 0 and 1 and together sum to 1. It is often used as the final layer of a neural network performing a classification task.

Filter by
Sorted by
Tagged with
0 votes
0 answers
9 views

Can exponentiated coefficients in multinominal logistic regression (with softmax normalization) be interpreted as odds ratios?

Exponentiated logistic regression coefficients are interpreted as odds ratios. Does this still hold in multinominal logistic regression, where softmax is typically used to normalize probabilities to ...
tomas's user avatar
  • 381
0 votes
0 answers
10 views

Ensuring numerical stability of probabilities in identifiable multinomial regression

I'm looking to understand the differences between the functions for probabilities induced in an identified multinomial GLM, and the softmax function commonly used in classification for neural networks....
bill's user avatar
  • 1
0 votes
0 answers
30 views

Is this the correct hypothesis of softmax regression?

Consider a dataset of $m$ training examples, $n$ features and $K$ classes. So we have a feature matrix $\mathbf{X} \in \mathbb{M}_{m, n}(\mathbb{R})$ and a weight matrix $\boldsymbol{\Theta} \in \...
Sagnik Taraphdar's user avatar
1 vote
0 answers
18 views

Compare multiclass probabilities for two classifiers

I have two algorithms that produce, for every observation, a vector of probabilities for 3 classes ...
Alessandro Bitetto's user avatar
0 votes
0 answers
103 views

Is it better to use KL Divergence for soft labels instead of cross entropy?

So I was going through this paper: Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (2017) by Junnan Li et al., in this they perform contrastive loss using KL ...
mutli-arm-bandit's user avatar
0 votes
0 answers
27 views

Can a softmax probability distribution be approximated to Beta distribution?

I am wondering if attention weights in BERT language model layers ( which ,in essence, are softmax probability distribution) can be approximated with beta distribution? With approximated beta ...
gray KK's user avatar
  • 11
1 vote
0 answers
89 views

How can the softmax distribution be used to detect out-of-distribution samples?

I am reading this paper and it states that - "In what follows we retrieve the maximum/predicted class probability from a softmax distribution and thereby detect whether an example is erroneously ...
desert_ranger's user avatar
5 votes
1 answer
196 views

What's the relation between the output of a neural network and a Multinomial distribution?

I am reading this paper, which has the following paragraph - "The gold standard for deep neural nets is to use the softmax operator to convert the continuous activations of the output layer to ...
desert_ranger's user avatar
0 votes
0 answers
25 views

Classification in the context of a time series

I would like to exchange with you about how to best do this. Let me try to clearly describe the situation in the most non-verbose way I can! (Stack: Python, Darts (from unit8), pandas, numpy) I have a ...
MDSv's user avatar
  • 33
1 vote
1 answer
48 views

Overparameterization at the final layer of a neural network

I'm considering a multi-class classifying neural network with softmax on the final layer. Should one of the classes of the weight matrix of the final layer not be optimised over to account for the ...
oweydd's user avatar
  • 225
0 votes
1 answer
254 views

Do ROC curves require probabilities?

In the binary case, the implementation of ROC curve in torchmetrics automatically applies a sigmoid when it detects logit inputs (i.e. when the values of scores are ...
Alex Bogatskiy's user avatar
2 votes
1 answer
293 views

Newton's method and the Hessian for Softmax Logistic Regression

I'm having some trouble with optimising softmax regression via Newton's method. I'm not sure if the problem is arising with my equations for the Hessian and Gradient or with the code I've written. For ...
oweydd's user avatar
  • 225
0 votes
0 answers
623 views

Why use Gumbel softmax instead of taking the (soft)argmax of the logits (or softmax then argmax)?

My understanding is that the goal of using Gumbel softmax is to change an output that contains logits into a one-hot vector corresponding to the highest probability choice (based on those logits). ...
Dawid's user avatar
  • 1
1 vote
0 answers
28 views

Monotonicity of softmax (considering updates from all variables)

There's a relevant question here that doesn't quite answer my question, but I'm unable to comment. Define softmax to be $$a_i = \text{softmax}(u_i)= \frac{e^{u_i}}{\sum_j{e^{u_j}}}$$ As the linked ...
gcpx100's user avatar
  • 11
1 vote
0 answers
142 views

Understanding Backpropagation with Softmax and Quadratic Error

I'm trying to understand how to compute the derivative of the Softmax activation function in order to compute the gradient of the quadratic cost function w.r.t. the weights of the last layer. ...
Dario Ranieri's user avatar
0 votes
0 answers
27 views

How to calculate softmax for decreasing values?

I am using a model for a multi-classification / ranking task, however for each choice problem it associates the different options with a number that is in a range with negative numbers with the caveat ...
Iqigai's user avatar
  • 101
1 vote
1 answer
275 views

Why the Transformer model does not require negative sampling but word2vec does?

Both word2vec and transformer model compute a SOFTMAX function over the words/tokens on the output side. For word2vec models, negative sampling is used for computational reasons: Is negative sampling ...
CyberPlayerOne's user avatar
0 votes
0 answers
30 views

Specifying priors over softmax outputs

Looking to train a simple single-layered NN with a N-dim softmax output (and a relatively small feature vector size, ~2-10) in a streaming fashion accumulating K samples in a buffer and then ...
Sharpfawkes's user avatar
0 votes
0 answers
105 views

How to prevent Softmax to be overconfidence/getting stuck in one solution

I have a Neural Network (Feed Forward Neural Network) with Softmax as the output. The neural network is maximizing some loss (call it $L$), with respect to the input called VAMP2 (there is not True ...
Daniel Wiczew's user avatar
1 vote
0 answers
52 views

How does softmax work for vectors?

In skipgram we predict the context words. That is the output layer before applying the softmax function is a number $V$ of words, where $V$ is the dictionary size. But each word is represented as a ...
Ruediger's user avatar
1 vote
0 answers
27 views

Strange high values in the probability vector in Image Classification

I have successfully trained a ResNet50V2 model. I used transfer learning on ImageNet database, and launch the inference on the unseen data. I obtain a high accuracy (>80%), but when I print the ...
Jonny_92's user avatar
1 vote
0 answers
18 views

How can we measure the "fit" between the softmax outputs and Dirichlet distribution?

For simplicity, I'll consider classification with 3 classes. Then, softmax outputs can be considered as the set of points in 2-simplex. I want to measure the 'fit' of this softmax output with target ...
Erik's user avatar
  • 11
0 votes
0 answers
25 views

Classification output from a neural network vs SVM

I have a classification problem which i have run through both a neural network and an SVM. I am interested in the probabilities of all the different classifications rather than just assigning to ...
user379716's user avatar
0 votes
0 answers
31 views

Fast renormalization of distributions

Say I have a matrix probs $\in N\times4$ where each row is a categorical distribution. However I have also a mask $\in N\times4$ that is meant to remove actions that are not available. What I used to ...
Alberto's user avatar
  • 1,159
2 votes
1 answer
138 views

What happens if we follow the gradient of a softmax activation

Given a softmax output layer, what does it mean to "follow the gradient"? Usually that would consist in "increasing the output" but obviously the softmax has no notion of "...
Alberto's user avatar
  • 1,159
2 votes
1 answer
31 views

would adding redundant classes in the softmax affect the network performance?

I'm training an OCR model for a non-Latin language with 40 chars. I'll need to combine English at some point which will require some changes to the Architecture and training from scratch. Would it be ...
Mohamed Nabil's user avatar
0 votes
0 answers
59 views

Derivation of (5.76) in "Pattern Recognition and Machine Learning"

The book "Pattern Recognition and Machine Learning" by Christopher M. Bishop says in page 248 ... for softmax outputs we have: $$\frac{\partial y_k}{\partial a_l}=\delta_{kl}y_k-y_ky_l.\tag{...
zzzhhh's user avatar
  • 323
2 votes
1 answer
80 views

How do you optimize multiple objective functions simultaneously?

For example, suppose we want to maximize the 3 expressions on the right, subject to some constraints. To give some context, this is a problem about generating prototypes in unsupervised learning. In ...
Machine Learner's user avatar
0 votes
1 answer
66 views

Classifying time horizon of event

Im working on a classification class, where I want to predict the occurrence of a specific event. That is, does the event occur in the next 6 hours, 12 hours or 24 hours? Using softmax here seems not ...
f_3464gh's user avatar
1 vote
1 answer
534 views

Calculating the variance of softmax

I'm working through Dive Into Deep Learning right now and am struggling with the following question: We can explore the connection between exponential families and the softmax in some more depth. ...
jimac82's user avatar
  • 21
2 votes
0 answers
23 views

How can classifiers with softmax output detect new novel/unseen items?

A classifier is said to perform online learning if it can detect novel class from the new input and register this class with classifier. Since in most classifiers (particularly neural networks) we use ...
wd violet's user avatar
  • 777
1 vote
0 answers
230 views

How does the full derivative of softmax + cross entropy have the correct dimensions?

The blog post the softmax function and its derivative explains the following: Imagine that each input has $N$ features / pixels / etc. Imagine each input can be classified into $C$ classes Let the ...
Foobar's user avatar
  • 349
1 vote
0 answers
485 views

Final Layer and Inference with CE vs BCE

I have read a similar question here: 1 neuron BCE loss VS 2 neurons CE loss that suggests there is no difference between softmax cross entropy loss and binary cross entropy loss, when choosing between ...
Anonymous's user avatar
  • 181
1 vote
0 answers
393 views

Why Gumbel softmax and not other types of softmax?

Ignore the method name. Is there a reason why in the Gumbel softmax trick we sample from Gumbel distribution? Since we are doing something similar to a reparameterization trick, can't we just sample ...
Junhan Ouyang's user avatar
0 votes
2 answers
1k views

using logsumexp in softmax

I saw this equation in somebody's code which is an alternative approach to implementing the softmax in order to avoid underflow by division by large numbers. softmax = e^(matrix - logaddexp(matrix)) = ...
zipline86's user avatar
  • 287
0 votes
0 answers
80 views

How can I mathematically explain the convolutional neural network?

How can I give notations to the whole CNN network? Like, if I feed input 'x' to feature extractor, we will get the extracted feature embedding vector at the end that will be passed to the linear ...
Tariq Hussain's user avatar
1 vote
0 answers
65 views

Why is there a meaningful relationship between probabilities of non-ground-truth classes, in the context of knowledge distillation?

In the original knowledge distillation paper Distilling the Knowledge in a Neural Network, Geoffrey Hinton, Oriol Vinyals and Jeff Dean state the following: "much of the information about the ...
Aditya Mehrotra's user avatar
1 vote
0 answers
40 views

Hybrid Bayesian Network with Continuous Parents and Ordinal Child Node (Softmax Function)

Upon reading about hybrid BNs with discrete child nodes and multiple continuous parents, I came across the possibility of using the softmax (multinomial logit) function (below) in order to query ...
fluxeroth's user avatar
1 vote
1 answer
937 views

calculating the top k accuracy using logits vs softmax probabilites

I am working on calculating the top k accuracy of a model my model output logits (I am working on pythorch) so in order to calculate the top k accuracy using sklearn i was wondering what would be the ...
oubo21's user avatar
  • 11
1 vote
0 answers
999 views

Formal steps for gradient boosting with softmax and cross entropy loss function

Consider some data $\{(x_i,y_i)\}^n_{i=1}$ and a differentiable loss function $\mathcal{L}(y,F(x))$ and a multiclass classification problem which should be solved by a gradient boosting algorithm. ...
mugdi's user avatar
  • 90
1 vote
1 answer
85 views

Weird Term of Log likelihood

Recall the setup of logistic regression: We assume that the posterior probability is of the form $p(Y=1|x) = \frac{1}{1+e^{\beta^Tx}}$ This assumes that Y|X is a Bernoulli Random variable. We now turn ...
kursat's user avatar
  • 13
1 vote
1 answer
423 views

Binary classification neural network - equivalent implementations with sigmoid and softmax

in order to solidify my understanding, I am doing some simple calculations with pen and paper for some very simple NN for binary classification (input vector with two entries, 1 hidden layer and just ...
user120112's user avatar
3 votes
0 answers
330 views

Softmax computation in Transformers

In word embedding model word2vec, computation of softmax is expensive process and hence we use many alternative as provided here. Prominently, ...
Lawhatre's user avatar
  • 171
0 votes
1 answer
120 views

Sigmoid equivalent to Softmax exercise 2 [duplicate]

This question is not the same as this one I asked previously. In the previous question I asked to prove that the sigmoid and softmax are equivalent. I found a solution here, but I think it's not ...
Slim Shady's user avatar
1 vote
1 answer
450 views

Sigmoid equivalent to Softmax exercise

I am currently studying the Sutton and Barto Intro To RL Book, and I'm trying to do exercise 2.9 (at the bottom of the following picture): So the exercise wants me to show that the softmax is ...
Slim Shady's user avatar
2 votes
1 answer
524 views

Why we use hard targets (generated using the Softmax) but not soft targets or logits

I was reading about knowledge distillation (in student-teacher networks, here) and it is stated that: Advantages of Soft Targets: Soft targets contain valuable information on the rich similarity ...
Mas A's user avatar
  • 203
4 votes
2 answers
598 views

Knowledge Distillation for sigmoid function

I am studying the usage of knowledge distilling and all the contents found on youtube and papers suggests the use of softmax layer as the last layer because of the temperature value and the ...
e_rcon's user avatar
  • 41
1 vote
1 answer
50 views

Model for percent resource allocation between groups

I have a panel data set indicating each entities percent resource allocation among a number of different options. I have been using a logistic regression model to predict the share of allocation for ...
acertainKnight's user avatar
3 votes
1 answer
325 views

exp(log_softmax) vs softmax as neural network activation

I have read about log_softmax being more numerical stable than softmax, since it circumvents the division. I need to use softmax, probabilities between 0 and 1, for my neural network loss function. So ...
CreeperPower storing's user avatar
1 vote
1 answer
153 views

softmax equation from Goodfellow's Deep Learning book

I'm having trouble understanding the notation in equation 6.31 Goodfellow/Bengio/Courville's Deep Learning book available here. The equation is $$ \text{softmax}(\pmb{z}(\pmb{x},\pmb{\theta}))_i = \...
NNN's user avatar
  • 155

1
2 3 4 5