Questions tagged [softmax]

Normalizing exponential function which transforms a numeric vector such that all its entries become between 0 and 1 and together sum to 1. It is often used as the final layer of a neural network performing a classification task.

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

Why Gumbel softmax and not other types of softmax?

Ignore the method name. Is there a reason why in the Gumbel softmax trick we sample from Gumbel distribution? Since we are doing something similar to a reparameterization trick, can't we just sample ...
user avatar
0 votes
2 answers
33 views

using logsumexp in softmax

I saw this equation in somebody's code which is an alternative approach to implementing the softmax in order to avoid underflow by division by large numbers. softmax = e^(matrix - logaddexp(matrix)) = ...
user avatar
  • 237
0 votes
0 answers
10 views

Do we use softmax even in binary classification where there is only one neuron in the output layer? [duplicate]

I think it's recommended to use one neuron in the output layer of a network if you are doing binary classification, because one neuron by itself can represent two states (0 or 1). But then softmax ...
user avatar
  • 55
0 votes
0 answers
21 views

How can I mathematically explain the convolutional neural network?

How can I give notations to the whole CNN network? Like, if I feed input 'x' to feature extractor, we will get the extracted feature embedding vector at the end that will be passed to the linear ...
user avatar
1 vote
0 answers
49 views

Why is there a meaningful relationship between probabilities of non-ground-truth classes, in the context of knowledge distillation?

In the original knowledge distillation paper Distilling the Knowledge in a Neural Network, Geoffrey Hinton, Oriol Vinyals and Jeff Dean state the following: "much of the information about the ...
user avatar
1 vote
0 answers
18 views

Hybrid Bayesian Network with Continuous Parents and Ordinal Child Node (Softmax Function)

Upon reading about hybrid BNs with discrete child nodes and multiple continuous parents, I came across the possibility of using the softmax (multinomial logit) function (below) in order to query ...
user avatar
1 vote
1 answer
114 views

calculating the top k accuracy using logits vs softmax probabilites

I am working on calculating the top k accuracy of a model my model output logits (I am working on pythorch) so in order to calculate the top k accuracy using sklearn i was wondering what would be the ...
user avatar
  • 11
1 vote
0 answers
229 views

Formal steps for gradient boosting with softmax and cross entropy loss function

Consider some data $\{(x_i,y_i)\}^n_{i=1}$ and a differentiable loss function $\mathcal{L}(y,F(x))$ and a multiclass classification problem which should be solved by a gradient boosting algorithm. ...
user avatar
  • 218
1 vote
1 answer
39 views

Weird Term of Log likelihood

Recall the setup of logistic regression: We assume that the posterior probability is of the form $p(Y=1|x) = \frac{1}{1+e^{\beta^Tx}}$ This assumes that Y|X is a Bernoulli Random variable. We now turn ...
user avatar
  • 13
1 vote
1 answer
67 views

Binary classification neural network - equivalent implementations with sigmoid and softmax

in order to solidify my understanding, I am doing some simple calculations with pen and paper for some very simple NN for binary classification (input vector with two entries, 1 hidden layer and just ...
user avatar
1 vote
0 answers
19 views

Softmax computation in Transformers

In word embedding model word2vec, computation of softmax is expensive process and hence we use many alternative as provided here. Prominently, ...
user avatar
  • 101
0 votes
1 answer
42 views

Sigmoid equivalent to Softmax exercise 2 [duplicate]

This question is not the same as this one I asked previously. In the previous question I asked to prove that the sigmoid and softmax are equivalent. I found a solution here, but I think it's not ...
user avatar
0 votes
1 answer
47 views

Sigmoid equivalent to Softmax exercise

I am currently studying the Sutton and Barto Intro To RL Book, and I'm trying to do exercise 2.9 (at the bottom of the following picture): So the exercise wants me to show that the softmax is ...
user avatar
2 votes
1 answer
83 views

Why we use hard targets (generated using the Softmax) but not soft targets or logits

I was reading about knowledge distillation (in student-teacher networks, here) and it is stated that: Advantages of Soft Targets: Soft targets contain valuable information on the rich similarity ...
user avatar
  • 189
3 votes
2 answers
69 views

Knowledge Distillation for sigmoid function

I am studying the usage of knowledge distilling and all the contents found on youtube and papers suggests the use of softmax layer as the last layer because of the temperature value and the ...
user avatar
  • 31
1 vote
1 answer
21 views

Model for percent resource allocation between groups

I have a panel data set indicating each entities percent resource allocation among a number of different options. I have been using a logistic regression model to predict the share of allocation for ...
user avatar
2 votes
1 answer
48 views

exp(log_softmax) vs softmax as neural network activation

I have read about log_softmax being more numerical stable than softmax, since it circumvents the division. I need to use softmax, probabilities between 0 and 1, for my neural network loss function. So ...
user avatar
1 vote
1 answer
60 views

softmax equation from Goodfellow's Deep Learning book

I'm having trouble understanding the notation in equation 6.31 Goodfellow/Bengio/Courville's Deep Learning book available here. The equation is $$ \text{softmax}(\pmb{z}(\pmb{x},\pmb{\theta}))_i = \...
user avatar
  • 155
11 votes
4 answers
1k views

What is an intuitive interpretation for the softmax transformation?

A recent question on this site asked about the intuition of softmax regression. This has inspired me to ask a corresponding question about the intuitive meaning of the softmax transformation itself. ...
user avatar
  • 94.4k
0 votes
0 answers
29 views

Approximation of a confidence scores from a neural network with a final softmax layer: Softmax vs other normalization methods

Say that there is a neural network for classification and the 2nd to last layer are 3 nodes, and the final layer is a softmax layer. During training the softmax layer is needed, but for inference it ...
user avatar
0 votes
0 answers
32 views

Can we perform any machine learning modelling by using Neural Network?

I am trying to understand the powerful modelling capacity of neural network. As we know Logistic regression can be performed by using neural network with single output node, no hidden layer and ...
user avatar
0 votes
0 answers
21 views

Can we have an explicit representation of inverse of a softmax function?

As we know, logit is the inverse of logistic function in case of binary classification. Similar to this, I am willing to derive results for multinomial classification, for that I need to get an ...
user avatar
0 votes
0 answers
26 views

How does prediction of the fraction of counts of each outcome observed in the training set by Softmax Units work?

Currently, I am reading a book "Deep Learning" by Aaron Courville, Ian Goodfellow, and Yoshua Bengio. They provide the next formula of Softmax function $ softmax(\boldsymbol{z};\boldsymbol{x,...
user avatar
1 vote
0 answers
58 views

Categorical Cross Entropy Loss Derivation

I understand the categorical cross entropy based loss function to be the following. $$J(w) = \sum_{i=1}^ny_i\ln[P(y_i|x_i,w)]$$ where $$\ln\left[P(y_i|x_i,w)\right] = \sum_{i=k}^Kr_{ik}P(y_i=k|x_i,w)$$...
user avatar
5 votes
1 answer
1k views

Why does torchvision.models.resnet18 not use softmax?

I see image-classification models from torchvision package don't have a softmax layer as final layer. For instance, the following snippet easily shows that the resnet18 output doesn't have a sum = 1, ...
user avatar
1 vote
1 answer
55 views

How to prove the monotonicity of Softmax when decreasing the weighted inputs?

Softmax function $a^L_j = z^L_j / sum_k(z^L_k)$. When we think about the monotonicity of the Softmax function, $∂a^L_j/∂z^L_k$ is positive if j=k, and negative if j≠k. As a consequence, increasing $z^...
user avatar
  • 79
2 votes
1 answer
32 views

Need help understanding inverting softmax from Michael Nielsen's book

I have a hard time understanding a softmax problem from the book: Inverting the softmax layer Suppose we have a neural network with a softmax output layer and the > activations $a^L_j$ are known. ...
user avatar
  • 79
0 votes
1 answer
143 views

For prediction problems, why cant we simply use softmax as activation for hidden layers and no activation function for output layer

I am a beginner who is learning how to use neural networks for prediction problems, and almost all tutorials I have found use relu for hidden layer and sigmoid function for output later. My question ...
user avatar
0 votes
0 answers
19 views

Why the output of softmax can be saturated when input differences become extreme? [duplicate]

Like sigmoid function that can be saturated when inputs are extremely different. But how these inputs can cause this function to be saturated?
user avatar
10 votes
2 answers
2k views

What is the role of temperature in Softmax?

I'm recently working on CNN and I want to know what is the function of temperature in the softmax formula? and why should we use high temperatures to see a softer norm in probability distribution? The ...
user avatar
  • 203
0 votes
0 answers
9 views

How to obtain low validation loss and loss in CNN Keras as low as 0.1? [duplicate]

I have a dataset of 1400 images, training=1050, testing=350 and 7 classes, any idea on how I can have a low loss? I have been adding more layers randomly to get the best result as I can. loss=0.3032, ...
user avatar
0 votes
1 answer
113 views

Word2vec/SkipGram: Why softmax?

In Word2Vec (SkipGram version), there is a softmax layer at the end of the neural net. As this is expansive to calculate, some approximations are used instead, such as negative sampling. But if in the ...
user avatar
  • 3
0 votes
0 answers
16 views

Predicting in Neural Network with Softmax or Logistic activation output [duplicate]

So I'm wanting to make a neural network from scratch for predicting classifications and labels. The activation layer for the output in these networks would be softmax for classification and logistic ...
user avatar
  • 111
1 vote
0 answers
74 views

Matrix dimensions of Derivative of Softmax on Vectorized version

I am trying to get the matrix dimensions right for computing derivative of a two layers network where the last layer is softmax function. For simplicity I am only interested to get derivatives of W2 w....
user avatar
  • 133
0 votes
2 answers
958 views

Sigmoid vs Softmax Accuracy Difference

I have trained a neural network on DNA sequences data and my training set has exactly the same number of data in both classes. When I select a softmax function at the end, my accuracy remains at 47% ...
user avatar
  • 1
1 vote
1 answer
123 views

How does Word2Vec CBOW softmax work with multiple context words?

I'm referring to following paper from Xin Rong - "word2vec Parameter Learning Explained", to be precise the equation (4): $$ p(w_j|w_I) = \frac{\exp(\mathbf{v’}^{T}_{w_{j}}\mathbf{v}_{w_{I}})...
user avatar
  • 113
2 votes
1 answer
382 views

Metrics for multiclass classification model accuracy

Usually the last layer in multiclass classification models is a softmax, which is essentially a vector with elements the confidences for each class. The standard top-1 accuracy takes account only if ...
user avatar
  • 133
2 votes
1 answer
194 views

Large difference in accuracy for sigmoid vs softmax

I am experimenting on a neural network model I found on Kaggle for Titanic dataset where the problem statement is to determine whether a person has survived or not. The input I am providing is of this ...
user avatar
2 votes
1 answer
374 views

Number of features in multiclass Logistic Regression with categorical predictor

Assume that I want to predict a response with 3 classes. I have two features $X_1$ and $X_2$ where $X_1$ is continuous and $X_2$ is categorical with 5 categories. What would be the number of ...
user avatar
2 votes
0 answers
32 views

How to compute/estimate the probability of the mean value of a number of x results of a classification NN?

I use a neural network to classify the sentiment of some news articles per day, regarding a specific topic. Possible results are $[1,2,3,4,5]$ (1=very negative, ..., 5 = very positive). Using one ...
user avatar
  • 21
3 votes
1 answer
135 views

Deriving the gradients for Softmax logistic regression classifier

In the softmax logistic regression classifier, we have that $$\textbf{a} = W\textbf{x} + b\\[1ex] \textbf{z} = \text{softmax}(\textbf{a})\\[1ex] L(\textbf{z},\textbf{y}) = -\sum_k \log(z_k)y_k$$ In ...
user avatar
  • 876
0 votes
1 answer
89 views

Reference request of softmax function [closed]

What paper should I cite to reference softmax? Thanks in advance.
user avatar
2 votes
0 answers
55 views

Justification for softmax and sigmoid functions in classification models

In multi-class classification problems, we often build a model that first computes some score $s_i=f(X_i;\theta)$ for each class of instance $i$, where $\theta$ are parameters of the model. Then, we ...
user avatar
1 vote
0 answers
262 views

Understanding loss function gradient in asynchronous advantage actor-critic (A3C) algorithm

In the A3C algorithm from the original paper: the gradient with respect to log policy involves the term $$\log \pi(a_i|s_i;\theta')$$ where $s_i$ is the state of the environment at time step $i$, and ...
user avatar
2 votes
0 answers
226 views

Overparameterization with softmax with neural networks

I have encountered some applications of the softmax (multinomial logistic regression) in neural network applications where the sum-to-one constraint is ignored (e.g. see this link or this link). That ...
user avatar
1 vote
0 answers
20 views

Is $K$ independent binary logistic regression the same as a $K$-class softmax regression?

One of the "correct" regression models to multi-nominal classification problems is softmax regression, with the final prediction being the category with the highest predicted probability. ...
user avatar
  • 1,385
1 vote
0 answers
19 views

Can softmax probabilities be used for mixture classification?

I am trying to convince my peer about this model training and testing paradigm since it does not make sense. Let's say that you have two classes of signals in your training set, Class A and B. You can ...
user avatar
0 votes
0 answers
21 views

Regress 4 values based on an input matrix with shape of (16,5)

Problem: I need to regress 4 values (which quantify how much a user likes a specific topic). I am performing simulations, so I know the ground truth (the real 4 values of the preferences) Input Data ...
user avatar
1 vote
2 answers
260 views

Would the softmax classifier ever yield equal probabilities for more than 1 class?

The question might be quite straightforward but I can't seem to be find any relevant resources from Google. All the sources I found are focused on explaining difference between softmax and sigmoid ...
user avatar
2 votes
1 answer
84 views

Are there any multi-class classification techniques that do not involve the softmax?

It seems that the softmax is basically a key component of multi-class classification. Does there exist any classifiers that do not rely on the softmax?
user avatar

1
2 3 4 5