# Questions tagged [softmax]

Normalizing exponential function which transforms a numeric vector such that all its entries become between 0 and 1 and together sum to 1. It is often used as the final layer of a neural network performing a classification task.

202 questions
Filter by
Sorted by
Tagged with
12 views

### Why Gumbel softmax and not other types of softmax?

Ignore the method name. Is there a reason why in the Gumbel softmax trick we sample from Gumbel distribution? Since we are doing something similar to a reparameterization trick, can't we just sample ...
33 views

### using logsumexp in softmax

I saw this equation in somebody's code which is an alternative approach to implementing the softmax in order to avoid underflow by division by large numbers. softmax = e^(matrix - logaddexp(matrix)) = ...
10 views

### Do we use softmax even in binary classification where there is only one neuron in the output layer? [duplicate]

I think it's recommended to use one neuron in the output layer of a network if you are doing binary classification, because one neuron by itself can represent two states (0 or 1). But then softmax ...
21 views

### How can I mathematically explain the convolutional neural network?

How can I give notations to the whole CNN network? Like, if I feed input 'x' to feature extractor, we will get the extracted feature embedding vector at the end that will be passed to the linear ...
1 vote
49 views

### Why is there a meaningful relationship between probabilities of non-ground-truth classes, in the context of knowledge distillation?

In the original knowledge distillation paper Distilling the Knowledge in a Neural Network, Geoffrey Hinton, Oriol Vinyals and Jeff Dean state the following: "much of the information about the ...
1 vote
18 views

### Hybrid Bayesian Network with Continuous Parents and Ordinal Child Node (Softmax Function)

Upon reading about hybrid BNs with discrete child nodes and multiple continuous parents, I came across the possibility of using the softmax (multinomial logit) function (below) in order to query ...
1 vote
114 views

### calculating the top k accuracy using logits vs softmax probabilites

I am working on calculating the top k accuracy of a model my model output logits (I am working on pythorch) so in order to calculate the top k accuracy using sklearn i was wondering what would be the ...
1 vote
229 views

### Formal steps for gradient boosting with softmax and cross entropy loss function

Consider some data $\{(x_i,y_i)\}^n_{i=1}$ and a differentiable loss function $\mathcal{L}(y,F(x))$ and a multiclass classification problem which should be solved by a gradient boosting algorithm. ...
1 vote
39 views

### Weird Term of Log likelihood

Recall the setup of logistic regression: We assume that the posterior probability is of the form $p(Y=1|x) = \frac{1}{1+e^{\beta^Tx}}$ This assumes that Y|X is a Bernoulli Random variable. We now turn ...
1 vote
67 views

### Binary classification neural network - equivalent implementations with sigmoid and softmax

in order to solidify my understanding, I am doing some simple calculations with pen and paper for some very simple NN for binary classification (input vector with two entries, 1 hidden layer and just ...
1 vote
19 views

### Softmax computation in Transformers

In word embedding model word2vec, computation of softmax is expensive process and hence we use many alternative as provided here. Prominently, ...
42 views

### Sigmoid equivalent to Softmax exercise 2 [duplicate]

This question is not the same as this one I asked previously. In the previous question I asked to prove that the sigmoid and softmax are equivalent. I found a solution here, but I think it's not ...
47 views

### Sigmoid equivalent to Softmax exercise

I am currently studying the Sutton and Barto Intro To RL Book, and I'm trying to do exercise 2.9 (at the bottom of the following picture): So the exercise wants me to show that the softmax is ...
83 views

### Why we use hard targets (generated using the Softmax) but not soft targets or logits

I was reading about knowledge distillation (in student-teacher networks, here) and it is stated that: Advantages of Soft Targets: Soft targets contain valuable information on the rich similarity ...
69 views

### Knowledge Distillation for sigmoid function

I am studying the usage of knowledge distilling and all the contents found on youtube and papers suggests the use of softmax layer as the last layer because of the temperature value and the ...
1 vote
21 views

### Model for percent resource allocation between groups

I have a panel data set indicating each entities percent resource allocation among a number of different options. I have been using a logistic regression model to predict the share of allocation for ...
48 views

### exp(log_softmax) vs softmax as neural network activation

I have read about log_softmax being more numerical stable than softmax, since it circumvents the division. I need to use softmax, probabilities between 0 and 1, for my neural network loss function. So ...
1 vote
60 views

382 views

### Metrics for multiclass classification model accuracy

Usually the last layer in multiclass classification models is a softmax, which is essentially a vector with elements the confidences for each class. The standard top-1 accuracy takes account only if ...
194 views

### Large difference in accuracy for sigmoid vs softmax

I am experimenting on a neural network model I found on Kaggle for Titanic dataset where the problem statement is to determine whether a person has survived or not. The input I am providing is of this ...
374 views

### Number of features in multiclass Logistic Regression with categorical predictor

Assume that I want to predict a response with 3 classes. I have two features $X_1$ and $X_2$ where $X_1$ is continuous and $X_2$ is categorical with 5 categories. What would be the number of ...
32 views

### How to compute/estimate the probability of the mean value of a number of x results of a classification NN?

I use a neural network to classify the sentiment of some news articles per day, regarding a specific topic. Possible results are $[1,2,3,4,5]$ (1=very negative, ..., 5 = very positive). Using one ...
135 views

### Deriving the gradients for Softmax logistic regression classifier

In the softmax logistic regression classifier, we have that $$\textbf{a} = W\textbf{x} + b\\[1ex] \textbf{z} = \text{softmax}(\textbf{a})\\[1ex] L(\textbf{z},\textbf{y}) = -\sum_k \log(z_k)y_k$$ In ...
89 views

### Reference request of softmax function [closed]

What paper should I cite to reference softmax? Thanks in advance.
55 views

### Justification for softmax and sigmoid functions in classification models

In multi-class classification problems, we often build a model that first computes some score $s_i=f(X_i;\theta)$ for each class of instance $i$, where $\theta$ are parameters of the model. Then, we ...
1 vote
262 views

In the A3C algorithm from the original paper: the gradient with respect to log policy involves the term $$\log \pi(a_i|s_i;\theta')$$ where $s_i$ is the state of the environment at time step $i$, and ...
226 views

### Overparameterization with softmax with neural networks

I have encountered some applications of the softmax (multinomial logistic regression) in neural network applications where the sum-to-one constraint is ignored (e.g. see this link or this link). That ...
1 vote
20 views

### Is $K$ independent binary logistic regression the same as a $K$-class softmax regression?

One of the "correct" regression models to multi-nominal classification problems is softmax regression, with the final prediction being the category with the highest predicted probability. ...
1 vote
19 views

### Can softmax probabilities be used for mixture classification?

I am trying to convince my peer about this model training and testing paradigm since it does not make sense. Let's say that you have two classes of signals in your training set, Class A and B. You can ...
21 views

### Regress 4 values based on an input matrix with shape of (16,5)

Problem: I need to regress 4 values (which quantify how much a user likes a specific topic). I am performing simulations, so I know the ground truth (the real 4 values of the preferences) Input Data ...
1 vote