Questions tagged [softmax]

Normalizing exponential function which transforms a numeric vector such that all its entries become between 0 and 1 and together sum to 1. It is often used as the final layer of a neural network performing a classification task.

Filter by
Sorted by
Tagged with
1
vote
1answer
21 views

How does Word2Vec CBOW softmax work with multiple context words?

I'm referring to following paper from Xin Rong - "word2vec Parameter Learning Explained", to be precise the equation (4): $$ p(w_j|w_I) = \frac{\exp(\mathbf{v’}^{T}_{w_{j}}\mathbf{v}_{w_{I}})...
2
votes
1answer
36 views

Metrics for multiclass classification model accuracy

Usually the last layer in multiclass classification models is a softmax, which is essentially a vector with elements the confidences for each class. The standard top-1 accuracy takes account only if ...
0
votes
0answers
9 views

In multiclass classification, why do we have K but not (K-1) output units for softmax layer?

In binary classification, if we can transform the softmax function (needs 2 outputs) to sigmoid function (needs 1 output): $$\begin{align*}\mathrm{Pr}(Y=0|X)&=\frac{e^{b_0\cdot X}}{e^{b_0 \cdot X}+...
2
votes
1answer
45 views

Large difference in accuracy for sigmoid vs softmax

I am experimenting on a neural network model I found on Kaggle for Titanic dataset where the problem statement is to determine whether a person has survived or not. The input I am providing is of this ...
1
vote
1answer
24 views

Number of features in multiclass Logistic Regression with categorical predictor

Assume that I want to predict a response with 3 classes. I have two features $X_1$ and $X_2$ where $X_1$ is continuous and $X_2$ is categorical with 5 categories. What would be the number of ...
0
votes
0answers
20 views

Softmax for classification (confusion about notation for log likelihood)

I am (once again) bamboozled by the notation used by my lecturer. I am attempting to determine the log likelihood using a softmax function for classification. The problem setup is: Feature space $V=\...
2
votes
0answers
26 views

How to compute/estimate the probability of the mean value of a number of x results of a classification NN?

I use a neural network to classify the sentiment of some news articles per day, regarding a specific topic. Possible results are $[1,2,3,4,5]$ (1=very negative, ..., 5 = very positive). Using one ...
2
votes
1answer
30 views

Deriving the gradients for Softmax logistic regression classifier

In the softmax logistic regression classifier, we have that $$\textbf{a} = W\textbf{x} + b\\[1ex] \textbf{z} = \text{softmax}(\textbf{a})\\[1ex] L(\textbf{z},\textbf{y}) = -\sum_k \log(z_k)y_k$$ In ...
0
votes
0answers
13 views

Reference request for softmax function probability

I need a reference to illustrate that the softmax function can be used outside the neural network or used with classification algorithms that are not Neura
0
votes
1answer
17 views

Reference request of softmax function

What paper should I cite to reference softmax? Thanks in advance.
1
vote
0answers
35 views

Justification for softmax and sigmoid functions in classification models

In multi-class classification problems, we often build a model that first computes some score $s_i=f(X_i;\theta)$ for each class of instance $i$, where $\theta$ are parameters of the model. Then, we ...
1
vote
0answers
41 views

Understanding loss function gradient in asynchronous advantage actor-critic (A3C) algorithm

In the A3C algorithm from the original paper: the gradient with respect to log policy involves the term $$\log \pi(a_i|s_i;\theta')$$ where $s_i$ is the state of the environment at time step $i$, and ...
0
votes
0answers
30 views

Computing the gradient of Categorical Cross Entropy Loss

The categorical cross entropy loss is expressed as: $$L(y,t) = -\sum_{k=1}^{K}t_k\ln{y_k}$$ where $t$ is a one-hot encoded vector. $y_k$ is the softmax function defined as: $$y_k = \frac{e^{z_k}}{\...
0
votes
0answers
15 views

Softmax as a measure of uncertainty, not certainty?

I am aware that the softmax output of a neural network is not a good confidence measure (see Gal 2016, page 13 and 14). The reasoning behind this is that they are too over confident when they actually ...
0
votes
0answers
50 views

Temperature in Softmax and simulated annealing in Metropolis-Hastings?

We can add a temperature to the Softmax to make the Softmax softer or harder by setting it higher or lower(refer to this answer). And in reinforcement learning, a high temperature will lead to the ...
0
votes
0answers
11 views

Use multiple softmax in transformers output layer and calculate loss

Can I use multiple softmax in the last output layer in transformers? If so, how can I calculate loss from that. I am working in pytorch. And I am asking because my data is a sequence of tuples where, ...
0
votes
0answers
57 views

Overparameterization with softmax with neural networks

I have encountered some applications of the softmax (multinomial logistic regression) in neural network applications where the sum-to-one constraint is ignored (e.g. see this link or this link). That ...
1
vote
0answers
17 views

Is $K$ independent binary logistic regression the same as a $K$-class softmax regression?

One of the "correct" regression models to multi-nominal classification problems is softmax regression, with the final prediction being the category with the highest predicted probability. ...
0
votes
0answers
48 views

Hierarchical softmax vs softmax in hyper-parameter search

I am training an NLP model using fasttext where fasttext allows you to use either hierarchical softmax or softmax. It is my understanding that hierarchichal softmax is orders of magnitude faster ...
0
votes
0answers
21 views

Understanding difference between gradient of sigmoid vs softmax in the context of back propagation

Consider the step by step backpropagation shown in this article. The neural network given there is: The outer layer employs sigmoid. It also employs mean squared error loss function. During back-...
1
vote
0answers
18 views

Can softmax probabilities be used for mixture classification?

I am trying to convince my peer about this model training and testing paradigm since it does not make sense. Let's say that you have two classes of signals in your training set, Class A and B. You can ...
0
votes
0answers
19 views

Regress 4 values based on an input matrix with shape of (16,5)

Problem: I need to regress 4 values (which quantify how much a user likes a specific topic). I am performing simulations, so I know the ground truth (the real 4 values of the preferences) Input Data ...
1
vote
1answer
21 views

Would the softmax classifier ever yield equal probabilities for more than 1 class?

The question might be quite straightforward but I can't seem to be find any relevant resources from Google. All the sources I found are focused on explaining difference between softmax and sigmoid ...
0
votes
0answers
14 views

Strange situation with softmax activation vs sigmoid activation in the last layer of a binary classification problem?

I am building a neural network as a binary classifier with one output neuron at the last output layer. I deliberately balancing out my train label so that the number label corresponding to the binary ...
2
votes
1answer
40 views

Are there any multi-class classification techniques that do not involve the softmax?

It seems that the softmax is basically a key component of multi-class classification. Does there exist any classifiers that do not rely on the softmax?
0
votes
1answer
39 views

Why does softmax cause my nn to not converge

I have a pytorch model made up of a several convolutional and groupnorm layers which eventually feed into fully connected and eventually a softmax. With the softmax, the model never converges and ...
1
vote
1answer
158 views

Using Softmax activation function for multi-class classification

In the last layer of  'CNNs'   it is common to use softmax activation functions for multi-class classification.I would like to know if is it necessary using a softmax activation function when ...
1
vote
0answers
40 views

Compositional Data Analysis: What is the connection between Soft-max Regression / Logistic Regression and Linear Regression in the Simplex Space?

As someone just learning Compositional Data Analysis, my understanding is the following: The sample space for Compositional Data Analysis is the Simplex Space. Useful transformations like ALR/ILR/CLR ...
1
vote
0answers
47 views

Convolution of Gaussian with Softmax

I am trying to integrate the function $f = \int \sigma(\boldsymbol{x})_i \mathcal{N}(x_i | \mu_i, \sigma_i^2) d x_i$ where $\sigma(\boldsymbol{x})_i$ is the softmax function over variables $\...
0
votes
2answers
227 views

Softmax vs the Dirichlet distribution

As far as I understand one can in principle model the distribution over a set of $k$ categories using e.g.: the Dirichlet distribution A softmax model. As far as I can tell, both use $k$ parameters ...
2
votes
0answers
18 views

Softmax classifier with class priors

Softmax classifier is a discriminative model that directly models $p(Y|X, w)$ where $Y$ is the label for input $X$. We can write it as follows: $$p(y_i=k|x_i,w_k)=\frac{\exp \lbrace w_k^T x_i \rbrace}{...
0
votes
0answers
28 views

Getting gradient for weights in a softmax classifier

So consider the following scenario: $X$ ($N\times D$) is the input matrix containing all the inputs, $W$ ($D \times C $) is the weight. So $$scores = X.dot(W) $$. Then we're using the softmax ...
0
votes
0answers
49 views

Auxiliary loss function for softmax confidence

I have a softmax as part of a neural net model. There are no labels for these softmax outputs, but I want to enforce an auxiliary loss function on the confidence of the softmax output, essentially ...
1
vote
1answer
897 views

How to set a threshold on softmax probabilities in a multi-class classification task?

I have a large image dataset that was classified by a ConvNet into different classes (objects). For each image the top-1 softmax probability is given, ranging between 0 and 1. It´s the output of a ...
4
votes
2answers
168 views

What is the relationship between Boltzmann / Gibbs sampling and the softmax function?

I'm looking at sampling functions in the context of reinforcement learning; specifically the explore/exploit problem. A method I've seen pretty often is to derive the action by assigning a score to ...
6
votes
3answers
685 views

log(1 - softmax(X))? [closed]

Let $\vec X$ be a vector. The $\vec V = \mathrm{logsoftmax}(\vec{X})$ function is defined as: $$v_i = \ln\left(\frac{e^{x_i}}{\sum_i e^{x_i}}\right)$$ This is provided in machine learning numerical ...
0
votes
0answers
56 views

How to customize activation and loss functions for multilabel classification problem?

I'm trying to develop a model using keras able to perform a particular multilabel problem: My target vectors are five components vector in which there are elements between 0 and 4 and Nan. ...
3
votes
1answer
64 views

Does log-likelihood cost function in a multinomial classification consider only the output at the neuron that should be active for that class?

Consider a neural network with an output layer of softmax neurons and a log likelihood cost function. For easiness consider one wants to train a MNIST classifier. The output layer will have 9 neurons ...
0
votes
1answer
59 views

How is the class label applied in the softmax function?

Im reading this paper: Uncertainty in Deep Learning and in it (page4), the softmax loss is defined as \begin{align*} E(X,Y) = -\frac{1}{N} \sum^N_{n=1} \log(\hat{p}_{n,c_n}), \end{align*} where $c_n ...
1
vote
1answer
124 views

Is multinomial logistic regression really the same as softmax regression

Multinomial logistic regression (MLR) is an extension of logistic regression for more than $2$ classes. The extension is made up by keeping linear boundaries between classes and using the class $K$ as ...
1
vote
0answers
42 views

binary cross entropy vs multi cross entropy

i am new to neural networks I know that multi class entropy is same as binary class entropy when the categories are only (0,1), but can some one explain it mathematically with an example that ...
1
vote
0answers
234 views

Derivative of softmax function as a matrix

I have a generalised n-layer neural network. Currently, I am using it to perform digit classification (on the MNIST dataset), using a softmax + cross-entropy loss setup with simple stochastic gradient ...
0
votes
0answers
84 views

Softmax backpropagation

I know there are similar questions out there, but none of the answers really helped me. I'm working on an own neural network implementation and I want to implement the softmax activation function. I'm ...
1
vote
1answer
36 views

Number of parameters in sigmoid vs. softmax cross entropy

Assume I have a data point $\mathbf{x} = [x_1, x_2, \ldots, x_D]^\top$ which I want to classify into one of two mutually exclusive categories $\mathcal{C}_0$ and $\mathcal{C}_1$. I can create a simple ...
4
votes
2answers
458 views

Predicting proportions with Machine Learning

I am working on a machine learning problem where I have to predict a set of $N$ numbers (proportions) for each data point, all of them summing to one. One toy example to illustrate my problem would be ...
0
votes
1answer
596 views

Softmax derivative implementation [closed]

I know there are already multiple similar questions out there, but still don't really understand the derivative of the softmax function. That's how I implemented the softmax function in java: ...
1
vote
1answer
133 views

Is there a name for the composition of the cross-entropy and softmax functions?

This is a simple question but I'll give some background. The softmax function $S: \mathbb R^K \to \mathbb R^K$ is defined by $$ S(u) = \begin{bmatrix} \frac{e^{u_1}}{\sum_j e^{u_j}} \\ \frac{e^{u_2}}{\...
1
vote
1answer
78 views

Does there exist ReLu regression?

If softmax regression is multinomial logistic regression, is there anything called ReLu regression?
1
vote
2answers
31 views

Softmax: can't wrap my head around these values

I've got three simple classes each with some count values and I want to calculate the probability distribution. Column $B$ is the count and column $C$ is $exp(count)$. The last column then devides ...
0
votes
2answers
506 views

With Sigmoid activation and Softmax normalization with cross entropy, are we fitting distributions?

Let's consider I have a multi layer neural network that is doing multi class classification. So each input sample belongs to one on N classes. Now, lets say the last layer has Sigmoid activation ...