Questions tagged [softmax]

Normalizing exponential function which transforms a numeric vector such that all its entries become between 0 and 1 and together sum to 1. It is often used as the final layer of a neural network performing a classification task.

Filter by
Sorted by
Tagged with
0
votes
0answers
22 views

Backpropagation in an MLP with 1-hidden layer - What am I missing?

I want to do the math behind backpropagation of an MLP with 1 hidden layer and softmax output layer for a simple classification problem, and I can't for the life of me figure out what do I do wrong. ...
6
votes
3answers
537 views

log(1 - softmax(X))? [closed]

Let $\vec X$ be a vector. The $\vec V = \mathrm{logsoftmax}(\vec{X})$ function is defined as: $$v_i = \ln\left(\frac{e^{x_i}}{\sum_i e^{x_i}}\right)$$ This is provided in machine learning numerical ...
0
votes
0answers
4 views

How to customize activation and loss functions for multilabel classification problem?

I'm trying to develop a model using keras able to perform a particular multilabel problem: My target vectors are five components vector in which there are elements between 0 and 4 and Nan. ...
2
votes
1answer
34 views

Does log-likelihood cost function in a multinomial classification consider only the output at the neuron that should be active for that class?

Consider a neural network with an output layer of softmax neurons and a log likelihood cost function. For easiness consider one wants to train a MNIST classifier. The output layer will have 9 neurons ...
0
votes
1answer
48 views

How is the class label applied in the softmax function?

Im reading this paper: Uncertainty in Deep Learning and in it (page4), the softmax loss is defined as \begin{align*} E(X,Y) = -\frac{1}{N} \sum^N_{n=1} \log(\hat{p}_{n,c_n}), \end{align*} where $c_n ...
1
vote
1answer
14 views

Is multinomial logistic regression really the same as softmax regression

Multinomial logistic regression (MLR) is an extension of logistic regression for more than $2$ classes. The extension is made up by keeping linear boundaries between classes and using the class $K$ as ...
1
vote
0answers
19 views

binary cross entropy vs multi cross entropy

i am new to neural networks I know that multi class entropy is same as binary class entropy when the categories are only (0,1), but can some one explain it mathematically with an example that ...
1
vote
0answers
32 views

Derivative of softmax function as a matrix

I have a generalised n-layer neural network. Currently, I am using it to perform digit classification (on the MNIST dataset), using a softmax + cross-entropy loss setup with simple stochastic gradient ...
0
votes
0answers
17 views

Softmax backpropagation

I know there are similar questions out there, but none of the answers really helped me. I'm working on an own neural network implementation and I want to implement the softmax activation function. I'm ...
0
votes
0answers
9 views

Number of parameters in sigmoid vs. softmax cross entropy

Assume I have a data point $\mathbf{x} = [x_1, x_2, \ldots, x_D]^\top$ which I want to classify into one of two mutually exclusive categories $\mathcal{C}_0$ and $\mathcal{C}_1$. I can create a simple ...
3
votes
1answer
128 views

Predicting proportions with Machine Learning

I am working on a machine learning problem where I have to predict a set of $N$ numbers (proportions) for each data point, all of them summing to one. One toy example to illustrate my problem would be ...
0
votes
0answers
11 views

Estimating coefficients in softmax model formulations

Suppose we model $P[X=l]$ for some data using a softmax formulation, say : $$ P[X=l]=\dfrac{e^{\gamma_l}}{\sum_{i=1}^L \gamma_i} $$ Now I have read some papers where they constraint the ...
0
votes
0answers
17 views

How do you combine the losses across batches for a softmax cross-entropy loss? [duplicate]

I am trying to implement a softmax layer for practice. I can calculate the loss for each individual input, but I am confused on how to combine the losses together into one single loss for the entire ...
0
votes
1answer
347 views

Softmax derivative implementation [closed]

I know there are already multiple similar questions out there, but still don't really understand the derivative of the softmax function. That's how I implemented the softmax function in java: ...
0
votes
1answer
34 views

Is there a name for the composition of the cross-entropy and softmax functions?

This is a simple question but I'll give some background. The softmax function $S: \mathbb R^K \to \mathbb R^K$ is defined by $$ S(u) = \begin{bmatrix} \frac{e^{u_1}}{\sum_j e^{u_j}} \\ \frac{e^{u_2}}{\...
1
vote
1answer
34 views

Does there exist ReLu regression?

If softmax regression is multinomial logistic regression, is there anything called ReLu regression?
1
vote
2answers
22 views

Softmax: can't wrap my head around these values

I've got three simple classes each with some count values and I want to calculate the probability distribution. Column $B$ is the count and column $C$ is $exp(count)$. The last column then devides ...
0
votes
2answers
66 views

With Sigmoid activation and Softmax normalization with cross entropy, are we fitting distributions?

Let's consider I have a multi layer neural network that is doing multi class classification. So each input sample belongs to one on N classes. Now, lets say the last layer has Sigmoid activation ...
0
votes
0answers
48 views

Soft Cross Entropy to increase precision?

Okay, so lets say that I have a network that is predicting a distribution of the depth of single laser point from some other data. We could regress on the ground truth directly, or in my case, map the ...
0
votes
0answers
37 views

Back-propagation through cross entropy or logistic loss function

I have neural network which ends with softmax function and I want to minimize cross-entropy cost function which takes output of this network and one-hot labels as arguments. To calculate partial ...
0
votes
0answers
28 views

Why does estimating $k - 1$ probabilities in multinomial regression is way better than $k$? [duplicate]

I am probably going to ask dumb question, but I think I can't understand the crucial advantage of the multinomial regression compared to just logistic regression. Or is it just generalization of the ...
0
votes
0answers
28 views

Hierarchical SoftMax for Skip Gram?

While I am reading the following article on the Internet, I am kind of feeling that I am not getting the full understanding of the picture. https://d2l.ai/chapter_natural-language-processing/approx-...
0
votes
1answer
70 views
0
votes
1answer
25 views

Probability that a population mean exceeds a threshold

In Googling this question, I see that there are a variety of similar tests but I couldn't find anything given the exact way I'm approaching this problem. This might be something obvious but I'm not ...
3
votes
1answer
114 views

Difference between ONNX and Caffe2 softmax vs. PyTorch and Tensorflow softmax

We've been looking at softmax results produced by different frameworks (TF, PyTorch, Caffe2, Glow and ONNX runtime) and were surprised to find that the results differ between the frameworks. From the ...
3
votes
1answer
99 views

Is there something like softmax but for top k values?

I have a dataset with binary labels of which exactly k outputs are 1, on which I want to train a neural network. If k=1, softmax can do the job of representing the output distribution. I am interested ...
0
votes
2answers
28 views

Is softmax an activation function? [closed]

Is softmax an activation function? Because it is usually used in the output layer, why?
0
votes
1answer
67 views

Softmax function makes my machine to train much slower

I have two machines: CNN without softmax function and CNN with softmax function. But softmax function makes my machine to learn much slower and less accurate. Does anyone know why this happens? Here's ...
1
vote
0answers
94 views

Softmax + CE vs Sigmoid + BCE for batched training with negative sampling, for training similarity properties

This is a follow up to this question Machine Learning: Should I use a categorical cross entropy or binary cross entropy loss for binary predictions? I am training cos similarity properties for ...
1
vote
0answers
94 views

Why is softmax considered counter-intuitive for multi-label classification?

In the FB paper on Instagram multi-label classification (Exploring the Limits of Weakly Supervised Pretraining), the authors characterize as "counter-intuitive" their finding that softmax + ...
0
votes
1answer
54 views

Confusion with softmax

I was wondering if someone could explain why, if I do softmax on [683, 861, 981, 834] I get ...
0
votes
1answer
40 views

Not understanding backpropagation correctly

So I'm trying to build a simple NN where the layers are as follows: linear layer-> ReLu -> ...
0
votes
0answers
23 views

Softmax Linear Regression/ Multinomial Logistic Regression with shared coefficients and different inputs

I am trying to build a Softmax Regression model for 3 classes, where, unlike what is usually done, the coefficients between different options are shared and what varies are the input variables. ...
0
votes
1answer
34 views

Why aren't the outputs of the softmax and logistic function the same in this example? [duplicate]

I am trying to understand the equivalence between the logistic function and the softmax function, when k = 2. My understanding is that the following code should output the same values. Where is my ...
0
votes
0answers
10 views

How to adapt softmax model to tied outcomes

Consider the Harville model for modeling ordered outcomes: you observe a contest between $n$ participants, and model for each of them a $\mu_i$ such that the probability that the $i$th participant ...
1
vote
1answer
137 views

Softmax in last layer - error rises but when using sigmoid error decreases [closed]

I wrote a neural network from scratch in Python. It has 1 hidden layer which uses tanh activation function. I train it on Iris and MNIST datasets. When I use Sigmoid in the last layer results are very ...
1
vote
1answer
137 views

How a softmax function can be used in multiclass classification?

I'm trying to understand how a softmax function help classify in multi-class classification. In Andrew Ng video, he shows how a simple 1 layer neural network, with 2 inputs ($x_1, x_2$) and 3 outputs ...
20
votes
2answers
3k views

Why is softmax function used to calculate probabilities although we can divide each value by the sum of the vector?

Applying the softmax function on a vector will produce "probabilities" and values between $0$ and $1$. But we can also divide each value by the sum of the vector and that will produce ...
0
votes
1answer
40 views

Neural language model: Derivation of MLE

Recently, I studied NNLM and I saw the derivation of softmax using MLE: \begin{align} & \frac{\partial\log P(w_t\mid h)}{\partial\theta} \\[8pt] = {} & \frac{\partial \log \exp(s_\theta(w_t,...
0
votes
0answers
134 views

Can we minimize counting cost function for perceptron algorithm?

In perceptron algorithm (the following analysis might apply to other classification algorithms), a smooth approximation of perceptron cost function $$\sum_i^n{\max(0, -y_i\mathbf{w}^T\mathbf{x}_i)}$$ ...
1
vote
1answer
240 views

Softmax with Cross Entropy optimization vs Backpropagation

I am following a tutorial from Analytics Vidhya on creating a neural network to recognize handwritten digits (the classic example). The code from the tutorial states "First we need to define the ...
1
vote
1answer
277 views

Softmax where the max probability is less than one

I have a neural network with a softmax at the end. Something like this: ...
0
votes
1answer
706 views

How to implement thresholded softmax in Keras?

I would like to implement a threshold after the final softmax layer in a Keras-built classification problem so that class assignments with probability below some threshold alpha are disregarded (i.e. ...
0
votes
3answers
92 views

Two logistic regression or one Softmax regression

The following question is from Geron's Hans-On Machine Learning book. Suppose you want to classify pictures as outdoor/indoor and daytime/nighttime. Should you use two Logistic regression or one ...
0
votes
1answer
54 views

NLP - how do you randomly draw negative samples?

From my understanding, negative sampling randomly samples K negative samples from a noise distribution, P(w). The noise ...
2
votes
1answer
237 views

Having trouble figuring out how loss was calculated for SQuAD task in BERT paper

The BERT Paper https://arxiv.org/pdf/1810.04805.pdf Section 4.2 covers the SQuAD training. So from my understanding, there are two extra parameters trained, they are two vectors with the same ...
1
vote
0answers
115 views

Reinforcement Learning partial derivative of loss function w.r.t. input of softmax

In the paper "Self-critical sequence training for image captioning" (link) on page 3 they define the loss function (of the parameters $\theta$) of an image captioning system as the negative expected ...
1
vote
2answers
285 views

Softmax layer derivative by hand

I would like to compute the gradient of the loss function with respect to the input to a sigmoid layer. This is a question in some online course I found (see 1:09:22 in https://www.youtube.com/watch?v=...
1
vote
1answer
442 views

Problem with Softmax decision boundary

While reading this paper: sphere face on page 2, it explains that original softmax boundary is given by: $$(W_1 −W_2)x+b_1 −b_2 = 0$$ While trying to obtain the boundary on a toy generated 2D dataset ...
5
votes
1answer
962 views

Is a neural network consisting of a single softmax classification layer only a linear classifier?

Since the softmax function is a generalization of the logistic function it is continuous and non-linear. So the output of the softmax layer is: softmax( weight_matrix * input_activation) ...