Questions tagged [softmax]

Normalizing exponential function which transforms a numeric vector such that all its entries become between 0 and 1 and together sum to 1. It is often used as the final layer of a neural network performing a classification task.

Filter by
Sorted by
Tagged with
0
votes
0answers
33 views

Derivaitve of Softmax [closed]

Hi I am trying to figure out the derivative of Softmax and find myself confused. I have been referencing this example Backpropagation with Softmax / Cross Entropy and have been following Vivek ...
2
votes
0answers
16 views

Is there something like softmax but for top k values?

I have a dataset with binary labels of which exactly k outputs are 1, on which I want to train a neural network. If k=1, softmax can do the job of representing the output distribution. I am interested ...
0
votes
2answers
23 views

Is softmax an activation function? [closed]

Is softmax an activation function? Because it is usually used in the output layer, why?
0
votes
1answer
22 views

Softmax function makes my machine to train much slower

I have two machines: CNN without softmax function and CNN with softmax function. But softmax function makes my machine to learn much slower and less accurate. Does anyone know why this happens? Here's ...
1
vote
0answers
23 views

Softmax + CE vs Sigmoid + BCE for batched training with negative sampling, for training similarity properties

This is a follow up to this question Machine Learning: Should I use a categorical cross entropy or binary cross entropy loss for binary predictions? I am training cos similarity properties for ...
1
vote
0answers
33 views

Why is softmax considered counter-intuitive for multi-label classification?

In the FB paper on Instagram multi-label classification (Exploring the Limits of Weakly Supervised Pretraining), the authors characterize as "counter-intuitive" their finding that softmax + ...
0
votes
1answer
32 views

Confusion with softmax

I was wondering if someone could explain why, if I do softmax on [683, 861, 981, 834] I get ...
0
votes
1answer
34 views

Not understanding backpropagation correctly

So I'm trying to build a simple NN where the layers are as follows: linear layer-> ReLu -> ...
0
votes
0answers
18 views

Softmax Linear Regression/ Multinomial Logistic Regression with shared coefficients and different inputs

I am trying to build a Softmax Regression model for 3 classes, where, unlike what is usually done, the coefficients between different options are shared and what varies are the input variables. ...
0
votes
1answer
32 views

Why aren't the outputs of the softmax and logistic function the same in this example? [duplicate]

I am trying to understand the equivalence between the logistic function and the softmax function, when k = 2. My understanding is that the following code should output the same values. Where is my ...
0
votes
0answers
9 views

How to adapt softmax model to tied outcomes

Consider the Harville model for modeling ordered outcomes: you observe a contest between $n$ participants, and model for each of them a $\mu_i$ such that the probability that the $i$th participant ...
1
vote
1answer
79 views

Softmax in last layer - error rises but when using sigmoid error decreases [closed]

I wrote a neural network from scratch in Python. It has 1 hidden layer which uses tanh activation function. I train it on Iris and MNIST datasets. When I use Sigmoid in the last layer results are very ...
1
vote
1answer
64 views

How a softmax function can be used in multiclass classification?

I'm trying to understand how a softmax function help classify in multi-class classification. In Andrew Ng video, he shows how a simple 1 layer neural network, with 2 inputs ($x_1, x_2$) and 3 outputs ...
20
votes
2answers
2k views

Why is softmax function used to calculate probabilities although we can divide each value by the sum of the vector?

Applying the softmax function on a vector will produce "probabilities" and values between $0$ and $1$. But we can also divide each value by the sum of the vector and that will produce ...
0
votes
1answer
35 views

Neural language model: Derivation of MLE

Recently, I studied NNLM and I saw the derivation of softmax using MLE: \begin{align} & \frac{\partial\log P(w_t\mid h)}{\partial\theta} \\[8pt] = {} & \frac{\partial \log \exp(s_\theta(w_t,...
0
votes
0answers
123 views

Can we minimize counting cost function for perceptron algorithm?

In perceptron algorithm (the following analysis might apply to other classification algorithms), a smooth approximation of perceptron cost function $$\sum_i^n{\max(0, -y_i\mathbf{w}^T\mathbf{x}_i)}$$ ...
0
votes
1answer
131 views

Softmax with Cross Entropy optimization vs Backpropagation

I am following a tutorial from Analytics Vidhya on creating a neural network to recognize handwritten digits (the classic example). The code from the tutorial states "First we need to define the ...
0
votes
1answer
162 views

Softmax where the max probability is less than one

I have a neural network with a softmax at the end. Something like this: ...
0
votes
1answer
493 views

How to implement thresholded softmax in Keras?

I would like to implement a threshold after the final softmax layer in a Keras-built classification problem so that class assignments with probability below some threshold alpha are disregarded (i.e. ...
0
votes
3answers
63 views

Two logistic regression or one Softmax regression

The following question is from Geron's Hans-On Machine Learning book. Suppose you want to classify pictures as outdoor/indoor and daytime/nighttime. Should you use two Logistic regression or one ...
0
votes
0answers
35 views

NLP - how do you randomly draw negative samples?

From my understanding, negative sampling randomly samples K negative samples from a noise distribution, P(w). The noise ...
2
votes
1answer
140 views

Having trouble figuring out how loss was calculated for SQuAD task in BERT paper

The BERT Paper https://arxiv.org/pdf/1810.04805.pdf Section 4.2 covers the SQuAD training. So from my understanding, there are two extra parameters trained, they are two vectors with the same ...
0
votes
0answers
37 views

Neural Networks output functions [duplicate]

Studying about neural networks as a newbie, I learned (I think) about the different types of output functions, but I have been confused when to use each. (softmax, cross-entropy, sigmoid, none) For ...
1
vote
0answers
87 views

Reinforcement Learning partial derivative of loss function w.r.t. input of softmax

In the paper "Self-critical sequence training for image captioning" (link) on page 3 they define the loss function (of the parameters $\theta$) of an image captioning system as the negative expected ...
0
votes
0answers
26 views

Difference between sigmoid and softmax [duplicate]

I know, that usually we use sigmoid in the hidden layer, and softmax as an output but what is the difference between neural network with one output and sigmoid activation layer and network with two ...
1
vote
2answers
241 views

Softmax layer derivative by hand

I would like to compute the gradient of the loss function with respect to the input to a sigmoid layer. This is a question in some online course I found (see 1:09:22 in https://www.youtube.com/watch?v=...
1
vote
1answer
310 views

Problem with Softmax decision boundary

While reading this paper: sphere face on page 2, it explains that original softmax boundary is given by: $$(W_1 −W_2)x+b_1 −b_2 = 0$$ While trying to obtain the boundary on a toy generated 2D dataset ...
4
votes
1answer
730 views

Is a neural network consisting of a single softmax classification layer only a linear classifier?

Since the softmax function is a generalization of the logistic function it is continuous and non-linear. So the output of the softmax layer is: softmax( weight_matrix * input_activation) ...
2
votes
2answers
2k views

Softmax weights initialization

I am a new to deep learning and neural networks, and I need to know if there is a good weights initialization method to use if the activation function is Softmax like Tanh, ReLU and Sigmoid. Related ...
1
vote
1answer
52 views

Meaning of different softmax notations in papers

I was wondering if the different notations of the softmax input mean different things especially about the size of the output. For example, in the paper Pointer Networks, it sometimes state the input ...
0
votes
1answer
200 views

How to get meaningful results from Softmax activation in Deep Unsupervised Clustering Network

I found this interesting paper regarding deep unsupervised clustering and am looking to mimic some of the things done, however there is one thing that is not clear to me. In the paper, they use a ...
0
votes
1answer
705 views

Candidate Sampling for Softmax - Tensorflow; Sampling Probability

I am trying to understand the mathematics behind the sampled softmax in Tensorflow. They have the following document, trying to explain how the sampling process works: https://www.tensorflow.org/...
1
vote
1answer
132 views

Proof for the efficiency of Softmax in multi-classification

I already search for this question but I can't find any convincing explanation so I want to ask it here. my problem is with softmax activation function and cross-entropy.why they can produce a better ...
0
votes
2answers
108 views

Softmax derivation - case i==j

I am reading this article: https://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative about softmax derivation w.r.t. to the input. Please confirm my understanding of the case when j==...
0
votes
2answers
48 views

Neural Network Layer for Binary Outputs

I'm currently using an LSTM network to make a yes or no decision in a robot. The network has a single output with the values in training data being 0 for one decision and 1 for another. The problem ...
1
vote
1answer
519 views

machine learning - Derivative of log-likelihood function in softmax regression

I'm trying to find the derivative of the log-likelihood function in softmax regression. I have (with $\Theta$ being the parameters, and $x^{(i)}$ being the $i$th training example, and $s_j$ ...
0
votes
0answers
184 views

change hinge loss error function with cross-entropy

I'm trying to implement cross-entropy as an error function in RBF neural networks instead of hinge loss error function. I need to find cross-entropy error for each output neuron, like hinge loss error ...
0
votes
1answer
187 views

In multi-class logistic regression, does SGD one training example update all the weights?

In multi-class logistic regression, lets say we use softmax and cross entropy. Does SGD one training example update all the weights or only a portion of the weights which are associated to the label ? ...
8
votes
1answer
619 views

Definition of softmax function

This question follows up on stats.stackexchange.com/q/233658 The logistic regression model for classes {0, 1} is $$ \mathbb{P} (y = 1 \;|\; x) = \frac{\exp(w^T x)}{1 + \exp(w^T x)} \\ \mathbb{P} (y =...
1
vote
0answers
87 views

How to interpret Gelman's multivariate Gaussian prior for multinomial distribution?

Andrew Gelman suggested the use of a multivariate Normal distribution as prior for hierarchical models that have a multinomial distribution at the lowest level (http://andrewgelman.com/2009/04/29/...
1
vote
1answer
364 views

Does hierarchical softmax of skip gram and CBOW only update output vectors on the path from the root to the actual output word?

After reading word2vec Parameter Learning Explained by Xin Rong, I understand that in the hierarchical softmax model, there is no output vector representation for words, instead, each of the $V-1$ ...
0
votes
1answer
192 views

Optimal neural net weights when using cross entropy loss

I'm trying to understand how cross-entropy works for finding the optimal weights in neural networks. According to Eli Bendersky's website and neural networks and deeplearning tutorial, we can find the ...
2
votes
1answer
1k views

Difference between mathematical and Tensorflow implementation of Softmax Crossentropy with logit

Softmax cross entropy with logits is define as follows: $a_i = \frac{e^{z_i}}{\sum_{\forall j} e^{z_j}}$ $l={\sum_{\forall i}}y_ilog(a_i)$ Where $l$ is the actual loss. But when you look deep into ...
2
votes
0answers
28 views

Cost function to output reliable error probabilities?

I'm using softmax regression but my goal is not to increase the 'accuracy' but instead to create probabilities which are accurate. Using cross entropy or the means squared error (which I'm aware I'm ...
10
votes
2answers
596 views

How deep is the connection between the softmax function in ML and the Boltzmann distribution in thermodynamics?

The softmax function, commonly used in neural networks to convert real numbers into probabilities, is the same function as the Boltzmann distribution, the probability distribution over energies for en ...
1
vote
0answers
237 views

How do I implement softmax forward propagation and backpropagation to replace sigmoid in a neural network?

I'm currently using 3Blue1Brown's tutorial series on neural networks and lack extensive calculus knowledge/experience. I'm using the following equations to calculate the gradients for weights and ...
0
votes
0answers
73 views

question about backpropagation with a softmax

When we have a neural network with a softmax layer, I'm a little confused how the weights get updated which are NOT associated with the correct answer. The issue is that the cost function will ...
1
vote
0answers
56 views

Neural Nets: Benefits of Mixing Activation Functions? [duplicate]

I'm currently studying and would like to know if there are any actual benefits of using a mixture of activation functions for each layer. For example, if I am trying to predict the probabilities of ...
3
votes
1answer
1k views

How does the subtraction of the logit maximum improve learning?

My question comes from another question answered on Stackoverflow; the Keras implemantion of softmax activation function is customized to subtract the maximum value ...
1
vote
1answer
250 views

Softmax dealing with non present classes [closed]

Is it okey to assume that, if I have trained my neural network for image classification using k classes using a Softmax output function, whenever I feed an image that does not belong to any of the k ...