Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [softmax]

Normalizing exponential function which transforms a numeric vector such that all its entries become between 0 and 1 and together sum to 1. It is often used as the final layer of a neural network performing a classification task.

1
vote
1answer
13 views

Problem with Softmax decision boundary

While reading this paper: sphere face on page 2, it explains that original softmax boundary is given by: $$(W_1 −W_2)x+b_1 −b_2 = 0$$ While trying to obtain the boundary on a toy generated 2D dataset ...
0
votes
0answers
28 views

Softmax regression vector features?

I'm working on a difficult classification problem where there are a very large number of known classes (~50k). I have only about 20k labelled points, but these only represent <1% of the possible ...
0
votes
0answers
36 views

confusion matrix for one hot vectors

Let $\mathbf{Y}$ be a matrix where each row is a one-hot encoding of true labels and $\hat{\mathbf{Y}}$ be a similar matrix for predicted labels that are generated by a softmax function. How is the ...
3
votes
1answer
299 views

Is a neural network consisting of a single softmax classification layer only a linear classifier?

Since the softmax function is a generalization of the logistic function it is continuous and non-linear. So the output of the softmax layer is: softmax( weight_matrix * input_activation) ...
0
votes
1answer
132 views

Softmax weights initialization

I am a new to deep learning and neural networks, and I need to know if there is a good weights initialization method to use if the activation function is Softmax like Tanh, ReLU and Sigmoid. Related ...
0
votes
0answers
74 views

Classification confidence at Multi-class classifier with Soft-max output

what I'm NOT asking about is : Confidence interval So the question I'm trying to find the answer of would be : how to measure the confidence of a multi-class classifier with a Softmax output (eg a ...
0
votes
0answers
54 views

Tensorflow tf.nn.sampled_softmax_loss: Should uniform_candidate_sampler be the default sampler instead of log_uniform_candidate_sampler?

I was reading the documentation for tf.nn.sampled_softmax_loss: https://www.tensorflow.org/api_docs/python/tf/nn/sampled_softmax_loss It says for choosing the negative samples, if we don't put ...
0
votes
0answers
10 views

Classication given several softmax probabilities

I am classifying images over time in categories such as office, bathroom, living room, etc. using CNNs (in this case VGG-16). The idea is to use all room categories' confidences (softmax probabilities)...
0
votes
0answers
45 views

Does Scikit Learn Logistic Regression use Sigmoid Function if We Apply multi_class=multinomial?

By looking at various documentations/blogs, my answer to this above question is No. By default, sklearn.linear_model.LogisticRegression using multi_class = ‘ovr’. I understand Sigmoid function for ...
0
votes
1answer
25 views

Meaning of different softmax notations in papers

I was wondering if the different notations of the softmax input mean different things especially about the size of the output. For example, in the paper Pointer Networks, it sometimes state the input ...
0
votes
1answer
60 views

How to get meaningful results from Softmax activation in Deep Unsupervised Clustering Network

I found this interesting paper regarding deep unsupervised clustering and am looking to mimic some of the things done, however there is one thing that is not clear to me. In the paper, they use a ...
0
votes
0answers
33 views

multi class logistic regression : $K-1$ regressors or $K$ regressors (softmax)?

I read that, in multiclass logistic regression, we have a pivot class $K$ and $K-1$ set of $\vec{w}$ weights, then, for the pivot class: \begin{eqnarray} P( C_K | \vec{x} ) &= 1- \sum_\limits{t=...
0
votes
0answers
22 views

What does this cross entropy notation $J(o, v_c, U) = CE(y, \hat y)$ mean?

From http://web.stanford.edu/class/cs224n/assignment1/index.html under "complementary written problems" problem $3(a)$: What does $J_{\text{Softmax---CE}}(o, v_c, U) = CE(y, \hat y)$ fully stand for? ...
0
votes
1answer
217 views

Candidate Sampling for Softmax - Tensorflow; Sampling Probability

I am trying to understand the mathematics behind the sampled softmax in Tensorflow. They have the following document, trying to explain how the sampling process works: https://www.tensorflow.org/...
1
vote
1answer
81 views

Proof for the efficiency of Softmax in multi-classification

I already search for this question but I can't find any convincing explanation so I want to ask it here. my problem is with softmax activation function and cross-entropy.why they can produce a better ...
0
votes
2answers
35 views

Softmax derivation - case i==j

I am reading this article: https://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative about softmax derivation w.r.t. to the input. Please confirm my understanding of the case when j==...
0
votes
2answers
43 views

Neural Network Layer for Binary Outputs

I'm currently using an LSTM network to make a yes or no decision in a robot. The network has a single output with the values in training data being 0 for one decision and 1 for another. The problem ...
1
vote
1answer
153 views

machine learning - Derivative of log-likelihood function in softmax regression

I'm trying to find the derivative of the log-likelihood function in softmax regression. I have (with $\Theta$ being the parameters, and $x^{(i)}$ being the $i$th training example, and $s_j$ ...
0
votes
0answers
88 views

change hinge loss error function with cross-entropy

I'm trying to implement cross-entropy as an error function in RBF neural networks instead of hinge loss error function. I need to find cross-entropy error for each output neuron, like hinge loss error ...
0
votes
1answer
79 views

In multi-class logistic regression, does SGD one training example update all the weights?

In multi-class logistic regression, lets say we use softmax and cross entropy. Does SGD one training example update all the weights or only a portion of the weights which are associated to the label ? ...
0
votes
0answers
15 views

Softmax regression learning curve intersecting

I've implemented softmax regression on python. I've separated the Iris data into training, validation and test data. When I plot the learning curve for training data and validation data I get this: ...
0
votes
0answers
43 views

Loss function for partial/splitted softmax outputs on binary ground truth

I need to find a loss function for the scenario where each output is a vector instead of a scalar. And each of these vectors in one-hot-encoded. So I would like to use something like softmax loss on ...
0
votes
0answers
129 views

Are OvA/OvR Logistic regression and Softmax regression are the same?

I hear a lot that the Softmax function is a generalization of the Logistic function. Does that mean Softmax Regression is just a case of OvA/OvR Logistic regression? Will 2 class Softmax Regression ...
4
votes
0answers
255 views

Definition of softmax function

This question follows up on stats.stackexchange.com/q/233658 The logistic regression model for classes {0, 1} is $$ \mathbb{P} (y = 1 \;|\; x) = \frac{\exp(w^T x)}{1 + \exp(w^T x)} \\ \mathbb{P} (y =...
1
vote
0answers
33 views

How to interpret Gelman's multivariate Gaussian prior for multinomial distribution?

Andrew Gelman suggested the use of a multivariate Normal distribution as prior for hierarchical models that have a multinomial distribution at the lowest level (http://andrewgelman.com/2009/04/29/...
1
vote
1answer
165 views

Does hierarchical softmax of skip gram and CBOW only update output vectors on the path from the root to the actual output word?

After reading word2vec Parameter Learning Explained by Xin Rong, I understand that in the hierarchical softmax model, there is no output vector representation for words, instead, each of the $V-1$ ...
0
votes
1answer
149 views

Optimal neural net weights when using cross entropy loss

I'm trying to understand how cross-entropy works for finding the optimal weights in neural networks. According to Eli Bendersky's website and neural networks and deeplearning tutorial, we can find the ...
1
vote
1answer
469 views

Difference between mathematical and Tensorflow implementation of Softmax Crossentropy with logit

Softmax cross entropy with logits is define as follows: $a_i = \frac{e^{z_i}}{\sum_{\forall j} e^{z_j}}$ $l={\sum_{\forall i}}y_ilog(a_i)$ Where $l$ is the actual loss. But when you look deep into ...
2
votes
0answers
23 views

Cost function to output reliable error probabilities?

I'm using softmax regression but my goal is not to increase the 'accuracy' but instead to create probabilities which are accurate. Using cross entropy or the means squared error (which I'm aware I'm ...
8
votes
1answer
182 views

How deep is the connection between the softmax function in ML and the Boltzmann distribution in thermodynamics?

The softmax function, commonly used in neural networks to convert real numbers into probabilities, is the same function as the Boltzmann distribution, the probability distribution over energies for en ...
1
vote
0answers
235 views

Tensorflow Sampled_Softmax_loss - correct usage [closed]

I am working on a Speaker recognition problem, I have very big number of classes so I need to use, tf.nn.sampled_softmax_loss to speed up training time. The problem is I am using Keras with Tensorflow ...
1
vote
0answers
154 views

How do I implement softmax forward propagation and backpropagation to replace sigmoid in a neural network?

I'm currently using 3Blue1Brown's tutorial series on neural networks and lack extensive calculus knowledge/experience. I'm using the following equations to calculate the gradients for weights and ...
0
votes
0answers
38 views

question about backpropagation with a softmax

When we have a neural network with a softmax layer, I'm a little confused how the weights get updated which are NOT associated with the correct answer. The issue is that the cost function will ...
1
vote
0answers
34 views

Neural Nets: Benefits of Mixing Activation Functions? [duplicate]

I'm currently studying and would like to know if there are any actual benefits of using a mixture of activation functions for each layer. For example, if I am trying to predict the probabilities of ...
2
votes
1answer
296 views

How does the subtraction of the logit maximum improve learning?

My question comes from another question answered on Stackoverflow; the Keras implemantion of softmax activation function is customized to subtract the maximum value ...
1
vote
1answer
71 views

Softmax dealing with non present classes [closed]

Is it okey to assume that, if I have trained my neural network for image classification using k classes using a Softmax output function, whenever I feed an image that does not belong to any of the k ...
0
votes
2answers
266 views

Are weights 1-D or 2-D in softmax Regression?

I've learned ML and have been learning DL from Andrew Ng's coursera courses, and every time he talks about a linear classifier, the weights are just a 1-D vector. Even during the assignments, when we ...
1
vote
0answers
226 views

finding the loss derivative w.r.t. weights for a convolutional layer

Take the loss function: $$ \mathrm{loss} ~~=~~ \sum_{i=1}^N \left( -z_{}[y] + \log{\left( \sum_{c=1}^{10} \exp(z_{}[c]) \right)} \right)$$ where $z \in \mathbb{R}^{10}$ is the input to the softmax ...
1
vote
1answer
209 views

Why does deriving the softmax for a single vector come to 0 for me?

There is a lot of material explaining how to calculate the jacobian for the softmax backwards pass, but I find it confusing how to get to the actual errors from the jacobian. The obvious answer would ...
0
votes
1answer
182 views

What is the meaning of fully-convolutional cross entropy loss in the function below (image attached)?

I am trying to understand the loss function given below from this paper. I do not understand $l_{p}(x_{hw}; y_{hw})$ part.
3
votes
0answers
128 views

Expectation of the softmax transform for Gaussian multivariate variables

Prelims In the article Sequential updating of conditional probabilities on directed graphical structures by Spiegelhalter and Lauritzen they give an approximation to the expectation of a logistic ...
4
votes
2answers
351 views

How can't the Softmax layer never converge using hard targets

Here's a quote from the deep learning book by Ian Goodfellow (page 236): Maximum likelihood learning with a softmax classifier and hard targets may actually never converge -- the softmax can ...
4
votes
1answer
548 views

Is minimizing softmax cross entropy with two labels equivalent to minimizing sigmoid binary cross entropy with 1 label?

Is using a sigmoid at the end of a neural network with 1 label equivalent to using a softmax with 2 labels?
1
vote
0answers
116 views

Mutual exclusive classes for deciding Softmax Regression vs. k Binary Classifiers

I realise that similar questin is asked here also Softmax regression or $K$ binary logistic regression But my concern is related to last section of this article from stanford http://ufldl.stanford....
6
votes
2answers
828 views

What is distribution parameterization?

I encountered this term in the Stanford notes about softmax regression: we will begin by expressing the multinomial as an exponential family distribution. To parameterize a multinomial over k ...
0
votes
1answer
604 views

Training in one step vs multiple steps

I'm about 5 minutes in to learning about machine learning (using the Tensorflow MNIST tutorial) and have already managed to confuse myself. No big surprise there. But Google isn't giving me any good ...
8
votes
3answers
5k views

Why is softmax output not a good uncertainty measure for Deep Learning models?

I've been working with Convolutional Neural Networks (CNNs) for some time now, mostly on image data for semantic segmentation/instance segmentation. I've often visualized the softmax of the network ...
1
vote
1answer
1k views

Softmax with log-likelihood cost

I am working on my understanding of neural networks using Michael Nielsen's "Neural networks and deep learning." Now in the third chapter, I am trying to develop an intuition of how softmax works ...
3
votes
0answers
45 views

Why, empirically, does cross entropy composed with softmax have such a simple derivative? [closed]

Was one function chosen to simplify the derivatives based on the other? There has to be some intelligent design relating to their derivatives, right? The derivative is simply ...
1
vote
0answers
24 views

How can I make a classifier that adapts to increasing number of classes in the dataset. How many nodes should I use as output?

How can I make a classifier that adapts to an increasing number of classes in the dataset. How many nodes should I use as output? If I use a softmax I need to specify the classes but if the classes ...