# Softmax: can't wrap my head around these values

I've got three simple classes each with some count values and I want to calculate the probability distribution. Column $$B$$ is the count and column $$C$$ is $$exp(count)$$. The last column then devides the cell left to it by $$C6$$ which is the sum of all exp(count)'s.

Now the weird thing is, why would 'Fur Elise' (which has with 302 counts out of 600 total about 50% probability) have now full 100%? I double checked each cell and I simply apply softmax.

• You have two answer provided to you @WouterVandenputte , if they answer your question, can you please accept and/or upvote them? – gunes Mar 26 at 20:55

Because the counts are very large. You actually calculate the following: $$\frac{e^{302}}{e^{302}+e^{51}+e^{247}}$$ which is of course not $$1$$, but extremely close to $$1$$ and Excel just rounds it. Note that the other numbers are also extremely small.

• I've also thought of that. But if softmax is so easily unstable, why is it even used? Can it therefore only be used when operating with small values? Are there any techniques to use it with larger values? – Wouter Vandenputte Feb 22 at 12:58
• Normally, feature are standardised before going into such functions. It's not the instability of softmax. You can't use exponentials with these numbers. – gunes Feb 22 at 13:41

Exponentials of numbers this large are beyond the edge of numerical stability. This is the trade-off of working with finite-precision arithmetic generally, not softmax specifically.

The model you're using is a little strange because it treats counts (0,1,2,...) as if they are logits (any real). Softmax assumes that the inputs are on the logit scale. For example, one model would give coefficients to each input to go from counts to logits to probabilities. The model yields probabilities of the form $$p(y_j|x) = \frac{\exp(x_j \beta_j)}{\sum_j\exp(x_j\beta_j)}$$

The sample proportions are also probabilities, for example, and a nice model arises from some (strong) assumptions about a sequence of multinomial trials. Sample proportions exactly match your intuition about Fur Elise having roughly 50% of the total counts implying it should have roughly 50% of the probability.

• I always thought softmax was used to give higher values also higher probabilities and lower values lower probabilities. a mere $\frac{x_i}{\sum{x_i}}$ would give 302/600 – Wouter Vandenputte Feb 22 at 14:29
• Your first sentence is true, because exponentiation means larger inputs are assigned even larger probabilities. I don't understand what your objection is. What problem are you trying to solve? How does softmax relate to solving that problem? – Sycorax says Reinstate Monica Feb 22 at 14:31