# Soft version of the maximum function?

On book Deep Learning, it says softmax function is de facto soft argmax function, and the corresponding soft version of the maximum function is $$softmax (z)^T z$$ How to understand the latter?

Consider the function $\text{hardmax}(z)^Tz$ for $z = [1, 2, 3, 4, 5]$ where hardmax is a hard version of softmax which returns 1 for the maximum component and 0 for all the other components. Then we will have $[0, 0, 0, 0, 1] ^T [1, 2, 3, 4, 5] = 5$. On the other hand, softmax of $z$ will be $[0.01, 0.03, 0.09, 0.23, 0.64]$ so $[0.01, 0.03, 0.09, 0.23, 0.64] ^T [1, 2, 3, 4, 5] = 4.46$. As you can see, softmax causes a weighted average on the components where the larger components are weighted more heavily.
• It is the inner product of $z$ and the probabilities. – shimao Apr 14 '18 at 1:32