Soft version of the maximum function?

Question

In the book Deep Learning, it says the softmax function is de facto a soft argmax function, and the corresponding soft version of the maximum function is $$\text{softmax} (z)^T z$$

How to understand the latter?

Answer 1

Consider the function

$\text{hardmax}(z)^Tz$

for $z = [1, 2, 3, 4, 5]$

where hardmax is a hard version of softmax, which returns 1 for the maximum component and 0 for all the other components.

Then we will have

$[0, 0, 0, 0, 1] ^T [1, 2, 3, 4, 5] = 5$.

On the other hand, softmax of $z$ will be $[0.01, 0.03, 0.09, 0.23, 0.64]$

so $[0.01, 0.03, 0.09, 0.23, 0.64] ^T [1, 2, 3, 4, 5] = 4.46$.

As you can see, softmax causes a weighted average on the components where the larger components are weighted more heavily.

Answer 2

You can construct a smoother version of max function using softmax function, as the expression in your book suggests.

Consider the following formulation of a max function: $$\max(z_1,\dots,z_n)=\mathrm{argmax}(z)\times z^T$$ The function argmax returns a vector with 0s and 1s. Thus it produces a rough max function. Rough in sense that its first derivative wrt its arguments is discontinuous: it's either 0 or 1. Whenever $z_i=z_j$ the first derivative jumps between 0 and 1.

By replacing argmax with what machine learning people call softmax you get a smooth version of max function too, as suggested in your book. Here's a couple of charts to demonstrate the point. The following is a surface of an ordinary $\max(x_1,x_2)$ function.

Compare it to the version using the expression from your textbook $\mathrm{softmax}(x_1,x_2)^T\times (x_1,x_2)$:

Smoother version of max can be easier to deal with analytically.

Answer 3

softmax is a smooth approximation of the argmax function,* taking a vector and returning a vector:

$$\text{softmax}(x) = \frac{e^{\beta x}}{\sum{e^{\beta x}}} \to \text{argmax}(x)$$

This takes a vector as input and returns a vector as output (a one-hot encoding of the max's index, as opposed to an ordinal position).

In order to get a smooth approximation of the max function, which returns the largest value in a vector (not its index), one can take the dot product of the softmax with the original vector:

$$\text{softmax}(x)^Tx \to \text{argmax}(x)^Tx = \max(x)$$

---

* Note that softmax, in the case of multiple identical maximum values, will return a vector with $1/n$ in the maximum values' arguments' positions, not multiple 1s.

* In softmax, $\beta = 1$, and as it approaches infinity, the function approaches argmax.