# Soft version of the maximum function?

In the book Deep Learning, it says the softmax function is de facto a soft argmax function, and the corresponding soft version of the maximum function is $$\text{softmax} (z)^T z$$

How to understand the latter?

Consider the function

$$\text{hardmax}(z)^Tz$$

for $$z = [1, 2, 3, 4, 5]$$

where hardmax is a hard version of softmax, which returns 1 for the maximum component and 0 for all the other components.

Then we will have

$$[0, 0, 0, 0, 1] ^T [1, 2, 3, 4, 5] = 5$$.

On the other hand, softmax of $$z$$ will be $$[0.01, 0.03, 0.09, 0.23, 0.64]$$

so $$[0.01, 0.03, 0.09, 0.23, 0.64] ^T [1, 2, 3, 4, 5] = 4.46$$.

As you can see, softmax causes a weighted average on the components where the larger components are weighted more heavily.

• how is 4.46 is computed here ? Commented Apr 12, 2018 at 14:48
• It is the inner product of $z$ and the probabilities. Commented Apr 14, 2018 at 1:32
• how is possible to apply the inner product this way between two 4x4 arrays where the first array is the softmax of the second ? Commented Apr 18, 2018 at 15:18
• @Feras But they're vectors, not arrays Commented Apr 18, 2018 at 17:34
• I think I wrote my comment quite fast. No not every output is a vector. You can have an array from fully convolution network. I was thinking lately how to apply soft maximum version instead of hard or global average without having problems with negative numbers or small fractions. Your proposal seems workable I'll give it a try. Commented Apr 18, 2018 at 17:55

You can construct a smoother version of max function using softmax function, as the expression in your book suggests.

Consider the following formulation of a max function: $$\max(z_1,\dots,z_n)=\mathrm{argmax}(z)\times z^T$$ The function argmax returns a vector with 0s and 1s. Thus it produces a rough max function. Rough in sense that its first derivative wrt its arguments is discontinuous: it's either 0 or 1. Whenever $$z_i=z_j$$ the first derivative jumps between 0 and 1.

By replacing argmax with what machine learning people call softmax you get a smooth version of max function too, as suggested in your book. Here's a couple of charts to demonstrate the point. The following is a surface of an ordinary $$\max(x_1,x_2)$$ function.

Compare it to the version using the expression from your textbook $$\mathrm{softmax}(x_1,x_2)^T\times (x_1,x_2)$$:

Smoother version of max can be easier to deal with analytically.

• +1, can't believe no upvote for this answer... Commented Oct 25, 2021 at 5:45

softmax is a smooth approximation of the argmax function,* taking a vector and returning a vector:

$$\text{softmax}(x) = \frac{e^{\beta x}}{\sum{e^{\beta x}}} \to \text{argmax}(x)$$

This takes a vector as input and returns a vector as output (a one-hot encoding of the max's index, as opposed to an ordinal position).

In order to get a smooth approximation of the max function, which returns the largest value in a vector (not its index), one can take the dot product of the softmax with the original vector:

$$\text{softmax}(x)^Tx \to \text{argmax}(x)^Tx = \max(x)$$

* Note that softmax, in the case of multiple identical maximum values, will return a vector with $$1/n$$ in the maximum values' arguments' positions, not multiple 1s.

* In softmax, $$\beta = 1$$, and as it approaches infinity, the function approaches argmax.