2
$\begingroup$

On book Deep Learning, it says softmax function is de facto soft argmax function, and the corresponding soft version of the maximum function is $$softmax (z)^T z$$ How to understand the latter?

$\endgroup$
6
$\begingroup$

Consider the function $\text{hardmax}(z)^Tz$ for $z = [1, 2, 3, 4, 5]$ where hardmax is a hard version of softmax which returns 1 for the maximum component and 0 for all the other components. Then we will have $[0, 0, 0, 0, 1] ^T [1, 2, 3, 4, 5] = 5$. On the other hand, softmax of $z$ will be $[0.01, 0.03, 0.09, 0.23, 0.64]$ so $[0.01, 0.03, 0.09, 0.23, 0.64] ^T [1, 2, 3, 4, 5] = 4.46$. As you can see, softmax causes a weighted average on the components where the larger components are weighted more heavily.

$\endgroup$
  • $\begingroup$ how is 4.46 is computed here ? $\endgroup$ – Feras Apr 12 '18 at 14:48
  • $\begingroup$ It is the inner product of $z$ and the probabilities. $\endgroup$ – shimao Apr 14 '18 at 1:32
  • $\begingroup$ how is possible to apply the inner product this way between two 4x4 arrays where the first array is the softmax of the second ? $\endgroup$ – Feras Apr 18 '18 at 15:18
  • $\begingroup$ @Feras But they're vectors, not arrays $\endgroup$ – shimao Apr 18 '18 at 17:34
  • $\begingroup$ I think I wrote my comment quite fast. No not every output is a vector. You can have an array from fully convolution network. I was thinking lately how to apply soft maximum version instead of hard or global average without having problems with negative numbers or small fractions. Your proposal seems workable I'll give it a try. $\endgroup$ – Feras Apr 18 '18 at 17:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.