Timeline for Why do we need the temperature in Gumbel-Softmax trick?
Current License: CC BY-SA 4.0
9 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Oct 5, 2022 at 7:54 | comment | added | avocado | So is there any heuristic guideline when select the $\tau$? | |
| May 1, 2021 at 21:26 | comment | added | shimao | @CD86 what does "leaving out" temperature imply? setting it to 1? or 0? you might want to start a separate question for this | |
| May 1, 2021 at 19:58 | comment | added | CD86 | shimaos answer also does not satisfy me unfortunately. How does leaving out the temperature make the function non-differentiable it not clear to me | |
| Sep 17, 2018 at 14:35 | comment | added | shimao | @user3639557 $\tau = 1$ creates biased gradients. Choosing $\tau = 1$ is as arbitrary as choosing $\tau = \pi$. If having $\tau$ at all is a "hack" then setting $\tau$ to an arbitrary value of 1 is surely also a hack. The "default value" of $\tau$ is not 1, but 0, but we can't really use that because it makes the function non-differentiable. | |
| Sep 17, 2018 at 12:10 | comment | added | user3639557 | Simple normalization (via softmax) of Gumbel-perturbed values corresponds to the case that doesn't include $\tau$ (or $\tau$=1). The question is why don't we just use that. Based on your answer, it seems you are postulating $\tau$ is included to control the variance of gradient. And I am wondering if this was just a hack after all? Otherwise, there must be a mathematical derivation that starts from something and reaches this particular form of softmax with temperature. | |
| Sep 16, 2018 at 15:34 | comment | added | shimao | @user3639557 You asked why temperature is needed: without temperature (with temperature defaulting to 0), you have the nondifferentiable function argmax, which is a problem for backpropagation. | |
| Sep 14, 2018 at 18:53 | comment | added | user3639557 | Thanks for the comment. But I think this still doesn't answer the question. You are explaining the impact of $\tau$ on the gradient. If we consider the temperature inclusion as a hack to control the gradient variance, then your answer is fine. | |
| Sep 14, 2018 at 17:33 | history | edited | shimao | CC BY-SA 4.0 |
added 181 characters in body
|
| Sep 14, 2018 at 17:23 | history | answered | shimao | CC BY-SA 4.0 |