1
$\begingroup$

We know that one approach for re-parametrizing gradients for variational inference is taking the Gumbel-softmax estimator proposed in [1] and [2].

In [3], that is a talk on SVI, D. Blei at around 29:31 mentions that there is no re-parametrization for discrete variables for VI (also on the slides). The talk is one year after the papers. Since I find it highly unlikely that something like that would have gotten under the radar of a leading expert what am I missing?

More concretely, the questions I have are the following:

  1. Is the Gumbel softmax trick considered to have solved in a satisfying manner the issue of low-variance gradients in SVI with re-parametrization or not?
  2. Is the statement in the talk simply inaccurate?

This is not to be perceived as a criticism of the talk - I highly recommend it by the way. I am truly asking to clarify my confusion on the subject.

[1]: Categorical Reparametrization with Gumbel-Softmax (https://arxiv.org/pdf/1611.01144.pdf)

[2]: The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables (https://arxiv.org/abs/1611.00712)

[3]: Talk by D. Blei on SVI: https://www.youtube.com/watch?v=-H2N4tVDK7I

$\endgroup$

1 Answer 1

0
$\begingroup$

Is the Gumbel softmax trick considered to have solved in a satisfying manner the issue of low-variance gradients in SVI with re-parametrization or not?

I think in [1], at first Gumbel-max trick is used which is an exact method to obtain samples from the categorical distribution. But, Gumbel-max trick (eqn 1 in paper [1]) introduces arg-max which needs is approximated by using the softmax function (eqn. 2 in paper [1]). Basically, this approximation enables automatic differentiation but also introduces a hyperparameter $\tau$ which determines the variance. If $\tau$ is small, the arg-max approximation is accurate but it leads to gradients with high variance. Conversely, if $\tau$ is large, the arg-max approximation may become inaccurate (see figure 1 and section 2.1 in paper [1]). The paper introduces a novel method that solves gradient backpropagation problem for discrete latent variables but I think the solution is not perfect (as there is an approximation for arg-max)

2)Is the statement in the talk simply inaccurate?

I think the statement in the talk is correct. The solution is not perfect or exact as in Auto-Encoding Variational Bayes (Diederik P Kingma, Max Welling).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.