I am reading the work of Welling on Vartiational Auto-Encoders (VAE), and wonder if there is any way to generate Gamma distributed samples via a similar reparametrization?

The idea of reparameterization is quite simple. To generate a sample from a particular distribution, i.e. $z\sim N(\mu,\sigma)$, we generate samples from another distribution, i.e. $\epsilon\sim N(0,1)$, and then transform it into genuine samples from the desired distribution, i.e. via $z=\mu+\sigma\times\epsilon$. This has some nice consequences in the VAE framework, i.e., reduces the variance of the gradient (not really important here).

I basically wonder if such transformation exists for Gamma distribution.

  • $\begingroup$ can you add a little more detail/context for those who don't have time to dig in and read the whole linked article? $\endgroup$
    – Ben Bolker
    Sep 22 '18 at 15:48
  • $\begingroup$ @BenBolker I added the description. $\endgroup$ Sep 22 '18 at 15:52

The answer is "yes" in one sense and "no" in another sense.

Suppose $X \sim \operatorname{Gamma}(\alpha,\beta)$. Let $F_{\alpha,\beta}$ denote the cdf of a Gamma distribution. Then define $\epsilon = \Phi^{-1}[F_{\alpha,\beta}(X)]$. Then if you simulate $\epsilon \sim N(0,1)$ you can get the relevant gamma distribution by setting $X = F^{-1}_{\alpha,\beta}[\Phi(\epsilon)]$. This is a consequence of the probability integral transform. Additionally, the transform $T(\epsilon ; \alpha, \beta) = F^{-1}_{\alpha,\beta}[\Phi(\epsilon)]$ is differentiable. So you could use this idea with the reparametrization trick, at least in principle, to improve your stochastic variational inference. This implies that, in a liberal sense, the answer is "yes, there is a reparameterization trick", and in fact there is one for essentially any family of continuous distributions. If this seems sort of ad-hoc, notice that if you apply this trick with the Gaussian family in place of the gamma, you get back exactly the usual reparameterization trick.

In a more restrictive sense, I would say the answer is "no". The function $F^{-1}$ above is not available in closed form, so things are not so convenient to the point where we might disqualify this approach. Alternatively, there is no reason to restrict ourselves to $\epsilon \sim N(0,1)$, and we might just ask for $\epsilon \sim Q$ for some standard distribution $Q$ that is easy to sample from, such that $T(\epsilon; \alpha, \beta) \sim \text{Gamma}(\alpha,\beta)$ where $T$ is also easy to compute.

If you find such a transformation $T$ and standard distribution $Q$, let me know because I would be interested in it. The main problem is the shape parameter $\alpha$. If I know $\alpha$, then I can take $T(\epsilon; \alpha, \beta) = \epsilon \beta$ and set $\epsilon \sim \mbox{Gamma}(\alpha,1)$, because the gamma family with know $\alpha$ is a scale family. The shape parameter does not have any nice algebraic properties as far as I know, aside from the fact that $X_1 + X_2 \sim \mbox{Gamma}(\alpha_1 + \alpha_2, 1)$ provided that $X_i \sim \mbox{Gamma}(\alpha_i, 1)$ and they are independent. It's not clear how to take advantage of this. A negative result for us is that, if such a convenient $T$ existed, then R would probably use that to sample from the a generic gamma distribution, but instead it uses rejection sampling.

  • $\begingroup$ Hi; I'm recently encounter a similar problem; but I'm trying to do for Beta distribution such that $\mathcal{Q}$ is also Beta; do you have any idea how to do it ? $\endgroup$
    – ElleryL
    Oct 2 '19 at 20:41
  • 1
    $\begingroup$ Is there a reason why you draw from $N(0,1)$ instead of just doing $F^{-1}_{\alpha, \beta}[\epsilon]$ where $\epsilon \sim U(0,1)$? $\endgroup$
    – actinidia
    Aug 23 '20 at 1:44
  • 1
    $\begingroup$ @tiwa Not particularly. $\endgroup$
    – guy
    Aug 23 '20 at 3:29

In terms of the strategy from the VAE paper, no there's not a straightforward way of using a gamma distribution (as guy explained). However, there's a lot of research that's gone into other strategies for reparameterizing gammas to use in the VAE type framework:

You could alternatively use BBVI (here's the original Blei lab paper http://www.cs.columbia.edu/~blei/papers/RanganathGerrishBlei2014.pdf), as demonstrated in this tutorial http://ajbc.io/resources/bbvi_for_gammas.pdf


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.