4
$\begingroup$

In deep learning, especially generative models, sometimes we need to add some random noise to the input of model. To make the sampling of random noise learnable (or differentiable), we need to reparameterize the sampling, just like diffusion models do.

For example, given x ~ N (μ, σ2), if we sampled x directly, the mean and variance would not be differentiable. In practice we need to reparameterize it with x = μ + σ*ε, where ε ~ N (0, 1). So we sample ε instead of x, making μ and σ differentiable and learnable.

But if given x ~ Poisson (λ), how to reparameterize it to make λ learnable?

$\endgroup$

1 Answer 1

3
$\begingroup$

You can re-parameterize $x$ by using the quantile function of the Poisson distribution.

$$x = Q(\epsilon;\lambda) \qquad \text{where $\epsilon \sim U(0,1)$}$$

If you also want the function to be differentiable then you might want to consider a gamma distribution (with a continuous support) instead of a Poisson distribution (with a discrete support).

Compare below the Poisson distribution and the gamma distribution (with fixed scale=1) for equal values of the mean. The variance as function of the mean is in both cases $\text{Var}(X) = \text{Mean}(X)$.

comparison of gamma and Poisson

$\endgroup$
11
  • $\begingroup$ I may not use Gamma distribution because its probability density function differs from Poisson. If there are no precise methods, perhaps approximating the Poisson distribution by a Gaussian is better when lambda is big. $\endgroup$
    – Lorin60
    Jun 14, 2023 at 13:37
  • 2
    $\begingroup$ @Lorin60 the Poisson has $\text{Var}(X) = \text{Mean}(X)$ whereas the Gamma has $\text{Var}(X) \propto \text{Mean}(X)^2$ if the shape parameter is fixed. But if you keep the scale of the gamma distribution fixed and equal to one, then the Gamma distribution has the same behaviour $\text{Var}(X) = \text{Mean}(X)$. (In a glm framework they would be the same) $\endgroup$ Jun 14, 2023 at 16:32
  • $\begingroup$ An alternative is to use the conjugate prior for the shape parameter, which has the same form as the Poisson distribution, except that the discrete factorial is replaced by a gamma function. But that distribution may not be so easy to work with. $\endgroup$ Jun 14, 2023 at 16:32
  • $\begingroup$ Thanks. I see. Gamma is a good approximation of Poisson when the scale of Gamma equals to 1. $\endgroup$
    – Lorin60
    Jun 15, 2023 at 6:26
  • $\begingroup$ @Lorin60 and with different scales you can consider it as an over/under-dispersed Poisson distribution. $\endgroup$ Jun 15, 2023 at 6:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.