The max entropy method is a way of deriving a probability distribution given only the information you know about how the data is distributed and nothing more. For example the normal distribution can be derived by constraining the mean and variance to be fixed. This is a very intuitive and principled way of choosing how to model your data.

I'm trying to wrap my head around the constraints for the gamma distribution, which does not seem as intuitive. These constraints are $$\mathbf E[x] = k\theta,$$ $$\mathbf E[\log(x)] = \psi(k) + \log(\theta),$$ where $k$ is the shape parameter, $\theta$ is the scale parameter, and $\psi$ is the digamma function. There is also the implicit constraint that $x \ge 0$.

One possibility I see here is that perhaps it's less about the specifics of what values these constraints take, and more about the fact that they are fixed -- the values are circumstantial. But then is there any intuition behind these values? If the specific values do matter, under what circumstances would you assume the mean to be a product of two parameters? Given the scale parameter is included, are we implicitly making a constraint on the variance?

Also, I've noticed that the second term looks like one of the constraints for the max entropy that generates the beta distribution. What is the connection there?

Any insight here would be helpful. I've scoured the web and have found little to help me out here.


1 Answer 1


This touches on some interesting ideas in statistics.

Firstly, you are correct, the way that we choose to parameterize the family of distributions doesn't really matter. Given an invertible mapping from one set of parameters to another, $\eta = F(\theta)$, any distribution $p_\theta(x|\theta)$ can also be expressed as $$ p_\eta(x | \eta) = p_\theta(x| F^{-1}( \eta) ).$$ We can remap the parameters. For example, for a gaussian, rather that mean $\mu$ and variance $\sigma^2$, we could write $\eta_1 =\frac{\mu}{\sigma^2}$, $\eta_2 = -\frac{1}{2\sigma^2}$, so the gaussian pdf becomes: $$ p(x | \eta_1, \eta_2) \propto \exp( \eta_2 x^2 + \eta_1 x) $$

Even though the parameters are arbitrary, there are two pretty natural ways to write them from the perspective of the maximum entropy problem. Writing the max entropy problem: $$ \max_{p(x)} \int -p(x) \log p(x) dx \quad \textrm{such that} \quad \forall i \int p(x) T_i(x) dx = \mu_i$$ We get a solution: $$ p(x) \propto \exp \left( \sum_i \lambda_i T_i(x)\right)$$

Where $T_i(x)$ are the 'sufficient statistics' functions that we are constraining (eg. $T_1(x) = x$ for mean $T_2(x) = x^2$ for variance), $\mu_i$ are the values of these sufficient statistics, and $\lambda_i$ are the Lagrange multipliers for these constraints.

The $\mu_i$ are the 'moment parameters': they tell us about the values of our constraints. The $\lambda_i$ are the 'natural parameters': they are nice to calculate with (and, as Lagrange multipliers, give us 'prices' of constraint violation). For a gaussian, $\mu$ and $\sigma^2$ are moment parameters, while the $\eta_1$ and $\eta_2$ derived above are the natural parameters. The relationship between moment and natural parameters is the constraint value $\leftrightarrow$ Lagrange multiplier relationship, and is encoded in the partition function. See exponential families for more information and information geometry for a theory built on these types of observations.

For the particular case of the gamma distribution, the relationship of parameters $k$ and $\theta$ to the natural parameters is more intuitive than the relationship to mean parameters $k= \eta_1 +1$, $\theta = -\frac{1}{\eta_2}$. The beta distribution is usually written in terms of natural parameters as well and also imposes a constraint on $\mathbb{E}[\log x]$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.