# Estimating maximum entropy distribution given first n moments

Is there a good way to estimate the pdf, pdf up to a constant multiple, cdf, or quantile function of a distribution given the first n moments? A closed form for one of those functions in terms of the moments would be ideal, but I suspect that one does not exist in general.

The practical motivation here is that I'm trying to write a library for estimating various properties of a time series after the fact. Collections of moments have nice algebraic properties (because expectation is linear) which makes parallel processing easier.

A normal distribution is the maximum entropy distribution among distributions constrained to have mean $$\mu$$ and variance $$\sigma^2$$ .

According to this question and the Wikipedia article on variance, the mean (102) and variance (103) can be defined in terms of raw moments ($$\mu_n$$ where $$n$$ is the power).

$$\mu_n \stackrel{\small{\text{def}}}{=} \mathop{\mathbb{E}} \left[ x^n \right] \tag{101}$$ $$\mu \stackrel{\small{\text{def}}}{=} \mu_1 \tag{102}$$ $$\sigma^2 \stackrel{\small{\text{def}}}{=} \mu_2 - \mu_1^2 \tag{103}$$

The normal distribution is naturally parameterized by $$\mu$$ and $$\sigma^2$$, but can also be thought of as being parameterized by $$\mu$$ and $$\mu_2$$ .

The answer by Glen_b references a theorem by Ludwig Boltzmann which suggests to me that the pdf in a situation where the first $$n$$ moments are known will be proportional to (104).

$$\exp \circ \left( \text{some n degree polynomial with no constant term} \right) \tag{104}$$

Assuming I haven't horribly misinterpreted the answer, is there a computationally efficient way to estimate these coefficients?

Moreover, is there a way to do an online estimate of the coefficients? Ideally I'd like to be able to update the coefficients of the polynomial in (104) at the same time that I update the moments when "merging in" new observations.

Maximum entropy principle helps you find the distribution with the highest entropy (most uncertainty, least biased) among all distributions that match first n sample moments. As you mentioned, the maximum entropy distribution for $$n=2$$ is a Gaussian distribution. For general case, such distribution is derived using first order optimality conditions of the corresponding optimization problem and is formulated as follows \begin{align} f(x) = \int \exp(-\sum_{j=0}^n \lambda_jx^j) \, dx\\ \text{s.t.} \int x^i \exp(-\sum_{j=0}^k \lambda_jx^j)\, dx = m_i, \forall i\in \{0,\dots,n\} [*] \end{align} where [*] are the moment constraints and $$m_i$$ are the sample moments with $$m_0=1$$.
So Lagrange Multipliers $$\lambda \in \mathcal{R}^{k+1}$$ are unknown parameters of the distribution. One way to find them is to switch to dual formulation and maximize the scaled log-likelihood function based on Gibbs distribution
\begin{align} -\big( \ln \int \exp(-\sum_{j=1}^n \lambda_jx^j) \, dx + \sum_{i=1}^n m_i\lambda_i \big) \rightarrow \text{max}_{\lambda} \end{align}
• Nice answer, thanks for contributing it. I don't suppose there are closed-form equations for the special cases of $n=3$ or $n=4$, like we have a closed form expression (the normal distribution) in the special case of $n=2$? May 17, 2021 at 14:26