1
$\begingroup$

Is there a good way to estimate the pdf, pdf up to a constant multiple, cdf, or quantile function of a distribution given the first n moments? A closed form for one of those functions in terms of the moments would be ideal, but I suspect that one does not exist in general.

The practical motivation here is that I'm trying to write a library for estimating various properties of a time series after the fact. Collections of moments have nice algebraic properties (because expectation is linear) which makes parallel processing easier.


A normal distribution is the maximum entropy distribution among distributions constrained to have mean $\mu$ and variance $\sigma^2$ .

According to this question and the Wikipedia article on variance, the mean (102) and variance (103) can be defined in terms of raw moments ($\mu_n$ where $n$ is the power).

$$ \mu_n \stackrel{\small{\text{def}}}{=} \mathop{\mathbb{E}} \left[ x^n \right] \tag{101} $$ $$ \mu \stackrel{\small{\text{def}}}{=} \mu_1 \tag{102} $$ $$ \sigma^2 \stackrel{\small{\text{def}}}{=} \mu_2 - \mu_1^2 \tag{103} $$

The normal distribution is naturally parameterized by $\mu$ and $\sigma^2$, but can also be thought of as being parameterized by $\mu$ and $\mu_2$ .

The answer by Glen_b references a theorem by Ludwig Boltzmann which suggests to me that the pdf in a situation where the first $n$ moments are known will be proportional to (104).

$$ \exp \circ \left( \text{some $n$ degree polynomial with no constant term} \right) \tag{104} $$

Assuming I haven't horribly misinterpreted the answer, is there a computationally efficient way to estimate these coefficients?

Moreover, is there a way to do an online estimate of the coefficients? Ideally I'd like to be able to update the coefficients of the polynomial in (104) at the same time that I update the moments when "merging in" new observations.

$\endgroup$

1 Answer 1

2
$\begingroup$

Maximum entropy principle helps you find the distribution with the highest entropy (most uncertainty, least biased) among all distributions that match first n sample moments. As you mentioned, the maximum entropy distribution for $n=2$ is a Gaussian distribution. For general case, such distribution is derived using first order optimality conditions of the corresponding optimization problem and is formulated as follows \begin{align} f(x) = \int \exp(-\sum_{j=0}^n \lambda_jx^j) \, dx\\ \text{s.t.} \int x^i \exp(-\sum_{j=0}^k \lambda_jx^j)\, dx = m_i, \forall i\in \{0,\dots,n\} [*] \end{align} where [*] are the moment constraints and $m_i$ are the sample moments with $m_0=1$.

So Lagrange Multipliers $\lambda \in \mathcal{R}^{k+1}$ are unknown parameters of the distribution. One way to find them is to switch to dual formulation and maximize the scaled log-likelihood function based on Gibbs distribution

\begin{align} -\big( \ln \int \exp(-\sum_{j=1}^n \lambda_jx^j) \, dx + \sum_{i=1}^n m_i\lambda_i \big) \rightarrow \text{max}_{\lambda} \end{align}

The problem is concave so you can use gradient-based methods to solve this maximization problem.

You can find more details in the paper of Mead and Papanicolaou "Maximum entropy in the problem of moments".

$\endgroup$
1
  • 1
    $\begingroup$ Nice answer, thanks for contributing it. I don't suppose there are closed-form equations for the special cases of $n=3$ or $n=4$, like we have a closed form expression (the normal distribution) in the special case of $n=2$? $\endgroup$ May 17, 2021 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.