6
$\begingroup$

Suppose I know the value of n specific quantiles from a large sample. What is the maximum entropy distribution over the real line given those values?

$\endgroup$
3
  • $\begingroup$ Have you tried formulating and solving this as an optimization problem of maximizing entropy subject to equality constraints on the n known quantiles? I think you should be able to get the idea of things by Googling and looking at similar problems. $\endgroup$ Aug 11, 2016 at 1:47
  • $\begingroup$ I'll poke about and see what I can find. But if g1ul10's response, below, is correct -- and I think it is -- then that approach will not work without further constraints. $\endgroup$
    – andrewH
    Aug 18, 2016 at 0:23
  • $\begingroup$ Here is a similar problem with answers: stats.stackexchange.com/questions/361476/… $\endgroup$ Jul 1, 2023 at 1:08

1 Answer 1

6
$\begingroup$

Maximum entropy problems do not always admit a solution. The generic expression for the maximum entropy density $f(x)$ given a set of integral constraints \begin{equation} \int dx \, h_i(x) \, f(x) = c_i \end{equation} with $i=1\ldots N$ is \begin{equation} f(x) = e^{\mu + \sum_{i=1}^N \lambda_i h_i(x)} \;. \end{equation} The values of the parameter $\mu$ and $\lambda_i$ have to be found by imposing the fulfillment of the constraints and the fact the $f(x)$ integrates to one, that is it is a proper density. I left the boundaries of the integral in the constraints unspecified on purpose. The reason will become clear in a moment.

The quantile constraints $F(x_i)=q_i$ where $F$ is the distribution function (the integral of the $f(x)$), translate in having $h_i(x)=1-\theta_{x_i}(x)$ and $c_i=q_i$, where $\theta_z(x)$ is the Heaviside theta function, which is equal to $1$ if $x>z$ and zero otherwise. The problem is that now the expression for $f(x)$ given above is not an (improper) integrable function on the real line. Summarizing, your problem does not have a solution. If however you add further constraints, like a finite support or some specified moment, than the problem might become solvable.

$\endgroup$
4
  • $\begingroup$ I don't have moments, etc., because high and low values are censored. Would it be sufficient if I had a candidate functional form? $\endgroup$
    – andrewH
    Aug 18, 2016 at 0:20
  • $\begingroup$ So, that follow-up question was meaningless. The maximum entropy distribution would be the candidate distribution if there was one. Sorry. Here is a question that makes more sense: Would the observed quantiles suffice for a strictly positive distribution? $\endgroup$
    – andrewH
    Aug 26, 2016 at 20:19
  • $\begingroup$ @andrewH No. The solution would have a constant behavior for large $x$ and the integral on the positive half-line would not exist. You could find a solution if you consider a finite domain, like $[0,1]$. In this case, however, the solution will be rather trivial, with a step-wise shape. $\endgroup$
    – g1ul10
    Aug 29, 2016 at 8:13
  • $\begingroup$ @g1ul10 can the quantiles be used to estimate the information entropy of the sample? $\endgroup$
    – Kuzeko
    Dec 11, 2018 at 22:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.