Suppose you have i.i.d. variables $x_i$ in ${1,\ldots,K}$ modeled as

$$P(x_i = k) = \theta_k$$

and and you want to infer the probability vector $\theta$. A Bayesian approach puts a prior over $\theta$, and the Dirichlet distribution is often used for this purpose when $K$ is not too large.

I am interested in the case where $K$ may be very large -- for example, $i$ may correspond to the $i$-th word in a very large dictionary. Furthermore, we expect to see some sort of power-law behavior. If we order the elements of $\theta$ by size, that is we have some permutation $\pi$ on ${1,\ldots,K}$ with $\theta_{\pi(k)} \geq \theta_{\pi(k+1)}$ for all $k$, then we expect the sum

$$\sum_{k>n} \theta_{\pi(k)}$$

to be roughly proportional to $n^{-a}$ for some $a > 0$. Put another way, I'm looking for a prior with

$$E\left[\sum_{k>n} \theta_{\pi(k)}\right] \propto n^{-a}$$

for $0\leq n < K$ and

$$E\left[\theta_k\right] = 1/K$$

for $1\leq k\leq K$.

Does anyone know of a prior that has this property? Any academic papers that look at this kind of thing?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.