6
$\begingroup$

Suppose you have i.i.d. variables $x_i$ in ${1,\ldots,K}$ modeled as

$$P(x_i = k) = \theta_k$$

and and you want to infer the probability vector $\theta$. A Bayesian approach puts a prior over $\theta$, and the Dirichlet distribution is often used for this purpose when $K$ is not too large.

I am interested in the case where $K$ may be very large -- for example, $i$ may correspond to the $i$-th word in a very large dictionary. Furthermore, we expect to see some sort of power-law behavior. If we order the elements of $\theta$ by size, that is we have some permutation $\pi$ on ${1,\ldots,K}$ with $\theta_{\pi(k)} \geq \theta_{\pi(k+1)}$ for all $k$, then we expect the sum

$$\sum_{k>n} \theta_{\pi(k)}$$

to be roughly proportional to $n^{-a}$ for some $a > 0$. Put another way, I'm looking for a prior with

$$E\left[\sum_{k>n} \theta_{\pi(k)}\right] \propto n^{-a}$$

for $0\leq n < K$ and

$$E\left[\theta_k\right] = 1/K$$

for $1\leq k\leq K$.

Does anyone know of a prior that has this property? Any academic papers that look at this kind of thing?

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.