Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Suppose I will be getting some samples from a binomial distribution. One way to model my prior knowledge is with a Beta distribution with parameters $\alpha$ and $\beta$. As I understand it, this is equivalent to having seen "heads" $\alpha$ times in $\alpha + \beta$ trials. As such, a nice shortcut to doing the full-blown Bayesian inference is to use $\frac{h+\alpha}{n+\alpha+\beta}$ as my new mean for the probability of "heads" after having seen $h$ heads in $n$ trials.

Now suppose I have more than two states, so I will be getting some samples from a multinomial distribution. Suppose I want to use a Dirichlet distribution with parameter $\alpha$ as a prior. Again as a shortcut I can treat this as prior knowledge of event $i$'s probability as being equivalent to $\frac{\alpha_i}{\sum \alpha_j}$, and if I witness event $i$ $h$ times in $n$ trials my posterior for $i$ becomes $\frac{h + \alpha_i}{n + \sum \alpha_j}$.

Now in the binomial case, it works out that prior knowledge of "heads" occurring $\alpha$ times in $\alpha + \beta$ trials is equivalent to "tails" occurring $\beta$ times in $\alpha + \beta$ trials. Logically I don't believe I can have stronger knowledge of "heads" likelihood than of "tails." This gets more interesting with more than two outcomes though. If I have say a 6-sided die, I can imagine my prior knowledge of side "1" being equivalent to 10 ones in 50 trials and my prior knowledge of side "2" as being equivalent to 15 twos in 100 trials.

So after all of that introduction, my question is how I can properly model such asymmetric prior knowledge in the multinomial case? It seems as though if I'm not careful I can easily get illogical results due to total probability/likelihood not summing to 1. Is there some way I can still use the Dirichlet shortcut, or do I need to sacrifice this altogether and use some other prior distribution entirely?

Please forgive any confusion caused by potential abuses in notation or terminology above.

share|improve this question
If I have say a 6-sided die, I can imagine my prior knowledge of side "1" being equivalent to 10 ones in 50 trials and my prior knowledge of side "2" as being equivalent to 15 twos in 100 trials. Is there a special reason why you do not want to unify your ' equivalent sample sizes' for "$1$" and "$2$" (namely $50$ and $100$) by using their least common multiple? In this case $\alpha_1 = 20/100$ and $\alpha_2 = 15/100$. – Zhubarb Nov 7 '13 at 11:44
It lets you represent asymmetric priors while ensuring $\sum_{i=1}^{i=6}\alpha_i = 1$ – Zhubarb Nov 7 '13 at 12:00
Finally, this paper by Kemp at al. is very relevant and fun to read. – Zhubarb Nov 7 '13 at 13:52
@Berkan I haven't read the paper yet (thanks though!), but the reason I didn't unify the sample sizes is because 20/100 is not the same prior as 10/50, and that fact is very important to the reason this question was asked. – Michael McGowan Nov 7 '13 at 14:43
Because you are more confident in the former due to a higher equiv. sample size? – Zhubarb Nov 7 '13 at 15:15
up vote 2 down vote accepted

You have framed your question very well.

I think what you are looking for here is a case of hierarchical modeling. And you may want to model multiple layers of hierarchy (at the moment you only talk about priors). Having another layer of hyper-priors for the hyper--parameters lets you model the additional variabilities in hyper-parameters (as you are concerned about the variability issues of hyper-parameters). It also makes your modeling flexible and robust (may be slower).

Specifically in your case, you may benefit by having priors for the Dirichlet distribution parameters (Beta is a special case). This post by Gelman talks about how to impose priors on the parameters of Dirichlet distribution. He also cites on of his papers in a journal of toxicology.

share|improve this answer
I can be more modeling specific, but I don't want to guess here. If you chose to modify your question, I will add more details. – suncoolsu Feb 9 '11 at 14:52

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.