9
$\begingroup$

Suppose I will be getting some samples from a binomial distribution. One way to model my prior knowledge is with a Beta distribution with parameters $\alpha$ and $\beta$. As I understand it, this is equivalent to having seen "heads" $\alpha$ times in $\alpha + \beta$ trials. As such, a nice shortcut to doing the full-blown Bayesian inference is to use $\frac{h+\alpha}{n+\alpha+\beta}$ as my new mean for the probability of "heads" after having seen $h$ heads in $n$ trials.

Now suppose I have more than two states, so I will be getting some samples from a multinomial distribution. Suppose I want to use a Dirichlet distribution with parameter $\alpha$ as a prior. Again as a shortcut I can treat this as prior knowledge of event $i$'s probability as being equivalent to $\frac{\alpha_i}{\sum \alpha_j}$, and if I witness event $i$ $h$ times in $n$ trials my posterior for $i$ becomes $\frac{h + \alpha_i}{n + \sum \alpha_j}$.

Now in the binomial case, it works out that prior knowledge of "heads" occurring $\alpha$ times in $\alpha + \beta$ trials is equivalent to "tails" occurring $\beta$ times in $\alpha + \beta$ trials. Logically I don't believe I can have stronger knowledge of "heads" likelihood than of "tails." This gets more interesting with more than two outcomes though. If I have say a 6-sided die, I can imagine my prior knowledge of side "1" being equivalent to 10 ones in 50 trials and my prior knowledge of side "2" as being equivalent to 15 twos in 100 trials.

So after all of that introduction, my question is how I can properly model such asymmetric prior knowledge in the multinomial case? It seems as though if I'm not careful I can easily get illogical results due to total probability/likelihood not summing to 1. Is there some way I can still use the Dirichlet shortcut, or do I need to sacrifice this altogether and use some other prior distribution entirely?

Please forgive any confusion caused by potential abuses in notation or terminology above.

$\endgroup$
6
  • $\begingroup$ If I have say a 6-sided die, I can imagine my prior knowledge of side "1" being equivalent to 10 ones in 50 trials and my prior knowledge of side "2" as being equivalent to 15 twos in 100 trials. Is there a special reason why you do not want to unify your ' equivalent sample sizes' for "$1$" and "$2$" (namely $50$ and $100$) by using their least common multiple? In this case $\alpha_1 = 20/100$ and $\alpha_2 = 15/100$. $\endgroup$
    – Zhubarb
    Nov 7, 2013 at 11:44
  • $\begingroup$ It lets you represent asymmetric priors while ensuring $\sum_{i=1}^{i=6}\alpha_i = 1$ $\endgroup$
    – Zhubarb
    Nov 7, 2013 at 12:00
  • $\begingroup$ Finally, this paper by Kemp at al. is very relevant and fun to read. $\endgroup$
    – Zhubarb
    Nov 7, 2013 at 13:52
  • $\begingroup$ @Berkan I haven't read the paper yet (thanks though!), but the reason I didn't unify the sample sizes is because 20/100 is not the same prior as 10/50, and that fact is very important to the reason this question was asked. $\endgroup$ Nov 7, 2013 at 14:43
  • $\begingroup$ Because you are more confident in the former due to a higher equiv. sample size? $\endgroup$
    – Zhubarb
    Nov 7, 2013 at 15:15

1 Answer 1

3
$\begingroup$

You have framed your question very well.

I think what you are looking for here is a case of hierarchical modeling. And you may want to model multiple layers of hierarchy (at the moment you only talk about priors). Having another layer of hyper-priors for the hyper--parameters lets you model the additional variabilities in hyper-parameters (as you are concerned about the variability issues of hyper-parameters). It also makes your modeling flexible and robust (may be slower).

Specifically in your case, you may benefit by having priors for the Dirichlet distribution parameters (Beta is a special case). This post by Gelman talks about how to impose priors on the parameters of Dirichlet distribution. He also cites on of his papers in a journal of toxicology.

$\endgroup$
1
  • $\begingroup$ I can be more modeling specific, but I don't want to guess here. If you chose to modify your question, I will add more details. $\endgroup$
    – suncoolsu
    Feb 9, 2011 at 14:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.