Suppose we have multinomial distribution in which we have 4 categories, and each one is associated with a probability of being selected, say $\theta_i$, $i=1,..,4$.

And I know for sure that $\theta_1=0.1, \theta_2=0.2$. Now I want to Bayesain inference the unknown $\theta_3$ (the $\theta_4$ can be deduced since $\sum \theta_i=1$) from evidence, say $n_i$ observstions for each category $i$.

The problem is, I see two ways to do this:

  1. As the one here :multinomial model with some certain parameters. In other words, discard the evidence of $n_1, n_2$ and rescale the $\theta_3,\theta_4$ as a binomial distribution then do the Bayesian inference only with evidence of $n_3,n_4$.

  2. Use marginal distribution of $\theta_3$, i.e. define a new binomial distribution with probabilities $p_3=\theta_3$ and $1-p_3=0.7-\theta_3$. Then do Bayesian inference for $p_3$ with evidence $n_3$ and $(n_1+n_2+n_4)$.

I guess these two ways are both correct and will reach the same results for a $E(\theta_3|data)$, am I right? just to confirm...

PS, once I built a new binomial distribution, to do the Bayesian, I do not have to use Beta distribution as my prior for $p_3$, right? I can use whatever I want, e.g. a two-point one..

Thanks guys.

  • $\begingroup$ Regarding your PS: in general you're free to use any prior you want. In practice, certain priors may not result in an easy-to-sample posterior. Point priors, however, shouldn't pose an issue. $\endgroup$ – zipzapboing Jun 8 '18 at 23:50

In general, these are not equivalent. Intuitively, if we assume $p_1$, $p_2$ are known, we should have less posterior uncertainty about $p_3$ and $p_4$. Another way to look at it is this: given that we know $p_1$ and $p_2$, we also know $p_3+p_4$. The only free parameter is equivalent to $p_3/p_4$, so $n_1$ and $n_2$ are not useful to us in any way.

Here is a little simulation bearing that out. (Note that I used Jeffrey's prior, $\operatorname{Beta}(0.5,0.5)$ and plots are facetted by sample size and method.)

The first plot shows that the posterior expectation has a similar sampling distribution and the second shows that the posterior standard deviation is larger using the marginal method as compared with the ignoring method.


results <- ldply(c(10, 100, 1000), function(sz){

  n <- rmultinom(1000, size=sz, prob = c(.1,.2,.3,.4))

  jeffreys_prior <- c(.5,.5)

  E_post <- function(f, s, prior){
    (s + prior[2])/(s+f+sum(prior))
  SD_post <- function(f, s, prior){
    sqrt((f + prior[1])*(s+prior[2])/((s+f+sum(prior))^2*(s+f+sum(prior)+1)))

  marginal_results = adply(n, 2, function(dat){
    data.frame(mean =  E_post(sum(dat[c(1,2,4)]), dat[3], jeffreys_prior),
               sd   = SD_post(sum(dat[c(1,2,4)]), dat[3], jeffreys_prior),
               method = "marginal")

  ignoring_results = adply(n, 2, function(dat){
    data.frame(mean =  E_post(dat[4], dat[3], jeffreys_prior) * .7,
                 sd = SD_post(sum(dat[c(1,2,4)]), dat[3], jeffreys_prior) * .7,
              method= "ignoring")
  result = rbind(marginal_results, ignoring_results)
  result$size = sz
ggplot(results, aes(x=mean)) + geom_histogram(bins=20) + facet_grid(method~size, scales = "free_x") + theme_bw()
ggplot(results, aes(x=sd)) + geom_histogram() + facet_grid(method~size) + theme_bw()

enter image description here enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.