3
$\begingroup$

I'm reading "The Indian Buffet Process: An Introduction and Review" by Griffiths and Ghahramani and wanted to confirm my understanding of one of the terms they use. On page 1188, they say that the "Discrete($\theta$) is the multiple-outcome analogue of a Bernoulli event. . ." It sounds like this is a Categorical distribution, which is a "generalized Bernoulli . . . or less precisely, a 'discrete distribution. . .'" The Wiki shows two parameters for Cat($K$,p). Additionally, the key words on the Wiki are "less precisely," and I'm not sure how to relate the single-parameter of the Discrete() to the two-parameters of the Categorical(). A layman's explanation would be helpful.

$\endgroup$
  • $\begingroup$ You are correct. In this case, the authors are using the two interchangeably. But I agree with wikipedia that this is less precise language -- could be confused with a distribution that has discrete outcomes like a Poisson distribution. $\endgroup$ – Alex Williams Feb 3 '16 at 0:37
  • 1
    $\begingroup$ categorical distributions are discrete, but discrete distributions are not necessarily categorical (unless you define categorical in a very particular way, at least) $\endgroup$ – Glen_b Feb 3 '16 at 1:40
  • $\begingroup$ @Glen_b How would I have to define categorical in order for discrete distributions to necessarily be categorical? $\endgroup$ – user2205916 Feb 3 '16 at 2:54
  • $\begingroup$ An example of an (implied) definition that seems to do this is found in the Wikipedia article Categorical variable, which says "Discretization is treating continuous data as if it were categorical"'; as phrased this seems to include (for example) taking a continuous uniform on (0,100) and producing a discrete uniform variable on (0,1,2,...,99) by recording the integer part of it. That would also be consistent with the discussion here. ... ctd $\endgroup$ – Glen_b Feb 3 '16 at 3:24
  • $\begingroup$ ctd... If you define categorical so that discrete uniform is categorical, the definition of categorical would seem to include discrete distributions. $\endgroup$ – Glen_b Feb 3 '16 at 3:28
2
$\begingroup$

This is a multinomial distribution with parameter $\mathbf{\theta}$. You are right that this is a categorical distribution per the Wiki page.

In page 1187, the paper mentions that $\mathbf{\theta}$ is a multinomial distribution over those $K$ classes. This means that the dimension of $\mathbf{\theta}$ is $1 \times K$. You can think of this as a line of $K$ seats, each of the seats have the probability to be 1 or 0. For example, the probability of seat 1 being 1 is $\theta_1$, the probability of seat 2 being 1 is $\theta_2$,...,the probability of seat $K$ being 1 is $\theta_K$.

When you understand how the distribution of $\mathbf{\theta}$ works. This section of the paper is discussing the finite mixture model. The "mixture model" means that the distribution is a combination of multiple components. In this paper, there are possible $K$ components. Each of the components have probability of $\theta_k$ to be included into this mixture model, $1 \leq k \leq K$. Every component is a distribution, such as a normal distribution with different mean and standard deviation. Normal distribution is the most frequent used distribution per my experiences. You can use other type of distribution if needed. Like discusses above,

$c_i|\theta \sim Discrete(\theta)$

is a method of choosing which component can be included. $\theta$ can be adjusted to increase or decrease the probability.

$\theta|\alpha \sim Dirichlet(\frac{\alpha}{K}, \frac{\alpha}{K}, \ldots, \frac{\alpha}{K})$

is the prior of $\theta$. The Dirichlet distribution is conjugate to the multinomial distribution. There are many resources online discussing the pattern of Dirichlet distribution corresponding to difference $\theta$. For example, page 4 on http://mayagupta.org/publications/FrigyikKapilaGuptaIntroToDirichlet.pdf This is a three dimensional Dirichlet distribution, that is $K =3$.

For example, the set up can be:

\begin{align*} x_i|c_i,\Theta &\sim N \bigg( \begin{bmatrix} \mu_1 \\ \mu_2 \\ \mu_3 \end{bmatrix}_{c_i}, \begin{bmatrix} S_{11} & S_{12} & S_{13} \\ S_{21} & S_{22} & S_{23} \\ S_{31} & S_{32} & S_{33} \end{bmatrix}_{c_i}^{-1} \bigg) \\ \begin{bmatrix} \mu_1 \\ \mu_2 \\ \mu_3 \end{bmatrix}_{c_i=1} &\sim N(\begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}, \begin{bmatrix} 1 & 1 & 1 \\ 1 & 1 & 1 \\ 1 & 1 & 1 \end{bmatrix}) \\ \begin{bmatrix} \mu_1 \\ \mu_2 \\ \mu_3 \end{bmatrix}_{c_i=1} &\sim N(\begin{bmatrix} 20 \\ 20 \\ 20 \end{bmatrix}, \begin{bmatrix} 1 & 1 & 1 \\ 1 & 1 & 1 \\ 1 & 1 & 1 \end{bmatrix}) \\ \begin{bmatrix} \mu_1 \\ \mu_2 \\ \mu_3 \end{bmatrix}_{c_i=1} &\sim N(\begin{bmatrix} 50 \\ 50 \\ 50 \end{bmatrix}, \begin{bmatrix} 1 & 1 & 1 \\ 1 & 1 & 1 \\ 1 & 1 & 1 \end{bmatrix}) \\ \begin{bmatrix} S_{11} & S_{12} & S_{13} \\ S_{21} & S_{22} & S_{23} \\ S_{31} & S_{32} & S_{33} \end{bmatrix}_{c_i=1} &\sim Wish_{p=2}(W_0=\begin{bmatrix} 10 & 1 & 1 \\ 1 & 10 & 1 \\ 1 & 1 & 10 \end{bmatrix}) \\ \begin{bmatrix} S_{11} & S_{12} & S_{13} \\ S_{21} & S_{22} & S_{23} \\ S_{31} & S_{32} & S_{33} \end{bmatrix}_{c_i=2} &\sim Wish_{p=2}(W_0=\begin{bmatrix} 10 & 1 & 1 \\ 1 & 10 & 1 \\ 1 & 1 & 10 \end{bmatrix}) \\ \begin{bmatrix} S_{11} & S_{12} & S_{13} \\ S_{21} & S_{22} & S_{23} \\ S_{31} & S_{32} & S_{33} \end{bmatrix}_{c_i=3} &\sim Wish_{p=2}(W_0=\begin{bmatrix} 10 & 1 & 1 \\ 1 & 10 & 1 \\ 1 & 1 & 10 \end{bmatrix}) \\ \boldsymbol{\pi}|\alpha &\sim Dir(\alpha_1, \alpha_2, \alpha_3) \\ c_i|\boldsymbol{\pi} &\sim Multi(\boldsymbol{\pi}) \end{align*}

Here is R code for this mixture of three normal distribution with Dirichlet prior:

#library(rgl)
threeDim = function(concen1, concen2, concen3) {
  stor = NULL
  S=1000
  for (i in 1:S){
    pi = rdirichlet(n=1, alpha=c(concen1, concen2, concen3))
    tag = rmultinom(1, 1, prob=pi)

    # parameter for x1
    sigma1 = matrix(c(1,1,1,1,1,1,1,1,1), ncol=3)
    wishSigma1 = matrix(c(10, 1, 1, 1, 10, 1, 1, 1, 10), ncol=3)
    mu1 = rmvnorm(n=1, mean=c(1, 1, 1), sigma=sigma1)
    phi1 = matrix(rWishart(1, 3, wishSigma1), ncol = 3)

    # parameter for x2
    sigma2 = matrix(c(1,1,1,1,1,1,1,1,1), ncol=3)
    wishSigma2 = matrix(c(10, 1, 1, 1, 10, 1, 1, 1, 10), ncol=3)
    mu2 = rmvnorm(n=1, mean=c(20, 20, 20), sigma=sigma2)
    phi2 =  matrix(rWishart(1, 3, wishSigma2), ncol = 3)

  # parameter for x3
    sigma3 = matrix(c(1,1,1,1,1,1,1,1,1), ncol=3)
    wishSigma3 = matrix(c(10, 1, 1, 1, 10, 1, 1, 1, 10), ncol=3)
    mu3 = rmvnorm(n=1, mean=c(50, 50, 50), sigma=sigma3)
    phi3 =  matrix(rWishart(1, 3, wishSigma3), ncol = 3)

    x1 = rmvnorm(n=1, mean=mu1, sigma=solve(phi1)) # sigma is covariance matrix 
    x2 = rmvnorm(n=1, mean=mu2, sigma=solve(phi2)) # sigma is covariance matrix 
    x3 = rmvnorm(n=1, mean=mu3, sigma=solve(phi3)) # sigma is covariance matrix 

    x = tag[1]*x1 + tag[2]*x2 + tag[3]*x3
    stor = rbind(stor, x)
  }
  return(stor)
}

Here are some of the plots regarding changing of the Dirichlet Prior. You will see how this change of prior can affect the Discrete($\theta$) and the component patterns.

enter image description here

enter image description here

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.