0
$\begingroup$

I watched a video on coursera, everything went well until the following slide around 12'50''. enter image description here

I read it in other papers that to estimate latent variables say $\Phi$ we can draw a sample $Z'$ of the full collection by collapsed Gibbs sampling and take $p(\Phi|W,Z')$ as an approximation.

This video says we can use $p(\Phi|W,\hat{Z})$ where $\hat{Z}$ is an average of a few samples.

My question is, does it make sense to average the samples of $Z$? If I understood correctly $z_{dn}$ represent the topic index that is responsible for the $n$th word in the $d$th document. But for two independent samples, the indices of the topics are not guaranteed to be the same.

$\endgroup$
4
  • 2
    $\begingroup$ You should write the equations, and not post a screenshot, here stats.stackexchange.com/questions/346302/… @Glen_b explains why $\endgroup$
    – niandra82
    May 15, 2018 at 13:55
  • $\begingroup$ @niandra82 thanks but I dont see how this is necessary here, the screenshot looks rather clear to me, even if it's for users who rely on screen readers there's link to the video with a lecturer who explains it better than a screen reader. $\endgroup$
    – dontloo
    May 15, 2018 at 15:48
  • 1
    $\begingroup$ You have a (lot) higher reputation than me, so you are probably right. But I don't see why I should see the entire video to understand what $\theta$ or $Z$ are, for example. If you write the equations, you may be forced to explain what they represent. Just my opinion $\endgroup$
    – niandra82
    May 15, 2018 at 16:10
  • $\begingroup$ @niandra82 if you're saying the equations need more explanation then that's a different problem which can not be improved by simply retyping all the equations, surely I'll try to add more explanation if i have time. Those who are familiar with lda should basically know what I'm asking only by the title so the screenshot is there to provide a common page of notations. $\endgroup$
    – dontloo
    May 16, 2018 at 2:44

1 Answer 1

1
$\begingroup$

The LDA has the following three latent variables:

  1. $\phi_t$ - Topics - which are $W$ dimensional discrete distributions.
  2. $\theta_d$ - per document topic proportions - which are $T$ dimensional discrete distributions.
  3. $z_{d,n}$ which is the topic association of the $n$th word in the $d$th document ($w_{d,n}$). $z_{d,n} \in \{1, 2, ... T\}$.

Using collapsed Gibbs sampling, the topic allocations $z_{d,n}$ for words $w_{d,n}$ are sampled. Using these topic allocations, $\phi_t$ and $\theta_d$ can be estimated as follows:

$$ \phi_t(w) = \frac{n_j^{(w)}+\beta}{n_j^{(.)}+W\beta} \\ \theta_d(t) = \frac{n_j^{(d)}+\alpha}{n_.^{(d)}+T\alpha} $$

where,

  • $n_j^{(w)}:$ number of times $w^{th}$ word was assigned to topic $j$
  • $n_j^{(.)}:$ total number of words assigned to topic $j$
  • $n_j^{(d)}:$ number of times topic $j$ was assigned in $d^{th}$ document
  • $n_.^{(d)}:$ total number of topics assigned in $d^{th}$ document
  • $T:$ number of topics
  • $W:$ number of words in vocabulary
  • $\alpha, \beta:$ Dirichlet hyperparameters

Reference: Griffiths et. al., (2004)

$\endgroup$
2
  • $\begingroup$ Hi thank you that makes perfect sense, but actually I want to asked about the method in the video, how they average different samples of Z? If I understood correctly all the $n$s needed are computed based on one sample of the full collection $Z\in R^{W}$, the video mentioned about averaging different $Z$s and that's where I'm confused. $\endgroup$
    – dontloo
    May 18, 2018 at 3:11
  • $\begingroup$ @dontloo you are correct that all the $n$'s are based on the topic assignments for the entire collection of documents. You don't really average over different $Z$'s. If you look at the implementation of the collapsed Gibbs sampler by the authors, nowhere do they average over $Z$'s. $\endgroup$
    – kedarps
    May 18, 2018 at 15:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.