3
$\begingroup$

I am working my way through LDA and I think I got they main idea of it. Please correct me if I am wrong. Given the Plate notation:

LDA model

The variables $\alpha$ and $\beta$ are Dirichlet distribution parameters. The variable $Z_{d,n}$ assigns observed word $W_{d,n}$ to topic $\phi_k$, which is a distribution over words. Variable $\theta_d$ is the document-specific topic distribution. Both distributions $\theta_d$ and $\phi_k$ are drawn from Dirichlet distributions.

Now, only $W_{d,n}$ is observed and can be "directly" calculated. My question: What exactly is inferred/calculated with e.g. Gibbs sampling, variational Inference and so on?


For instance: For a Gaussian Naive Bayes classifier one assumes that the likelihood of each feature is Gaussian. In other words each feature has a Gaussian distribution:

$$ P(x) = \frac{1}{{\sigma \sqrt {2\pi } }} e^{ \frac{ - ( {x - \mu } )^2} {2\sigma ^2 } } $$

To find this distribution $\sigma$ and $\mu$ have to be determined which is pretty straight forward.

However, plainly said: What Numbers do I determine for LDA?

$\endgroup$
  • $\begingroup$ How familiar are you with Gibbs sampling in general? $\endgroup$ – jerad Dec 7 '12 at 18:40
  • $\begingroup$ Unfortunately, I know just the very rough edges. Maybe, I should rephrase my question a little: Do I use Gibbs sampling and expectation propagation to find the distributions? And if so, what exactly do I need to find. Taking the simple Gaussian distribution example, I'd have to find $\sigma$ and $\mu$. $\endgroup$ – Karsten Dec 8 '12 at 9:47
4
$\begingroup$

The mean and variance of a Gaussian are the unknown parameters that specify that distribution in that case. Likewise, in topic modeling, you attempt to learn the unknown parameters of $K$ topics, where each topic is a multinomial distribution over words in the vocabulary. Thus, it is the parameters of each multinomial distribution (each topic) that you seek to infer.

Note that your learning algorithm will also output another set of multinomial paramaters that represent the distribution of topics for each document.

$\endgroup$
  • $\begingroup$ Ah, okay. I always had the notion that the parameters for the Dirichlet distribution had to be determined. However, that did not make sense to me since $\alpha$ and $\beta$ are set (at least for this version of LDA). Thank you very much! $\endgroup$ – Karsten Dec 8 '12 at 23:12
  • $\begingroup$ Well, it is possible to learn the hyperparameters of the Dirichlet distribution at the same time but they can be thought of as a "nuisance parameter" in any case. $\endgroup$ – jerad Dec 8 '12 at 23:43
  • $\begingroup$ @Karsten The process of determining $\alpha$ and $\beta$ is called "parameter estimation". $\endgroup$ – Sibbs Gambling Oct 14 '14 at 3:01
1
$\begingroup$

The LDA model infers the following:

  1. $\phi_k$ - Topics - which are $W$ dimensional discrete distributions.
  2. $\theta_d$ - per document topic proportions - which are $K$ dimensional discrete distributions.
  3. $z_{d,n}$ which is the topic association of the $n$th word in the $d$th document ($w_{d,n}$). $z_{d,n} \in \{1, 2, ... K\}$.

You can certainly write your own inference algorithm, or try one of these:

  1. Matlab Topic Modeling toolbox.

  2. gensim (Python)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.