Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Can somebody explain what is the natural interpretation for LDA hyperparameters? ALPHA and BETA are parameters of Dirichlet distributions for (per document) topic and (per topic) word distributions respectively. However can someone explain what it means to choose larger values of these hyperparameters versus smaller values? Does that mean putting any prior beliefs in terms of topic sparsity in documents and mutual exclusiveness of topics in terms of words?

This question is about latent Dirichlet allocation, but the comment by BGReene immediately below refers to linear discriminant analysis, which confusingly is also abbreviated LDA.

share|improve this question
    
I think you need to give some more detail on which LDA formulation you are using. Generally it is only RDA models that have those parameters, LDA usually is defined entirely by mean vector, covariance matrix and prior probabilities. –  BGreene Sep 17 '12 at 9:18

2 Answers 2

up vote 5 down vote accepted

David Blei has a great talk introducing LDA to students of a summer class: http://videolectures.net/mlss09uk_blei_tm/

In the first video he covers extensively the basic idea of topic modelling and how Dirichlet distribution come into play. The plate notation is explained as if all hidden variables are observed to show the dependencies. Basically topics are distributions over words and document distributions over topics.

In the second video he shows the effect of alpha with some sample graphs. The smaller alpha the more sparse the distribution. Also, he introduces some inference approaches.

share|improve this answer

The answer depends on whether you are assuming the symmetric or asymmetric dirichlet distribution (or, more technically, whether the base measure is uniform). Unless something else is specified, most implementations of LDA assume the distribution is symmetric.

For the symmetric distribution, a high alpha-value means that each document is likely to contain a mixture of most of the topics, and not any single topic specifically. A low alpha value puts less such constraints on documents and means that it is more likely that a document may contain mixture of just a few, or even only one, of the topics. Likewise, a high beta-value means that each topic is likely to contain a mixture of most of the words, and not any word specifically, while a low value means that a topic may contain a mixture of just a few of the words.

If, on the other hand, the distribution is asymmetric, a high alpha-value means that a specific topic distribution (depending on the base measure) is more likely for each document. Similarly, high beta-values means each topic is more likely to contain a specific word mix defined by the base measure.

In practice, a high alpha-value will lead to documents being more similar in terms of what topics they contain. A high beta-value will similarly lead to topics being more similar in terms of what words they contain.

So, yes, the alpha-parameters specify prior beliefs about topic sparsity/uniformity in the documents. I'm not entirely sure what you mean by "mutual exclusiveness of topics in terms of words" though.


More generally, these are concentration parameters for the dirichlet distribution used in the LDA model. To gain some intuitive understanding of how this works, this presentation contains some nice illustrations, as well as a good explanation of LDA in general.


An additional comment I'll put here, since I can't comment on your original question: From what I've seen, the alpha- and beta-parameters can somewhat confusingly refer to several different parameterizations. The underlying dirichlet distribution is usually parameterized with the vector $(\alpha_1, \alpha_2, ... ,\alpha_K)$ , but this can be decomposed into the base measure $u = (u_1, u_2, ..., u_K)$ and the concentration parameter $\alpha$, such that $\alpha * \textbf{u} = (\alpha_1, \alpha_2, ... ,\alpha_K)$ . In the case where the alpha parameter is a scalar, it is usually meant the concentration parameter $\alpha$, but it can also mean the values of $(\alpha_1, \alpha_2, ... ,\alpha_K)$, since these will be equal under the symmetrical dirichlet distribution. If it's a vector, it usually refers to $(\alpha_1, \alpha_2, ... ,\alpha_K)$. I'm not sure which parametrization is most common, but in my reply I assume you meant the alpha- and beta-values as the concentration parameters.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.