26
$\begingroup$

Can somebody explain what is the natural interpretation for LDA hyperparameters? ALPHA and BETA are parameters of Dirichlet distributions for (per document) topic and (per topic) word distributions respectively. However can someone explain what it means to choose larger values of these hyperparameters versus smaller values? Does that mean putting any prior beliefs in terms of topic sparsity in documents and mutual exclusiveness of topics in terms of words?

This question is about latent Dirichlet allocation, but the comment by BGReene immediately below refers to linear discriminant analysis, which confusingly is also abbreviated LDA.

$\endgroup$
1
  • $\begingroup$ I think you need to give some more detail on which LDA formulation you are using. Generally it is only RDA models that have those parameters, LDA usually is defined entirely by mean vector, covariance matrix and prior probabilities. $\endgroup$
    – BGreene
    Commented Sep 17, 2012 at 9:18

2 Answers 2

13
$\begingroup$

David Blei has a great talk introducing LDA to students of a summer class: http://videolectures.net/mlss09uk_blei_tm/

In the first video he covers extensively the basic idea of topic modelling and how Dirichlet distribution come into play. The plate notation is explained as if all hidden variables are observed to show the dependencies. Basically topics are distributions over words and document distributions over topics.

In the second video he shows the effect of alpha with some sample graphs. The smaller alpha the more sparse the distribution. Also, he introduces some inference approaches.

$\endgroup$
2
  • 1
    $\begingroup$ I guess you are right. I completely forgot that I wrote this. $\endgroup$
    – Karsten
    Commented Feb 27, 2015 at 14:50
  • $\begingroup$ oh! didnt expect to see a comment from the author! hehe :) $\endgroup$
    – samsamara
    Commented Mar 1, 2015 at 6:28
56
$\begingroup$

The answer depends on whether you are assuming the symmetric or asymmetric dirichlet distribution (or, more technically, whether the base measure is uniform). Unless something else is specified, most implementations of LDA assume the distribution is symmetric.

For the symmetric distribution, a high alpha-value means that each document is likely to contain a mixture of most of the topics, and not any single topic specifically. A low alpha value puts less such constraints on documents and means that it is more likely that a document may contain mixture of just a few, or even only one, of the topics. Likewise, a high beta-value means that each topic is likely to contain a mixture of most of the words, and not any word specifically, while a low value means that a topic may contain a mixture of just a few of the words.

If, on the other hand, the distribution is asymmetric, a high alpha-value means that a specific topic distribution (depending on the base measure) is more likely for each document. Similarly, high beta-values means each topic is more likely to contain a specific word mix defined by the base measure.

In practice, a high alpha-value will lead to documents being more similar in terms of what topics they contain. A high beta-value will similarly lead to topics being more similar in terms of what words they contain.

So, yes, the alpha-parameters specify prior beliefs about topic sparsity/uniformity in the documents. I'm not entirely sure what you mean by "mutual exclusiveness of topics in terms of words" though.


More generally, these are concentration parameters for the dirichlet distribution used in the LDA model. To gain some intuitive understanding of how this works, this presentation contains some nice illustrations, as well as a good explanation of LDA in general.


An additional comment I'll put here, since I can't comment on your original question: From what I've seen, the alpha- and beta-parameters can somewhat confusingly refer to several different parameterizations. The underlying dirichlet distribution is usually parameterized with the vector $(\alpha_1, \alpha_2, ... ,\alpha_K)$ , but this can be decomposed into the base measure $u = (u_1, u_2, ..., u_K)$ and the concentration parameter $\alpha$, such that $\alpha * \textbf{u} = (\alpha_1, \alpha_2, ... ,\alpha_K)$ . In the case where the alpha parameter is a scalar, it is usually meant the concentration parameter $\alpha$, but it can also mean the values of $(\alpha_1, \alpha_2, ... ,\alpha_K)$, since these will be equal under the symmetrical dirichlet distribution. If it's a vector, it usually refers to $(\alpha_1, \alpha_2, ... ,\alpha_K)$. I'm not sure which parametrization is most common, but in my reply I assume you meant the alpha- and beta-values as the concentration parameters.

$\endgroup$
3
  • 2
    $\begingroup$ +1 informative answer! I wanna ask how high/low is a high/low value for alpha and beta in general? $\endgroup$
    – samsamara
    Commented Jan 5, 2015 at 3:26
  • $\begingroup$ Beta is supposed to be a distribution over the words for each topic (a matrix), right? So how does a single value translates into a matrix? $\endgroup$
    – Noamiko
    Commented Apr 25, 2018 at 11:12
  • $\begingroup$ Am I right in concluding that high alpha means that documents are similar, and high beta means that topics are similar? $\endgroup$
    – Lewistrick
    Commented Oct 15, 2018 at 12:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.