23
$\begingroup$

I am working on a project where I want to extract some information about the content of a series of open-ended essays. In this particular project, 148 people wrote essays about a hypothetical student organization as part of a larger experiment. Although in my field (social psychology), the typical way to analyze these data would be to code the essays by hand, I'd like to do this quantitatively, since hand-coding is both labor-intensive and a bit too subjective for my taste.

During my investigations about ways to quantitatively analyze free response data, I stumbled upon an approach called topic modelling (or Latent Dirichlet Allocation, or LDA). Topic modeling takes a bag-of-words representation of your data (a term-document matrix) and uses information about the word co-occurrences to extract the latent topics of the data. This approach seems perfect for my application.

Unfortunately, when I've applied topic modeling to my data, I've discovered two issues:

  1. The topics uncovered by topic modelling are sometimes hard to interpret
  2. When I re-run my topic models with a different random seed, the topics seem to change dramatically

Issue 2 in particular concerns me. Therefore, I have a two related questions:

  1. Is there anything I can do in the LDA procedure to optimize my model fit procedure for interpretability and stability? Personally, I don't care as much about finding the model with the lowest perplexity and / or best model fit -- I mainly want to use this procedure to help me understand and characterize what the participants in this study wrote in their essays. However, I certainly do not want my results to be an artifact of the random seed!
  2. Related to the above question, are there any standards for how much data you need to do an LDA? Most of the papers I've seen that have used this method analyze large corpora (e.g., an archive of all Science papers from the past 20 years), but, since I'm using experimental data, my corpus of documents is much smaller.

I have posted the essay data here for anyone who wants to get his or her hands dirty, and I have pasted the R code I'm using below.

require(tm)
require(topicmodels)

# Create a corpus from the essay 
c <- Corpus(DataframeSource(essays))
inspect(c)

# Remove punctuation and put the words in lower case
c <- tm_map(c, removePunctuation)
c <- tm_map(c, tolower)

# Create a DocumentTermMatrix.  The stopwords are the LIWC function word categories
# I have a copy of the LIWC dictionary, but if you want to do a similar analysis,
# use the default stop words in tm
dtm <- DocumentTermMatrix(c, control = list(stopwords = 
  c(dict$funct, dict$pronoun, dict$ppron, dict$i, dict$we, dict$you, dict$shehe, 
    dict$they, dict$inpers, dict$article, dict$aux)))

# Term frequency inverse-document frequency to select the desired words
term_tfidf <- tapply(dtm$v/rowSums(as.matrix(dtm))[dtm$i], dtm$j, mean) * log2(nDocs(dtm)/colSums(as.matrix(dtm)))
summary(term_tfidf)

dtm <- dtm[, term_tfidf >= 0.04]

lda <- LDA(dtm, k = 5, seed = 532)
perplexity(lda)
(terms <- terms(lda, 10))
(topics <- topics(lda))

Edit:

I tried modifying nstart as suggested by Flounderer in the comments. Unfortunately, as shown below, even setting nstart to 1000 results in topics that vary quite dramatically from random seed to random seed. Just to emphasize again, the only thing I'm changing in the estimation of the two models below is the random seed used to start model estimation, and yet the topics do not seem to be at all consistent in these two runs.

lda <- LDA(dtm, k = 5, seed = 535, control = list(nstart = 1000))
(terms <- terms(lda, 10))

      Topic 1         Topic 2      Topic 3      Topic 4       Topic 5      
 [1,] "international" "ethnicity"  "free"       "credit"      "kind"       
 [2,] "communicate"   "true"       "team"       "mandatory"   "bridge"     
 [3,] "gain"          "asians"     "cooperate"  "music"       "close"      
 [4,] "use"           "hand"       "order"      "seen"        "deal"       
 [5,] "big"           "hold"       "play"       "barrier"     "designed"   
 [6,] "communication" "effective"  "big"        "stereotypes" "effort"     
 [7,] "america"       "emphasis"   "beginning"  "asians"      "implemented"
 [8,] "chinese"       "halls"      "china"      "fantastic"   "websites"   
 [9,] "ethnicity"     "minorities" "difference" "focusing"    "planned"    
[10,] "networks"      "population" "easier"     "force"       "body"

lda <- LDA(dtm, k = 5, seed = 536, control = list(nstart = 1000))
(terms <- terms(lda, 10))

      Topic 1       Topic 2         Topic 3        Topic 4       Topic 5    
 [1,] "kind"        "international" "issue"        "willing"     "play"     
 [2,] "easier"      "ethnicity"     "close"        "use"         "trying"   
 [3,] "gain"        "communication" "currently"    "hand"        "unity"    
 [4,] "websites"    "communicate"   "implemented"  "networks"    "decision" 
 [5,] "credit"      "bridge"        "particularly" "stereotypes" "gap"      
 [6,] "effort"      "america"       "credit"       "communicate" "normally" 
 [7,] "barriers"    "connection"    "fulfill"      "came"        "asians"   
 [8,] "effects"     "kind"          "grew"         "asians"      "created"  
 [9,] "established" "order"         "perspectives" "big"         "effective"
[10,] "strangers"   "skills"        "big"          "budget"      "prejudice"
$\endgroup$
  • 2
    $\begingroup$ Thank you for sharing your data! It was very interesting to look at. I don't have a good answer to your questions, but I do want to suggest some things. For Question 1, you can try adjusting the control parameters in the LDA function in the topicmodels package. In particular, you could try making nstart bigger. This is guaranteed to make your results more stable, because the LDA function will just run over and over again with different random seeds and then return the best result. Unfortunately, increasing nstart to, say, 1000 will make the algorithm do 1000 times more work (cont'd) $\endgroup$ – Flounderer Jul 5 '13 at 2:04
  • 1
    $\begingroup$ so it will be much slower. And there is no guarantee that it will be stable enough. Re: both questions, it seems to me that LDA is really designed to classify unseen documents when there is too much data for a human to process. For this, it is OK if the VEM algorithm only gives a "good enough" answer which can vary from one run to another. But for you, this is not desirable, and so LDA might not be the best choice. There are some excellent alternatives in the first few lectures of Shalizi's course here: stat.cmu.edu/~cshalizi/350 , for example, you could convert each (cont'd) $\endgroup$ – Flounderer Jul 5 '13 at 2:08
  • 2
    $\begingroup$ essay to a bag-of-words vector and then do a PCA on the results, then look for clusters. As for whether your corpus is big enough, to be honest I wouldn't be surprised if it is too big for VEM to give reliable results. Perhaps I am just bitter, but I spent a huge amount of time trying to get this method to work for another model by similar authors, and it was just completely inconsistent from run to run, even when using tiny examples. There aren't many papers which discuss choosing starting points for algorithms like this, as far as I know. $\endgroup$ – Flounderer Jul 5 '13 at 2:13
  • $\begingroup$ Flounderer, thank you so much for your input! It's a little disappointing for me to hear that there aren't more guidelines about LDA specifically, but I suppose that comes with the territory of an unsupervised method. I will try adjusting nstart and looking at the course website to see if either of those yield something useful. (BTW, if you insert your comments in an answer, I will vote it up. I would like to see if anyone else has advice before I accept anything, but I think your comments are more than sufficient to count as an answer). $\endgroup$ – Patrick S. Forscher Jul 5 '13 at 14:00
  • $\begingroup$ I feel your social-science gruntwork pain, Patrick, but I think your approach is wrong to begin with. If you want to use statistical tests, you will need to have humans code a portion of them to get classification error rates, have you (personally) done that? If so, then you will know what features are most prominent and you can design/pick a better algorithm. $\endgroup$ – Indolering Jul 7 '13 at 7:12
7
+50
$\begingroup$

For my own curiosity, I applied a clustering algorithm that I've been working on to this dataset.

I've temporarily put-up the results here (choose the essays dataset).

It seems like the problem is not the starting points or the algorithm, but the data. You can 'reasonably' (subjectively, in my limited experience) get good clusters even with 147 instances as long as there is some hidden topics/concepts/themes/clusters (whatever you would like to call).

If the data does not have well separated topics, then no matter whichever algorithm you use, you might not get good answers.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ @Siddharth.Gopal Thanks so much for the response! It's true that I would expect some overlap in the clusters given that all the participants are describing one hypothetical student organization (which we called "BadgerConnect"). So, in contrast to, for example, an application of topic modelling to papers from Science, where some of the topics are vastly different from paper to paper, the topics are all a little similar. However, it is true that some of the essays are in favor of BadgerConnect and some are written against BadgerConnect. $\endgroup$ – Patrick S. Forscher Jul 9 '13 at 21:49
  • $\begingroup$ It's also true that the essays vary widely in the kind of arguments they present and how the arguments are presented. I'd like to capture some of that variability, if possible. Do you have any idea whether catching some of these distinctions is possible (at the very least, the difference between essays in favor and essays against this hypothetical student program)? Also, were your clustering results stable when you used different random seeds? $\endgroup$ – Patrick S. Forscher Jul 9 '13 at 21:51
  • 1
    $\begingroup$ 1. If you are purely concerned about stability of the algorithm - try running the algorithm many times and choose the model with the highest likelihood. $\endgroup$ – Siddharth Gopal Jul 9 '13 at 22:31
  • 1
    $\begingroup$ (although stability seems like a secondary issue here). 2. Given your description of what you expect in terms of arguments and opinions, representing the essays as a bag-of-words is not a good idea in this context. Infact topic model might itself not be a good tool for this. I would suggest that you pick a few key-words that you're interested (like race,food,dorm etc) and try to analyze the sentiment of the sentence in which the word occurs. For e.g. have a look at here for a demo. $\endgroup$ – Siddharth Gopal Jul 9 '13 at 22:38
  • 1
    $\begingroup$ Python has an excellent NLP toolkit called nltk. You might want to have a look at what it offers. Regarding tf-idf, 'technically', the input to LDA should only be word-counts as the multinomial distribution is not defined for arbitrary real numbers. $\endgroup$ – Siddharth Gopal Jul 9 '13 at 22:44
10
$\begingroup$
  1. The notion of "topics" in so-called "topic models" is misleading. The model does not know or is not designed to know semantically coherent "topics" at all. The "topics" are just distributions over tokens (words). In other words, the model just capture the high-order co-occurrence of terms. Whether these structures mean something or not is not the purpose of the model.

  2. The "LDA" model has two parts (essentially all graphical models): a) model definition and b) an implementation of an inference algorithm to infer / estate model parameters. The thing you mentioned may or may not be the problem of "LDA" model but can be some bug / error / misconfig of the specific implementation you used (R package).

  3. Almost all implementations of "LDA" requires some randomization. And by the nature of inference algorithms (e.g., MCMC or variational inference), you'll get local minimum solutions or a distribution of many solutions. So, in short, what you observed is somehow expected.

Practical Suggestions:

  1. Try different R packages: For example, this package is done by David Blei's former graduate student. Or, even try another environment, such as this one. If you get similar results from all these stable packages, at least, you get reduce the problem a bit.

  2. Try playing a bit with not removing stop-words. The rationale is that, these stop-words play important role in connecting semantic meanings in such a small corpus (e.g., 100 or so articles). Also, try not filtering things.

  3. Try playing a bit with hyper-parameters, like different numbers of topics.

Papers about topic coherences:

  1. http://www.aclweb.org/anthology-new/D/D12/D12-1087.pdf

  2. http://people.cs.umass.edu/~wallach/publications/mimno11optimizing.pdf

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for your response. I'll respond to your comments one at a time. (1) I understand that the model knows nothing about topics, but your contention that the structures uncovered by topic models (and that whether these structures mean anything is not the purpose of these models) is directly contradictory to this review paper by David Blei, the creator of topic models. It seems to me that the purpose of topic models is exactly to help explore / characterize textual data, which is exactly what I want to do in this project. $\endgroup$ – Patrick S. Forscher Jul 9 '13 at 3:29
  • $\begingroup$ (2) Although it's possible that my results are due to a bug, I think it's more likely that they're due to some sort of error on my end (and if this does seem to be the case, please tell me!). I'm using the topicmodels package in R, which is essentially an R interface to the original algorithm implemented by Blei and colleagues. $\endgroup$ – Patrick S. Forscher Jul 9 '13 at 3:31
  • $\begingroup$ (3) I know that LDA requires some randomization, so I'm not asking for exact results from run to run, but I think it's reasonable to expect similar topics using different random seeds (indeed, I believe that this is a standard expectation when people use algorithms based on randomization). What I want to know is how to achieve said stability in results. $\endgroup$ – Patrick S. Forscher Jul 9 '13 at 3:33
  • $\begingroup$ @PatrickS.Forscher To your first comment: No. it's contradictory. The name "topics" are what we, as human beings, put labels on those distributions of words. These clustering structure may or may not be related to real-world, human readable topics. There're number of papers to introduce coherence into topic models to mitigate the exact problem. $\endgroup$ – Liangjie Hong Jul 9 '13 at 5:20
  • $\begingroup$ @PatrickS.Forscher To your second comment: Try some different implementation to see whether you get similar (unreasonable results) or not. For instance, UMASS Mallet. $\endgroup$ – Liangjie Hong Jul 9 '13 at 5:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.