0
$\begingroup$

Typically, I see people calculate the perplexity/likelihood of a topic model against a held out set of documents, but is this always necessary/appropriate?

I'm thinking in particular of a use case in which we're modeling a full corpus, with no plans to use the model for prediction on new documents. In other words, the corpus is complete and self-contained, and the topic model is only for exploration and dimensionality reduction on the original corpus.

At least intuitively, this seems to be a case where overfitting isn't really possible/meaningful, so isn't it best to calculate the model likelihood/perplexity against the full training set of documents, and try to minimize that value?

$\endgroup$
8
  • 2
    $\begingroup$ Why do you model? If you are not predicting on a holdout sample, how do you guard against overfitting? $\endgroup$ Nov 16 '15 at 14:59
  • $\begingroup$ Well I guess this is fairly conceptual, but the whole question hinges on precisely that. We're modeling a complete corpus, for the purposes of dimensionality reduction and topic exploration. My question boils down to, if we have the complete data we want to model (not a sample), then is "overfitting" really bad? Is it even possible? Overfitting is typically an issue where you have a sample of some larger population, and worry about fitting noise in the sample you're using. But if you have the full population, is overfitting really possible? $\endgroup$
    – moustachio
    Nov 16 '15 at 15:02
  • 2
    $\begingroup$ Overfitting does not just mean your model is not generalizable. Overfitted models are often based on spurious relationships between your variables. You mention dimensionality reduction. Wouldn't you want to know that the reduced dimensions are meaningful? If not, what is the purpose of the dimensionality reduction? $\endgroup$
    – Erik
    Nov 16 '15 at 15:19
  • $\begingroup$ Ok these are all helpful pointers. I see that I should use the traditional holdout method, then minimize perplexity on the test set. The issue this raises for my use case is that I need to know the topic weights for all words in my corpus, but when training on a sample I'm not guaranteed to get that (i.e.relatively uncommon words might never make it into the model, if they only occur in test documents). Given this, is it appropriate to use the holdout method to pick my "best" number of topics, then retrain a new model on complete corpus to ensure I get the topic weights for all terms? $\endgroup$
    – moustachio
    Nov 16 '15 at 15:44
2
$\begingroup$

"but is this always necessary/appropriate?"

No, this is not always necessary. Many papers, e.g. "Improving Topic Models with Latent Feature Word Representations", use topic coherence and/or document clustering and/or document classification and/or information retrieval to compare topic models other than computing perplexity on held-out data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.