Yes, in fact this is the cross validation method of finding the number of topics. But note that you should minimize the perplexity of a held-out dataset to avoid overfitting.

It's worth noting that a non-parametric extension of LDA can derive the number of topics from the data without cross validation.

Yes, in fact this is the cross validation method of finding the number of topics. But note that you should minimize the perplexity of a held-out dataset to avoid overfitting.

It's worth noting that a non-parametric extension of LDA can derive the number of topics from the data without cross validation. Implementations exist at David Blei's lab github, but at the time of this writing I haven't see HDP LDA implemented in any mainstream, open-source ML libraries.

Caveat: Hierarchal LDA is different. It finds a hierarchy of topics, whereas hierarchal Dirichlet processes let you fit a potentially infinite number of flat topics. (The dual use of "hierarchy" can be confusing.)