Consider forum posts or any text where we'd be interested in finding out related words, given the data. What would be a solution for creating a topic cluster based on this data? E.g. We are interested in finding a pre-determined topic "Tools" (e.g. Wrench, Screwdriver, ...) with the least possible effort. Is there any software/method available?
I understand a simple term frequency solution will go a long way (find out which words occur relatively often in the topic compared to how often the word occurs outside of documents).
Note that this is really a different problem than saying "we are trying to find 5 clusters based on LDA" or "let's classify these documents into 5 as distinct as possible clusters".
Is there any way not to have to come up with the "tools" ourselves?
I was thinking perhaps something semi automatic like:
- Start with 1 object belonging to the topic "Tools" (e.g. "Wrench")
- Sort all unique words for all documents containing the word wrench based on how often they occur in the documents in comparison with how often they occur in the whole corpus (these words might potentially be Tools)
- Take the first X (magic number) unique words and
- for these also consider newly "potential" documents, and reconsider which words might be belong to Tools
However, a severe limitation would be the fact that a story that might mention the word Wrench, will most likely not mention Screwdriver. So you definitely need a second step, where the words strongly correlating with Wrench might now be used instead to discover Screwdriver (if it would not stand out next to Wrench in documents)