Given a list of texts, annotated by topic (each text can have multiple topics), and the document-topic matrix output of an LDA (Latent Dirichlet Allocation), unsupervised model. For example:
Annotated topics by user, among M = 3 predetermined topics. Y if the user has annotated that the text belong to a topic.
Text ID | Topic 1 | Topic 2 | Topic 3 001 Y N N 002 Y N Y 003 Y Y N 004 N Y Y 005 Y N N
Text document matrix from LDA, set to find N = 4 topics
Text ID | Topic X | Topic Y | Topic W | Topic Z | 001 0.5 0 0.3 0.2 002 0.5 0 0 0.5 003 0.1 0.9 0 0 004 0 0.5 0.2 0.3 005 0 0.5 0 0.5
I would like to find two matrices. In the first MxN matrix, call it LDA_Given_Annotation, the value of the the element [1,X] should be the probability that LDA labeled as X topics annotated as topic 1 and so on.
In the first MxN matrix, call it Annotation_given_LDA, the value of the element [1,X] should be the probability that the user has annotated with topic 1 something that LDA has (stochastically) annotated X.
Does this algorithm exist already, and if so, what's its name? Google didn't help. If you can help, please keep in mind that probably, in order to visualise this with a heat map, cells should be normalised by the number of documents having a certain topic. Each row of LDA_Given_Annotation could be seen as a histogram by itself, and should be comparable to the other rows. Annotation_given_LDA the same, but column-wise.
The goal of this is to show in two different ways how accurate is the job of a human annotator against the latent topics.