# What's the relation between Matrix Factorization (MF) and Latent Dirichlet Allocation (LDA)?

My understanding is that both MF and LDA can be applied to do document classification. I will first summarize my understand about these two methods before I ask my questions.

Assuming we use a big matrix X to summarize the documents in a corpus and words in the vocabulary, where

X is a W x D matrix, and

$X_{ji}$ represents word-j's counts in doc-i,

D is the number of documents in the corpus,

W the number of words in the vocabulary.

# 1. Matrix Factorization

In its economical form,

$U$ is a $W \times K$ matrix,

$\Sigma$ a $K \times K$ matrix,

$V$ a $D \times K$ matrix

If we write $\Sigma \equiv S^2$, we can factorize $X$ into two matrices:

$X = U \Sigma V^T = U S^2 V^T = US S V^T \equiv A B^T$

# 2. Latent Dirichlet Allocation

Assume each document is a mixture of $\mathbb{K}$ topics, and each topic has its word distribution. Each word in a doc is drawn from one of the topics.

$$\pi_i | \alpha \sim Dir(\alpha \mathbb{1}_\mathbb{K}) \\ q_{il} | \pi_i \sim Cat(\pi_i) \\ b_k | \gamma \sim Dir(\gamma \mathbb{1}_W) \\ y_{il} | q_{il}=k, b_k \sim Cat(b_k)$$

where $\pi_i$ is doc-i's topic distribution,

$\alpha$ the user-specified parameter for the topic distribution (Dirichlet),

$\gamma$ the user-specified parameter for the topic-specific word distribution (also Dirichlet),

$q_{il} \in \{1, 2, \dots, \mathbb{K}\}$ the topic for the $l^{th}$ word in doc-i,

$y_{il} \in \{1, 2, \dots, W \}$ the identity for the $l^{th}$ word in doc-i,

$b_k$ the topic-specific word distributions, for $k=1, \dots, \mathbb{K}$,

$L_i$ the plate notation is the length of doc-i.

# Summary

In MF, we basically factorize $X$ into two matrices, $A$ and $B$. We can interpret one matrix as document-specific features and the other as word-specific features.

In LDA, we use the data to infer two distribution matrices:

$\Pi = \big(\pi_1, \pi_2, \dots, \pi_D\big),$

which is a $\mathbb{K} \times D$ matrix, with each column being the distribution over topics for a given doc. And

$\mathcal{B} = \big(b_1, b_2, \dots, b_\mathbb{K} \big),$

which is a $W \times \mathbb{K}$ matrix with each column being the topic-specific distribution over words in the vocabulary.

# Questions

1. Is there a way to formally interpret LDA as a form of matrix factorization? Namely, does that make sense to write X in terms of the product of $\mathcal{B}$ and $\Pi$ ?

2. As a common application in MF is to find the items that are similar to a target item. If there is indeed a correspondence between MF and LDA, how do I perform this similarity finding in LDA model?

• You may want to see this explanation by Wray Buntine: quora.com/… Nov 12, 2017 at 4:04