6
$\begingroup$

I am studying as a part of a class assignment a recent paper on Poisson factorization. Some points of the paper regarding the usage of some auxiliary variables are not clear to me. I would like to take advantage of your support to gain a deep understanding. This post is structured as follows. The context and approach of the cited paper are summarized first. The unclear point is then explicitly highlighted. Questions are asked finally.

Paper summarization

The idea behind the model is to recast the original Bayesian formulation of matrix factorization in terms of Poisson factorization, which is essentially the same approach apart from the exploitation of the Poisson distribution (rather than the Normal distribution) to model the probability of the observations. In the case of the cited paper, the applicative domain is a recommender system in which observations are implicit user ratings. We are given N users $u_1, \ldots, u_N$ and M items $i_1, \ldots, i_M$. User $u$ may rate item $i$. Thus, the rating $y_{ui}$ is a random variable that amounts to $1$ if $u$ rates $i$ and $0$ otherwise. $y_{ui}$ is determined in the paper through matrix factorization, i.e., by the interaction of $K$ latent factors. Accordingly, each user $u$ is modeled as a K-dimensional vector $\theta_u =\langle \theta_{u1}, \ldots, \theta_{uK}\rangle$ of latent-factor values, where the generic value $\theta_{uk}$ is sampled from a Gamma distribution with shape and rate parameters $a$ and $\epsilon$, respectively. More formally, $$\theta_{uk} \sim Gamma (a,\epsilon)$$ for each $k=1, \ldots, K$. Analogously, each item $i$ is modeled as a K-dimensional vector of latent-factor values $\beta_i =\langle \beta_{i1}, \ldots, \beta_{iK}\rangle$ such that $$\beta_{ik} \sim Gamma (c,\eta)$$ for each $k=1, \ldots, K$. The implicit ratings are the actual observations. The probability distribution over the generic rating $y_{ui}$ is $$ y_{ui} \sim Poisson(\theta_u^T\beta_i) $$.

Difficult point of the paper: introduction of the auxiliary variables and their manipulation

The probabilistic graphic model that describes the generative process incorporating the above distributions appears in Fig. 1 of the cited paper. There, it is said that some auxiliary random variables $$z_{uik} \sim Poisson(\theta_{uk}\beta_{ik})$$ are added to the model in to facilitate the derivation of the variational inference algorithm. For each user $u$ and each item $i$, such auxiliary random variables $z_{uik}$ are integers that sum to $y_{ui}$. The additivity of Poisson random variables ensures that the sum of the $z_{uik}$ is itself a Poisson whose rate is the sum of the rates of the individual auxiliary variables. In other words, $$y_{ui} = \sum_{k=1}^K z_{uik}$$

Questions

  1. It is not clear to me how to add the auxiliary variables $z_{ui}$ in the probabilistic graphic model of Figure 1.Could you please show how the expanded probabilistic graphic model would look like after the insertion of the auxiliary variables along with their respective dependencies?
  2. How do I show that such auxiliary variables $z_{uik}$ and their connections (i.e., dependencies) preserve the marginal distributions of the observations $y_{ui}$?
  3. Why does the addition of the auxiliary variables $z_{uik}$ make variational inference simpler?
  4. My class assignment also involves the derivation of the updates for the variational parameters from the complete conditional distributions of the latent model parameters, without resorting to partial derivatives. In particular, I am struggling to obtain the update in Eq. 9 for the variational parameters $\phi_{ui}$ from the complete conditional of the corresponding vector of Poisson counts $z_{ui}$ (as described in an interesting lecture). Because of a well known property, the complete conditional of $z_{ui}$ given (among others) $y_{ui}$ is multinomial. More precisely, $$z_{ui}| \beta,\theta,y_{ui} \sim Mult(y_{ui}, \mu_{ui} )$$ where $$\mu_{ui} = \langle \mu_{ui1}, \ldots, \mu_{uiK}\rangle$$ and the generic $\mu_{uik}$ is the below normalized rate $$ \mu_{uik} = \frac{\theta_{uk}\beta_{ik}}{\theta_{u1}\beta_{i1} + \ldots + \theta_{uK}\beta_{iK}} $$ To this point, the update rule in mean-field variational inference is to set the varitional parameter equal to the expected natural parameter under the variational posterior. In the case of the variational parameter $\phi_{ui}$, by rewriting the above Multinomial conditional over $z_{ui}$ in exponential form for convenience, it follows that $$q(z_{ui};\phi_{ui}) \propto e^{\sum_{k=1}^K E_{-z_{ui}} [log (\mu_{uik})]z_{uik}}$$ where $E_{-z_{ui}}[\cdot]$ stands for the expectation of the provided argument wrt to the variational posterior q without the factor over $z_{ui}$. By expanding the definition of $\mu_{uik}$, one finds that $$q(z_{ui};\phi_{ui}) \propto e^{\sum_{k=1}^K E_{-z_{ui}} [log (\theta_{uk}) + log(\beta_{ik}) - log(\sum_{k=1}^K\theta_{uk}\beta_{ik})]z_{uik}}$$ Starting from the above distribution, it is not clear to me how to find the update for the variational parameter $\phi_{ui}$ reported at Eq. 8 of a longer version of the paper (that includes some additional hints for the derivation of the variational inference algorithm). Can you please explain how to derive such an update?
$\endgroup$
  • $\begingroup$ Since this is an assignment this question should be tagged self-study. $\endgroup$ – Xi'an Jan 20 '16 at 19:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.