# User queries and neural embeddings for Recommendations

Google's Machine Learning Course includes a section on Recommender systems with Deep Neural Networks.

In this architecture (diagram below) $$X$$ is meant to represent what they call a user query, which is transformed by hidden layers into a user embedding $$\Psi(X)$$. This user embedding then goes through a SoftMax layer with weights $$V$$ where each $$V_j$$ vector is presumably learned and represents the embedding of an item j.

• Is this based on any particular paper/s?
• How are user queries supposed to be built / encoded? Would it be e.g. one-hot encoding?
• How exactly would one add user features to it? (just a concatenation on the query vector?)

I'm not sure if they have in mind any particular paper, since there were multiple papers on this topic. If you go to next section of the online course, you'll find them mentioning the Deep Neural Networks for YouTube Recommendations paper by Covington et al (2016), another example of early papers is Neural Collaborative Filtering by He et al (2017). There's also a review paper Deep Learning based Recommender System: A Survey and New Perspectives by Zhang et al (2017).

What the linked tutorial describes, is the generalization of the matrix factorization model, as a neural network. The matrix factorization model approximates the matrix of ratings by $$n \gg d$$ users of $$k \gg d$$ items $$R_{n\times k}$$, by factorizing it into matrices $$P_{n\times d}$$ for users and $$Q_{d \times k}$$ for the items

$$R \approx PQ$$

At first sight, the only data you have, are the $$r_{ui}$$ ratings by users, but in fact we are working with the triples $$(R_{ui}, u, i)$$, since what we want to be able to do, is to make predictions for the $$(u,i)$$ pairs of indexes. You could build such model using the neural network building blocks available in popular deep learning frameworks (TensorFlow/Keras, PyTorch etc.), by creating explicit features for the indexes using one-hot encoding. Let's call those one-hot encoded matrices as $$U$$ and $$I$$ for simplicity. Now $$U$$ and $$I$$ are the features for our model, and we want to predict the ratings $$R$$. Our data would translate $$R_{ui}$$ to triples $$(R_j, U_j, I_j)$$ with indexes $$j$$ per each $$u,i$$ pair. The resulting data could be combined in rows like this:

rating | user                | item
3      | 0 0 0 ... 0 1 0 0 0 | 0 0 1 0 0 ... 0 0
5      | 1 0 0 ... 0 0 0 0 0 | 1 0 0 0 0 ... 0 0
1      | 0 1 0 ... 0 0 0 0 0 | 0 0 0 0 0 ... 0 1

Using the neural network building blocks, you could ask the neural network to come up with embeddings for the one-hot encoded features $$\psi(U)$$ and $$\phi(I)$$, then combine them using dot product as an aggregation function

$$R_j \approx \psi(U_j)^T \cdot \phi(I_j)$$

If the embeddings are created using a single-layer sub-network (the tutorial mentions multi-layer), this is exactly the same model as standard matrix factorization, we just re-shaped the data, and used embeddings layers to create $$P$$ and $$Q$$. However, you could ask, why would we stop in here? Why not building a model that is able to learn more complicated relations between the users and the items? Instead of using dot product, you could use a function $$f$$ that maps the embeddings to ratings

$$R_j \approx f\Big(\psi(U_j),\, \phi(I_j)\Big)$$

As you could guess, we can use neural network to learn such function. There is a number of ways to pass the features (including embeddings) into the network, where the most simple one is by concatenating them. Below, you can find a diagram from He et al (2017) paper showing such approach. The tutorial seems to describe similar architecture.

As about the "queries", what they mean are the features. The term "query" is quite popular, e.g. used for describing the attention layers in neural networks. As mentioned in the tutorial, those does not need to be only the one-hot features, you could include any other features as inputs for your network (e.g. demographics).

There's nice example of very similar architecture used for predicting movie popularity that uses such additional features as inputs and combines them to pass forward through the network. It is not a recommender system, but the architecture is very similar to the one that could be used to build one.

• Thanks - Very helpful! Although just for clarity, the model (diagram) that I included in the OP (and actual focus of this question does) not seem to follow that architecture, right? Instead the item features $V_j$ are understood to be weights of the softmax layer and learned directly. In other words, items (features or one-hot encodings) are not an input to the network, as in e.g. NCF. So from what I gather, there is no specific paper (at least that we are aware of) that used this particular approach?
– Josh
Jun 1 '20 at 16:40
• @Josh the indexes are one-hot encoded and used to create the embeddings, then we use upper levels of neural network to combine the embeddings by calculating the weights. The fact that they describe something like $\psi(X) V_j$ seems to suggest that they have in mind some kind of attention-based architecture (hence "queries" term in there) where $V_j$ are the attention weights, but their description is rather brief in here.
– Tim
Jun 1 '20 at 16:48