Search, rank and recommend in large text datasets

Question

Imagine you are Spotify and you have billions of songs. Assume that each of these songs are transcribed into text. How do you design your search and recommendation pipeline such that when somebody searches for a specific song, i.e. searches for a specific text from a song text, you can recommend songs to them according to relevance with the searched query? Basically rank the recommendations according to their relevance with some probability?

A naive solution is of course to use a text comparison algorithm like cosine similarity and compute this for all billions of songs and output the ones with the highest similarity with the computed probabilities which could be ridiculously slow and inaccurate. How is this done in practice?

Answer 1

A method known as non-negative matrix factorization (NMF) is ideal for recommending other songs and videos a user might like. You don't have to drill down into songs and use words in each song and run text mining or natural language processing (NLP) of any sort. Instead, you construct a user-by-song matrix, and in each row for a user, place the number of times the user listened to each song. Do this for all users and all songs.

Next, run NMF on the data and then for each user, for a particular song $A$ that was just listened to, recommend other songs $C,K,T,Y$ based on what other users listened to the most frequently, who also listened to song $A$.

Microsoft has a machine-learning worked example on NMF and music recommendations.