I have an embeddings matrix of a large no:of items - of around 100k, with each embedding vector length of 100. So a matrix of size 100k x 100;
From this, I am trying to get the nearest neighbors for each item using cosine similarity. I have tried following approaches to do that:
Using the cosine_similarity function from sklearn on the whole matrix and finding the index of top k values in each array. But I am running out of memory when calculating topK in each array
Using Pandas Dataframe
applyfunction, on one item at a time and then getting top k from that
similarity = df[embField].apply(lambda x: cosine_similarity(v1, x)) nearestItemsIndex = similarity.sort_values(ascending=False).head(topK) nearestItems = df[itemField].ix[nearestItemsIndex.index]
But this approach is taking around 6-7 secs per item, and is not really scalable.
As this should be a common case in recommendation systems, I am guessing there should be some existing algo to solve this on large data. But unfortunately I couldn't find it. Would be great if someone can help me point to any such algo.