How to find nearest neighbors using cosine similarity for all items from a large embeddings matrix? I have an embeddings matrix of a large no:of items - of around 100k, with each embedding vector length of 100. So a matrix of size 100k x 100;
From this, I am trying to get the nearest neighbors for each item using cosine similarity. I have tried following approaches to do that:


*

*Using the cosine_similarity function from sklearn on the whole matrix and finding the index of top k values in each array. But I am running out of memory when calculating topK in each array

*Using Pandas Dataframe apply function, on one item at a time and then getting top k from that 
similarity = df[embField].apply(lambda x: cosine_similarity(v1, x))
nearestItemsIndex = similarity.sort_values(ascending=False).head(topK)
nearestItems = df[itemField].ix[nearestItemsIndex.index]

But this approach is taking around 6-7 secs per item, and is not really scalable.
As this should be a common case in recommendation systems, I am guessing there should be some existing algo to solve this on large data. But unfortunately I couldn't find it. Would be great if someone can help me point to any such algo.
 A: Actually, we can use cosine similarity in knn via sklearn.
The source code is here.
This works for me:
model = NearestNeighbors(n_neighbors=n_neighbor,
                         metric='cosine',
                         algorithm='brute',
                         n_jobs=-1)
model.fit(user_item_matrix_sparse)

A: one weak point is sorting, and second is creating new collection
so you could just make another collection (list) and then keep topK items until some point this way instead of sorting everytnig and allocate new portion of memory you could do it more robust
use this https://stackoverflow.com/questions/27672494/how-to-use-bisect-insort-left-with-a-key
# import my_insort_left

topK_items = [] # (index, value)
for i, row in df.iteritems():
    res = cos_sim(v1, row)
    my_insort_left(topK_items, (i, res), keyfunc=lambda v: v[1])
    topK_items.pop(-1)
# topK_items is the result
nearest_items = [k for k, v in topK_items]

*keep in mind that I wrote it without testing so it may have some tiny bugs
A: A Nearest Neighbours model is fairly fast to build, because the algorithm uses the triangle inequality. Sadly, Scikit-Learn's ball tree does not support cosine distances, so you will end up with a KDTree, which is less efficient for high-dimensional data.
from sklearn.neighbors import NearestNeighbors

embeddings = get_embeddings(words)
tree = NearestNeighbors(
    n_neighbors=30, algorithm='ball_tree',
    metric='cosine')
tree.fit(X)

