Why is the top result obtained using cosine similarity extremely close to 0 not the expected 1?
That implies complete orthogonality.
Data: 100k documents/rows with 2000 features(TF_IDF values of tokens, mostly zeroes/sparse)
Used NearestNearest neighbors with cosine similarity from this thread:
How to find nearest neighbors using cosine similarity for all items from a large embeddings matrix?
# import NearestNeighbors from sklearn.neighbors import NearestNeighbors n_neighbor = 10 model = NearestNeighbors(n_neighbors=n_neighbor, metric='cosine', algorithm='brute', n_jobs=-1) model.fit(df_tfidf) # let's check first 3 documents model.kneighbors(df_tfidf.iloc[:3, :]) Out: (array([[2.22044605e-16, 5.96041399e-01, 6.02695589e-01, 6.05811405e-01, 6.06062605e-01, 6.06394780e-01, 6.10842747e-01, 6.12047848e-01, 6.12241848e-01, 6.13224358e-01], [0.00000000e+00, 4.26139518e-01, 4.29424721e-01, 4.29424721e-01, 4.29424721e-01, 4.35292677e-01, 4.41962536e-01, 4.51103908e-01, 4.52376335e-01, 4.54416818e-01], [2.22044605e-16, 6.13395642e-01, 6.20716271e-01, 6.38328419e-01, 6.43845160e-01, 6.60639675e-01, 6.61006505e-01, 6.66453839e-01, 6.70172442e-01, 6.71674788e-01]]), array([[ 0, 40197, 7517, 41572, 40659, 95641, 15538, 36000, 3115, 36789], [ 1, 117889, 78484, 37726, 72102, 75169, 91649, 4410, 18514, 88808], [ 2, 80311, 63605, 40658, 18017, 33410, 14809, 8880, 2964, 157]], dtype=int64))
So we can see that the document is similar to itself - index matches. The cosine similarity for this top match is 0 not the expected 1.
Thus the vector appears completely transposed
Are the other 9 matches trustworthy - that is are they not similarly transposed?
Notably they are sorted in ascending order.
So the results come out my top match, then 10th match, 9th match, ... 2nd match
It seems like an inconsistency in the model displaying results.
Surely you would want the identical document to have a score of 1 not 0?
EDIT2: I rerun the model looking for 50 closest matches and got the exact same results for the first 10 matches and then 40 more increasing values.
So are the cosine similarity values here inverted - that is 1-real value?
EDIT PER Community Bot request: My question is why kneighbors gives similarity score of 0 for the top match not the expected 1 in the case of document itself.