How to interpret NearestNeighbor results obtained using cosine similarity for tf-idf vectors

Question

Why is the top result obtained using cosine similarity extremely close to 0 not the expected 1?

That implies complete orthogonality.

Data: 100k documents/rows with 2000 features(TF_IDF values of tokens, mostly zeroes/sparse)

Used NearestNearest neighbors with cosine similarity from this thread:

How to find nearest neighbors using cosine similarity for all items from a large embeddings matrix?

# import NearestNeighbors
from sklearn.neighbors import NearestNeighbors
n_neighbor = 10

model = NearestNeighbors(n_neighbors=n_neighbor,
                         metric='cosine',
                         algorithm='brute',
                         n_jobs=-1)
model.fit(df_tfidf)
# let's check first 3 documents
model.kneighbors(df_tfidf.iloc[:3, :])

Out:
(array([[2.22044605e-16, 5.96041399e-01, 6.02695589e-01, 6.05811405e-01,
         6.06062605e-01, 6.06394780e-01, 6.10842747e-01, 6.12047848e-01,
         6.12241848e-01, 6.13224358e-01],
        [0.00000000e+00, 4.26139518e-01, 4.29424721e-01, 4.29424721e-01,
         4.29424721e-01, 4.35292677e-01, 4.41962536e-01, 4.51103908e-01,
         4.52376335e-01, 4.54416818e-01],
        [2.22044605e-16, 6.13395642e-01, 6.20716271e-01, 6.38328419e-01,
     6.43845160e-01, 6.60639675e-01, 6.61006505e-01, 6.66453839e-01,
     6.70172442e-01, 6.71674788e-01]]),
 array([[     0,  40197,   7517,  41572,  40659,  95641,  15538,  36000,
           3115,  36789],
        [     1, 117889,  78484,  37726,  72102,  75169,  91649,   4410,
          18514,  88808],
        [     2,  80311,  63605,  40658,  18017,  33410,  14809,   8880,
           2964,    157]], dtype=int64))

So we can see that the document is similar to itself - index matches. The cosine similarity for this top match is 0 not the expected 1.

Thus the vector appears completely transposed

Are the other 9 matches trustworthy - that is are they not similarly transposed?

Notably they are sorted in ascending order.

So the results come out my top match, then 10th match, 9th match, ... 2nd match

It seems like an inconsistency in the model displaying results.

Surely you would want the identical document to have a score of 1 not 0?

EDIT2: I rerun the model looking for 50 closest matches and got the exact same results for the first 10 matches and then 40 more increasing values.

So are the cosine similarity values here inverted - that is 1-real value?

EDIT PER Community Bot request: My question is why kneighbors gives similarity score of 0 for the top match not the expected 1 in the case of document itself.

Answer 1

My hypothesis that model.kneighbors returns 1 - real_cos_similarity score appears correct.

I took a small 10 document x 2000 tf_idf score matrix from the original data and computed cosine similarity using

# import cosine_similarity
from sklearn.metrics.pairwise import cosine_similarity

# compute cosine similarity
cosine_sim = cosine_similarity(df_tfidf.iloc[:10, :])
cos_rounded = cosine_sim.round(2)


# convert kneighbors 
one_minus = (1 - kneighbors[0]).round(2)

print(one_minus[0])
# [1.  , 0.2 , 0.15, 0.08, 0.08, 0.06, 0.06, 0.05, 0.03, 0.02]

The results from one_minus makes sense now they are in sorted descending sequence now of similarity scores as expected.

print(cos_rounded[0])
# [1.  , 0.03, 0.08, 0.05, 0.15, 0.08, 0.02, 0.06, 0.06, 0.2 ]

Results from cos_rounded are as expected the scores are not sorted, first row has perfect match with itself.

# compare set of values in first row of one_minus with first row of  
cos_rounded

set(one_minus[0]) ==  set(cos_rounded[0])

Out: True

Only question is why does it happen that kneighbors returns 1 - real_cos_similarity?