# How to compare a Ranked List?

I have 50 products. For each product, I identify 3 related products (1 top related, 2 (partially related, least related)) using similarity measures. I want to compare the ranked list generated by my model (predicted) with the ranked list specified by the domain experts(ground truth).

For example, Product 1

• [3,2,1] --> ranked assigned by user (from most related to least related)
• [3,1,2] --> ranked predicted by the model.

Through reading, I found that I may use rank correlation based approaches such as Kendall Tau/Spearmen to compare the ranked lists. However, I am not sure if these approaches are suitable as my number of samples is low (4). Please correct me if i am wrong.

Another approach is to use Jaccard similarity (set intersection) to quantify the similarity between two ranked list. Then, I may plot histogram from the setbased_list (see below).

for index, row in evaluate.iterrows():
d= row['Id']
y_pred = [3,2,1,0]
y_true = [row['A'],row['B'],row['C'],row['D']]
sim = jaccard_similarity_score(y_true, y_pred)
setbased_list.append(sim)

1. Is my approach to the problem above correct?
2. What are other approaches that I may use if I want to take into consideration the positions of elements in the list (weight-based)?
• If you want to prefer more items on top than those in the tail, try NDCG (normalized discount cumulative gain) metric. It is often used in recommendation systems. This metric makes the model try harder to get the recommendations correct on top and not spend too much "effort" to rank correctly the rankings that are in the bottom of the list. – Vladislavs Dovgalecs Feb 7 '18 at 1:28