How to compare a Ranked List?

I have 50 products. For each product, I identify 3 related products (1 top related, 2 (partially related, least related)) using similarity measures. I want to compare the ranked list generated by my model (predicted) with the ranked list specified by the domain experts(ground truth).

For example, Product 1

• [3,2,1] --> ranked assigned by user (from most related to least related)
• [3,1,2] --> ranked predicted by the model.

Through reading, I found that I may use rank correlation based approaches such as Kendall Tau/Spearmen to compare the ranked lists. However, I am not sure if these approaches are suitable as my number of samples is low (4). Please correct me if i am wrong.

Another approach is to use Jaccard similarity (set intersection) to quantify the similarity between two ranked list. Then, I may plot histogram from the setbased_list (see below).

for index, row in evaluate.iterrows():
d= row['Id']
y_pred = [3,2,1,0]
y_true = [row['A'],row['B'],row['C'],row['D']]
sim = jaccard_similarity_score(y_true, y_pred)
setbased_list.append(sim)

1. Is my approach to the problem above correct?
2. What are other approaches that I may use if I want to take into consideration the positions of elements in the list (weight-based)?
• If you want to prefer more items on top than those in the tail, try NDCG (normalized discount cumulative gain) metric. It is often used in recommendation systems. This metric makes the model try harder to get the recommendations correct on top and not spend too much "effort" to rank correctly the rankings that are in the bottom of the list. Feb 7 '18 at 1:28

Jaccard similarity will always return 1, as you will always have the same elements in both lists. Order does not matter as you are comparing the ratio of the intersection over the union of the sets.

Kendall's tau computes the pairwise difference between two sets. This is does not capture the "closeness" of the rankings.

[1,2,3,4] vs [2,1,3,4] is as wrong as [1,2,3,4] vs [4,2,3,1]. Clearly the first difference is closer to the truth than the latter.

Spearman's footrule is able to capture this relationship, see this explanation here for more details

Your approach with set similarity may be on the right track. Look up 'Rank Based Overlap' described in this paper

Webber, William, Alistair Moffat, and Justin Zobel. "A similarity measure for indefinite rankings." ACM Transactions on Information Systems (TOIS) 28.4 (2010): 1-38.

as well as this clear explanation

https://ragrawal.wordpress.com/2013/01/18/comparing-ranked-list/