I have 50 products. For each product, I identify 3 related products (1 top related, 2 (partially related, least related)) using similarity measures. I want to compare the ranked list generated by my model (predicted) with the ranked list specified by the domain experts(ground truth).
For example, Product 1
- [3,2,1] --> ranked assigned by user (from most related to least related)
- [3,1,2] --> ranked predicted by the model.
Through reading, I found that I may use rank correlation based approaches such as Kendall Tau/Spearmen to compare the ranked lists. However, I am not sure if these approaches are suitable as my number of samples is low (4). Please correct me if i am wrong.
Another approach is to use Jaccard similarity (set intersection) to quantify the similarity between two ranked list. Then, I may plot histogram from the setbased_list (see below).
for index, row in evaluate.iterrows(): d= row['Id'] y_pred = [3,2,1,0] y_true = [row['A'],row['B'],row['C'],row['D']] sim = jaccard_similarity_score(y_true, y_pred) setbased_list.append(sim)
- Is my approach to the problem above correct?
- What are other approaches that I may use if I want to take into consideration the positions of elements in the list (weight-based)?