3
$\begingroup$

I want to compare different ranking models using the NDCG metric. I average the NDCG of the models, then use the difference between the avg. NDCGs to compare the models.

My questions:

  1. What does it mean for a ranker to have 0.1 (avg.) NDCG more than another ranker? Can i say more than: "It ranks a little better"? Especially, how big a difference in NDCG would be "significant" in absolute terms?
  2. Can i say something about statistical significance of the difference of the average NDCG w.r.t the 0-Hypothesis? I'm quite unsure about whether normality assumption holds here... Especially since i'm not looking at the average of the differences, but the difference of the averages.
$\endgroup$

1 Answer 1

2
$\begingroup$

After you get an insight from the 2 means, then compare the 2 NCDG arrays (do not use the means), using for example wilcoxon signed-rank test to check whether the difference previously seen is relevant or not. In other words, you use the test paired by queries. The same holds if you were comparing P@K, MAP etc

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.