I have a search engine that returns results with a normalized score on the scale 0.0 to 1.0. Higher score means higher relevancy to the input query.
The ranked output scores look like this, e.g. [1.0, 0.5, 0.1] vs [0.6, 0.5, 0.4] for k=3.
Assuming that these scores are indeed correct, I want to measure the quality of the results for a given query. The intuition is that if the collection contains very relevant results for a query, the aggregated score for the top k results should be high, whereas the score should be low if the collection has fewer good results.
Following that intuition, good results on the top ranks should be emphasized, whereas lower ranks are less important. DCG seems to be a good candidate as it weighs in the rank:
However, when looking at common implementations of (n)DCG, the output rankings are always compared against the true scores, for instance in scikit-learn.
Comparing against the true relevancy clearly makes sense for evaluating an information retrieval system against a manually labeled dataset. However, how does this fit into the DCG formula above? The formula only takes a single ranking into account.
So my questions are:
- How is DCG used to compare an output ranking against a "true" ranking?
- Does the DCG score as outlined above, for a single ranking, make sense to indicate the quality of the results for a query?
- Bonus question: can I use e.g. the (n)DCG implementation of scikit-learn without true scores?