(n)DCG without true relevance scores

Question

I have a search engine that returns results with a normalized score on the scale 0.0 to 1.0. Higher score means higher relevancy to the input query.

The ranked output scores look like this, e.g. [1.0, 0.5, 0.1] vs [0.6, 0.5, 0.4] for k=3.

Assuming that these scores are indeed correct, I want to measure the quality of the results for a given query. The intuition is that if the collection contains very relevant results for a query, the aggregated score for the top k results should be high, whereas the score should be low if the collection has fewer good results.

Following that intuition, good results on the top ranks should be emphasized, whereas lower ranks are less important. DCG seems to be a good candidate as it weighs in the rank:

However, when looking at common implementations of (n)DCG, the output rankings are always compared against the true scores, for instance in scikit-learn.

Comparing against the true relevancy clearly makes sense for evaluating an information retrieval system against a manually labeled dataset. However, how does this fit into the DCG formula above? The formula only takes a single ranking into account.

So my questions are:

• How is DCG used to compare an output ranking against a "true" ranking?
• Does the DCG score as outlined above, for a single ranking, make sense to indicate the quality of the results for a query?
• Bonus question: can I use e.g. the (n)DCG implementation of scikit-learn without true scores?

Answer 1

Consider we have 3 articles with true relevance scores as [10, 7, 2], and our search engine gave them predicted scores as [0.4, 0.7, 0.2]. In this case, we are ranking the $2^{nd}$ article first, followed by the $1^{st}$ and then the $3^{rd}$ at last. In other words, the true relevance scores (each representing one article) are re-ordered by the search engine into [7, 10, 2].

The following answer will be based on this [7, 10, 2] result.

How is DCG used to compare an output ranking against a "true" ranking?

by $DCG \over idealDCG$ reference

For example, normalized DCG of [7, 10, 2] is DCG of [7, 10, 2] divided by DCG of [10, 7, 2]

Does the DCG score as outlined above, for a single ranking, make sense to indicate the quality of the results for a query?

Given that you said

The intuition is that if the collection contains very relevant results for a query, the aggregated score for the top k results should be high, whereas the score should be low if the collection has fewer good results.

I think you don't care the ordering, so my answer is No. You use DCG when you specifically care about the ordering. Note that the denominator of the DCG penalize the score ranked later, which means you have a higher DCG if you rank your results by the scores.

High DCG: [10, 7, 2] Low DCG: [7, 10, 2] Lowest DCG: [2, 7, 10]

Certainly the absolute score values also impact the DCG such that DCG of [10, 7, 2] is better than DCG of [3, 2, 1], but DCG of [2, 7, 10] is also worse than [10, 7, 2]. In this case, is DCG still your choice?

Bonus question: can I use e.g. the (n)DCG implementation of scikit-learn without true scores?

Yes for DCG. You can pass the same value for both y_true and y_score and this is equivalent to saying that your search engine always predicts the ordering correctly. But this is not the right way to use it.

Summary: DCG is designed for ordering.