Normalised score for BM25

Question

BM25 provides a function that assigns a score that is a function of a query and a document. The score is computed for each document in the collection and can be used to rank documents, but this score is specific to the query and does not have an upper bound (e.g. there is no concept of some kind of 'ultimate' relevance).

Is it possible (from a theoretical perspective) to normalise this score, so that the ranking score is comparable across queries?

For example to compare whether the highest ranked document for query1 is a more relevant than the highest ranked document for query2.

Answer 1

The score that BM25 calculates is only usable to compare search results for a specific query to each other. It's not possible to transform that score to mean something independent of the query.

But there's one way to do something that might be OK in some cases. You decide if this works in your case:

Normalize each score, by dividing it with the sum of all scores (of say the top 10 results). Looking at score for the first hit, it now means: "Are there lots of other hits that also match this query?". If there are, the number will be low, else it will be high.

Raw BM25 query example (from an anonymized real query):

Query 1:
Result 1, score=0.5998919138986571
Result 2, score=0.5998919138986571
Result 3, score=0.5998919138986571
Result 4, score=0.4995426367770633
Result 5, score=0.4995426367770633
Result 6 score=0.0

Query 2:
Result 1, score=3.9278021306217763
Result 3, score=1.6993264645743775
Result 4, score=1.5989771874527836
Result 5, score=1.5989771874527836
Result 2, score=1.0994345506757204
Result 6, score=0.0

Normalized BM25 score:

Query 1:
Result 1, score=0.21434195725534308
Result 2, score=0.21434195725534308
Result 3, score=0.21434195725534308
Result 4, score=0.17848706411698534
Result 5, score=0.17848706411698534
Result 6, score=0.0

Query 2:
Result 1, score=0.3957675647605779
Result 3, score=0.17122509593204485
Result 4, score=0.1611138459985838
Result 5, score=0.1611138459985838
Result 2, score=0.11077964731020955
Result 6, score=0.0

As you might be able to see from the first query, that has "lower confidence", since there are many hits that get high scores for it.