Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I can't find on the Internet a proper source that explains this.
I have built a search engine that for a particular query retrieves 5 relevant document out of the 10 relevant documents.
When I calculate the average precision, I sum the Precision@k, where k is relevant. At this stage, should I divide by 10 or by 5?
Average Precision should be divided by 10 and not 5.
Formula from Manning's Introduction to Information Retrieval
$$ MeanAveragePrecision(Q) = \frac{1}{|Q|} \sum_{j=1}^{|Q|}\frac{1}{m_j} \sum_{k=1}^{m_j} Precision(R_{jk}) $$
for query $q_j \in Q$ containing relevance documents ${d_1, ..., d_{m_j}}$ and $R_{jk}$ is the set of ranked results until document $d_k$. You are only looking for the Average Precision which is the inner sum. You can see that $\sum Precision$ divides by $m_j$ which is the total number of relevant documents.
Wikipedia's Average Precision page has the formula
$$ AveragePrecision = \frac {\sum_{k=1}^{n} P(k) \times rel(k)}{numberOfRelevantDocuments} $$ where $P(k)$ is precision@k and $rel(k)$ is an indicator function equaling 1 if the item at rank k is a relevant document, zero otherwise. It also says the "is over all relevant documents and the relevant documents not retrieved get a precision score of zero"
