3
$\begingroup$

I'm trying to understand how sklearn's average_precision metric works. The reason I want to compute this by hand is to understand the details better, and to figure out why my code is telling me that the average precision of my model is the same as its roc_auc value (which doesn't make sense). The example they have is:

    Example
    >>> import numpy as np
    >>> from sklearn.metrics import average_precision_score
    >>> y_true = np.array([0, 0, 1, 1])
    >>> y_scores = np.array([0.1, 0.4, 0.35, 0.8])
    >>> average_precision_score(y_true, y_scores)  # doctest: +ELLIPSIS
    0.83...

How can I compute this by hand?

$\endgroup$

1 Answer 1

3
$\begingroup$

Late answer, but might be helpful for people who are looking into that at a later time, as I also had some trouble figuring that out. there is a formula on wikipedia and the scikit-learn docs that says the following:

$$\sum_n (R_n-R_{n-1})P_n$$

Now applying that to the example of yours:

Step 1: order the scores descending (because you want the recall to increase with each step instead of decrease):
y_scores = [0.8, 0.4, 0.35, 0.1]
y_true = [1, 0, 1, 0]

Step 2: calculate the precision and recall-(recall at n-1) for each threshhold. Note that the the point at the threshold is included, e.g. for threshold=0.35 the points that will be classified as 1 (positive) are [1, 0, 1]:
precision = [1, 1/2, 2/3, 2/4]
recall = [1/2, 1/2, 1, 1]
recall-(recall at n-1) = [1/2-0, 1/2-1/2, 1-1/2, 1-1] = [1/2, 0, 1/2, 0]

Step 3: build the sum for each index of precision and recall-(recall at n-1):
1*1/2+1/2*0+2/3*1/2+2/4*0 = 1/2+2/6 = 5/6 = 0.83...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.