Calculating sklearn's average precision by hand

Question

I'm trying to understand how sklearn's average_precision metric works. The reason I want to compute this by hand is to understand the details better, and to figure out why my code is telling me that the average precision of my model is the same as its roc_auc value (which doesn't make sense). The example they have is:

    Example
    >>> import numpy as np
    >>> from sklearn.metrics import average_precision_score
    >>> y_true = np.array([0, 0, 1, 1])
    >>> y_scores = np.array([0.1, 0.4, 0.35, 0.8])
    >>> average_precision_score(y_true, y_scores)  # doctest: +ELLIPSIS
    0.83...

How can I compute this by hand?

Answer 1

Late answer, but might be helpful for people who are looking into that at a later time, as I also had some trouble figuring that out. there is a formula on wikipedia and the scikit-learn docs that says the following:

$$\sum_n (R_n-R_{n-1})P_n$$

Now applying that to the example of yours:

Step 1: order the scores descending (because you want the recall to increase with each step instead of decrease):
y_scores = [0.8, 0.4, 0.35, 0.1]
y_true = [1, 0, 1, 0]

Step 2: calculate the precision and recall-(recall at n-1) for each threshhold. Note that the the point at the threshold is included, e.g. for threshold=0.35 the points that will be classified as 1 (positive) are [1, 0, 1]:
precision = [1, 1/2, 2/3, 2/4]
recall = [1/2, 1/2, 1, 1]
recall-(recall at n-1) = [1/2-0, 1/2-1/2, 1-1/2, 1-1] = [1/2, 0, 1/2, 0]

Step 3: build the sum for each index of precision and recall-(recall at n-1):
1*1/2+1/2*0+2/3*1/2+2/4*0 = 1/2+2/6 = 5/6 = 0.83...