I've seen precision/recall curve here and there, but I don't understand how it works. Aren't they supposed to be fixed numbers based on classification results? For example, using the example from Wikipedia, if you identify 7 dogs from a video containing 9 dogs and some cats, and 4 of them are correct, then your recall is 4/9, and precision is 4/7. Right? How does it vary as to make a curve?
A typical classifier produces decision values instead of binary labels. Decision values are a numeric level of confidence that a given instance belongs to the positive class, typically higher implies more confidence. Take logistic regression as an example: its output is a probability, rather than a binary label.
Using these decision values you can create a ranking of your test set, from most confident to least confident. Based on a ranking, you can get a set of contingency tables by considering a certain number $n$ at the top of the ranking as positive and the rest as negative. Based on this set of contingency tables you can then compute a set of precision-recall pairs to obtain a PR curve.
Lets say we have a ranked test set based on decision value, ie. $(true\ label, decision)$-pairs where $dog$ is considered positive: $$(dog, 1.0),\ (cat, 0.95),\ (dog, 0.90),\ (dog, 0.80),\ (cat, 0.3),\ (cat, 0.2)$$
- If we label the top ranked as positive and the rest as negative, the corresponding contingency table is 1 TP, 0 FP, 2 FN and 3 TN $\rightarrow$ precision=1.0 and recall=1/3.
- If we label the top 2 ranked as positive and the rest as negative, the corresponding contingency table is 1 TP, 1 FP, 2 FN and 2 TN $\rightarrow$ precision=0.5 and recall=1/3.
- If we label the top 3 ranked as positive and the rest as negative, the corresponding contingency table is 2 TP, 1 FP, 1 FN and 2 TN $\rightarrow$ precision=2/3 and recall=2/3.
- If we label the top 4 ranked as positive and the rest as negative, the corresponding contingency table is 3 TP, 1 FP, 0 FN and 2 TN $\rightarrow$ precision=3/4 and recall=1.0.
The information conveyed by the PR curve seems non-clear at first. The secret lies in reading it on both axis at the same time. It is widely used in Information Retrieval systems where it is important to return as much as relevant & precise information from some database given a query.
Now, how each point is computed? There is always a parameter (it is not x axis, nor y axis!) whose value corresponds to a point on the curve (e.g. threshold). This threshold defines a cutoff point for a ranked list. For example, above the cutoff will be returned to user, below won't be shown. Based on what falls in the two groups (True Positives, False Positives, True Negatives and False Negatives), you compute Precision and Recall. For each parameter you compute Precision and Recall.