I have the following averaged $precision-recall$ curves with $4$ models. Which one is the best?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
If we suppose that all computation process is done correctly, you have the highest precision-recall in model 3 definitely.
-
4$\begingroup$ To be more precise, Model 3's PR curve dominates the other three. For any value of recall, Model 3 has the highest precision, and for any value of precision, Model 3 has the highest recall. There is no case where any model outperforms Model 3. If the curves looked like Models 1 and 2, in that they cross, there wouldn't be a universally best model. $\endgroup$ Sep 3, 2019 at 18:53
-
$\begingroup$ The only caveat here is that model 3 does not have any points where the recall is greater than ~.26. If this is just a result of not trying enough hyper-parameters, its not a big deal. But, if no point in the hyper-parameter space can achieve such a recall and if, based on the use case, such a recall is important/vital, then a different model (model 4 or 2 depending on the required recall) would be preferred. $\endgroup$ Sep 5, 2019 at 21:45
$\begingroup$
$\endgroup$
6
As there's no much difference given that the scale on the y-axis doesn't start from zero, the difference between the models in the graph seem exaggerated than normal.
It would be best to go by taking a decision of favoring recall vs precision. If you would rather have a false positive and try and catch more of the target(better recall), then Model 2(lowest precision, highest recall). If its the other way around, then Model 3.
-
$\begingroup$ Good catch on the y-axis but Model 2 does not necessarily have the highest recall--its graph just continues furthest to the right. If Model 3 were graphed that far, it may have higher precision and equally good recall. The curve is about the tradeoff between precision and recall... $\endgroup$ Sep 5, 2019 at 20:29
-
$\begingroup$ I thought that model 2 would be good for a max recall use case as its min(recall) and max(recall) are the the highest among all the models. $\endgroup$ Sep 5, 2019 at 20:32
-
$\begingroup$ That's not how a PR curve works. The curve is defined on the entire range of values of both precision and recall; it's a curve running from (0,1) to (1,0). The lines here are just parts of the full PR curves for each model. For example, Model 2 can get you a precision of 0.9 at recall 0.2, but Model 3 will get you a better precision (0.93) at the same recall (0.2), and is therefore superior at that level of recall. $\endgroup$ Sep 5, 2019 at 20:41
-
1$\begingroup$ I understand that. My point was just that its not always a case of choosing a model with highest precision. If the objective was to arrive at max recall without being bothered about the small difference in precision between the models, then model 2 has shown that it can give the most recall. Merely suggesting that it depends on the use case if you want to go for max recall vs max precision. Happy to learn if I was wrong. $\endgroup$ Sep 6, 2019 at 13:12
-
$\begingroup$ It's true that no point demonstrates better recall than that point on the line for Model 2. But here's a model with way better recall: just say everything is 1 always. Crappy model, but the recall's great! $\endgroup$ Sep 9, 2019 at 18:48