I have a logistic regression model. I'm looking for a non-graphical way to find the optimal cut-off where sensitivity is above a threshold(say 0.95) and maximizes sensitivity+specificity. I don't have a fitted model. Only two vectors of observations and predicted probabilities.
$\begingroup$
$\endgroup$
2
-
$\begingroup$ You can do it from the analysis of the AUC. The pair with maximum distance from the diagonal gives you the solution. You can add the constraints to select only a subset of points. $\endgroup$– user289381Commented Jul 25, 2020 at 13:56
-
1$\begingroup$ That approach is completely disconnected from the loss/cost/utility function you wish to optimize. The optimum decision has nothing to do with backwards probabilities (P(known | unknown)) such as sensitivity and specificity. $\endgroup$– Frank HarrellCommented Mar 23, 2021 at 2:26
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Already answered here (using the pROC package for R):
$\begingroup$
$\endgroup$
As I understand you need to do this automatically. You could do so by using the threshold that minimizes $FPR^{K_P} + FNR^{K_N}$ (using all possibilities values below a certain $FPR$ of your choice). For example $FPR^2 + FNR^2$ would equally (quadratically) penalize $FP$ and $FN$ error, while e.g. $FPR^3 + FNR^2$ would penalize $FP$ more drastically than $FN$. The larger $K_N$ and $K_P$, the closer $FPR$ and $FNR$ will stay together - while different $K_P$ and $K_N$ will account for your needs of more strongly minimizing $FPR$ or $FNR$.
Still, $K_N$ and $K_P$ need to be chosen beforehand. What exact values are suitable for your problem will depend on its details and your needs towards $FPR$ and $FNR$.
answered Jun 6, 2016 at 20:05