4
$\begingroup$

I'm searching for a combination of sensitivity and specificity cost function because i want have more weight for sensitivity ( sensitivity is more impotent for me rather than specificity). After searching i found this :

Final_Cost = ( (Cb/Cg)/( 1+(Cb/Cg) )*Bg + ( 1/( 1+(Cb/Cg) ) )*Gb

Cb is misclassification cost of positive and Cg is misclassification cost of negative. Bg is number of false positive detected and Ggis number of false negative detected. We should specify Cb/Cg. Is this a good function for calculating cost? Is there any other better functions?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ You probably want something like the $F_\beta$ score. $\endgroup$
    – Marc Claesen
    Aug 31, 2014 at 9:23
  • $\begingroup$ Thank you Marc for your comment. Please add your answer with more informations. $\endgroup$
    – user2991243
    Aug 31, 2014 at 9:25
  • $\begingroup$ + How can i use that in my problem? Is this a good method for a neural network classification problem? Which F-score function should i use? $\endgroup$
    – user2991243
    Aug 31, 2014 at 9:45

2 Answers 2

Reset to default
3
$\begingroup$

(1) The ratio of FP to FN is the standard way defining a cost function. It is build into some packages: C50 and rpart or part packages I think.
(2) It is rare that I see a reasonable use of cost functions in the machine learning field. Most use the F1 score or similar metrics. If you are working in this field, I'd spend sometime finding out what the expectations are. They might not be reasonable, but you should at least know what they are.
(3) I would be think about the problem at hand before trying to develop a well-calibrated model as suggested above - unless your model is likelihood based. Most machine learning algorithms don't naturally produce well-calibrated results - they need costs defined upfront - and imagining that the output you're getting is a probability is misleading. Though much smarter people - referenced below - seem to think calibrated models are feasible. So you might want to ignore my comments.

  • Steyerberg, E. W., van der Ploeg, T., & Van Calster, B. (2014). Risk prediction with machine learning and regression methods. Biometrical Journal. Biometrische Zeitschrift, 56(4), 601–606. doi:10.1002/bimj.201300297
  • Kruppa, J., Liu, Y., Diener, H.-C., Holste, T., Weimar, C., König, I. R., & Ziegler, A. (2014). Probability estimation with machine learning methods for dichotomous and multicategory outcome: applications. Biometrical Journal. Biometrische Zeitschrift, 56(4), 564–583. doi:10.1002/bimj.201300077
Improve this answer
$\endgroup$
5
$\begingroup$

Sensitivity and specificity are in backwards time order (i.e., use reverse conditioning or Prob(X|Y)). Hence they are not relevant to decision making. I recommend developing a well-calibrating direct probability model then using standard decision theory.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for your answer. Can you add more details or help documents for your proposed idea? $\endgroup$
    – user2991243
    Aug 31, 2014 at 13:06
  • $\begingroup$ Come back after you have unbiasedly validated a probability (risk) model. See my handout at biostat.mc.vanderbilt.edu/CourseBios330, for example case study in logistic regression. $\endgroup$
    – Frank Harrell
    Aug 31, 2014 at 13:15
  • $\begingroup$ I am not sure how useful Frank's comment is, the handout is 401 pages long! $\endgroup$
    – user918967
    Feb 21, 2017 at 5:51
  • 1
    $\begingroup$ Shorter: Chapter 19 of biostat.mc.vanderbilt.edu/tmp/bbr.pdf $\endgroup$
    – Frank Harrell
    Feb 21, 2017 at 17:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.