Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm searching for a combination of sensitivity and specificity cost function because i want have more weight for sensitivity ( sensitivity is more impotent for me rather than specificity). After searching i found this :

Final_Cost = ( (Cb/Cg)/( 1+(Cb/Cg) )*Bg + ( 1/( 1+(Cb/Cg) ) )*Gb

Cb is misclassification cost of positive and Cg is misclassification cost of negative. Bg is number of false positive detected and Ggis number of false negative detected. We should specify Cb/Cg. Is this a good function for calculating cost? Is there any other better functions?

share|improve this question
1  
You probably want something like the $F_\beta$ score. –  Marc Claesen Aug 31 at 9:23
    
Thank you Marc for your comment. Please add your answer with more informations. –  user2991243 Aug 31 at 9:25
    
+ How can i use that in my problem? Is this a good method for a neural network classification problem? Which F-score function should i use? –  user2991243 Aug 31 at 9:45

2 Answers 2

up vote 3 down vote accepted

Sensitivity and specificity are in backwards time order (i.e., use reverse conditioning or Prob(X|Y)). Hence they are not relevant to decision making. I recommend developing a well-calibrating direct probability model then using standard decision theory.

share|improve this answer
    
Thank you for your answer. Can you add more details or help documents for your proposed idea? –  user2991243 Aug 31 at 13:06
    
Come back after you have unbiasedly validated a probability (risk) model. See my handout at biostat.mc.vanderbilt.edu/CourseBios330, for example case study in logistic regression. –  Frank Harrell Aug 31 at 13:15

(1) The ratio of FP to FN is the standard way defining a cost function. It is build into some packages: C50 and rpart or part packages I think.
(2) It is rare that I see a reasonable use of cost functions in the machine learning field. Most use the F1 score or similar metrics. If you are working in this field, I'd spend sometime finding out what the expectations are. They might not be reasonable, but you should at least know what they are.
(3) I would be think about the problem at hand before trying to develop a well-calibrated model as suggested above - unless your model is likelihood based. Most machine learning algorithms don't naturally produce well-calibrated results - they need costs defined upfront - and imagining that the output you're getting is a probability is misleading. Though much smarter people - referenced below - seem to think calibrated models are feasible. So you might want to ignore my comments.

  • Steyerberg, E. W., van der Ploeg, T., & Van Calster, B. (2014). Risk prediction with machine learning and regression methods. Biometrical Journal. Biometrische Zeitschrift, 56(4), 601–606. doi:10.1002/bimj.201300297
  • Kruppa, J., Liu, Y., Diener, H.-C., Holste, T., Weimar, C., König, I. R., & Ziegler, A. (2014). Probability estimation with machine learning methods for dichotomous and multicategory outcome: applications. Biometrical Journal. Biometrische Zeitschrift, 56(4), 564–583. doi:10.1002/bimj.201300077
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.