0
$\begingroup$

I have a learning algorithm that classifies points as 0 or 1 (haven't settled on which one to implement yet). Of the points I classify as 1, I want to ensure that the number of points correctly classified as 1 is between 40% and 60% (or generalizing, between any such threshold). This would mean that I want the same threshold to exist for false positives, i.e. points I classify as 1 that should really be 0. How would I go about doing this? My first guess would be some modification to the loss function for the algorithm, but I'm unsure what the rigorous way is to approach this.

$\endgroup$
1
$\begingroup$

Off-the-cuff - potentially you can draw up a Sensitivity / Specificity graph (e.g. with the Caret package) - determine the specificity you want to achieve, determine the cut-off rule that was used in the classifier to achieve this point on the curve, and use this for your future models. Otherwise - you can indirectly try to achieve through the cost function. There are probably faster and better ways.

$\endgroup$
  • $\begingroup$ Hi, thanks for your input. Is there a general theory of sensitivity/specifity I could learn about in addition to this package (e.g could you link me a paper or wiki article)? I'd like to understand the theory, and I probably won't be implementing this in R. $\endgroup$ – Andrew Mar 5 '14 at 23:07
  • $\begingroup$ I believe that the discussion of the caret package (extensive documentation at numerous places) gives a good place to start. $\endgroup$ – user1885116 Mar 6 '14 at 8:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.