I have a learning algorithm that classifies points as 0 or 1 (haven't settled on which one to implement yet). Of the points I classify as 1, I want to ensure that the number of points correctly classified as 1 is between 40% and 60% (or generalizing, between any such threshold). This would mean that I want the same threshold to exist for false positives, i.e. points I classify as 1 that should really be 0. How would I go about doing this? My first guess would be some modification to the loss function for the algorithm, but I'm unsure what the rigorous way is to approach this.
Off-the-cuff - potentially you can draw up a Sensitivity / Specificity graph (e.g. with the Caret package) - determine the specificity you want to achieve, determine the cut-off rule that was used in the classifier to achieve this point on the curve, and use this for your future models. Otherwise - you can indirectly try to achieve through the cost function. There are probably faster and better ways.