Logistic regression: How can I penalize false positives different than false negatives during training? I developed a simple fraud detection example to test logistic regression. I have n features (e.g. credit score, account balance, etc.), m samples for training and I try to compute my output y with 0 - fraud or 1 - no fraud. Everything works well so far.
As next step, I would like to penalize the misclassification. It isn't great, but not the end of the world, that the classifier computes 0 and the actual result is 1. However, the case 1 - no fraud calculated and training output was 0 should be penalized higher than the other case (real world: costs of not detecting fraud is higher than blocking a good customers credit card).
I assume I have to adjust my gradient function to change my theta's more in one case or the other. Is that right? Or do I need to adjust my cost function?
I am using this cost function below
J(θ) = (1/m) * Sum(−y(i)*log(hθ(x(i))) − (1 − y(i))*log(1 − hθ(x(i))))

and this standard gradient descent approach
∂J(θ)/∂θ = (1/m) * Sum((hθ(x(i)) − y(i)) * x(i))

Can anybody recommend some literature how such a penalty would be applied?
 A: I don't feel is is appropriate to change the coefficients away from the maximum likelihood estimates.  These provide optimal estimates of $\beta$ that yield optimal estimates $\hat{P}(Y=1 | X=x)$.  Once you verify the calibration of your model you can set up any decision function and study its characteristics.  The optimal Bayes decision is that which maximizes expected utility/minimizes expected cost.  The cost/utility/loss function is defined separately from the model.  You can translate this to an action threshold for $\hat{P}$ that yields a subject-specific cutoff for $\hat{P}$.
A: The output of the logistic model is a calibrated probability (not a classification or pseudo-probability as produced by many machine learning algorithms) - thus you can change the cost simply by changing the probability cut-off used for classification. So instead of using the default 50% probability cut-off you can try any cut-off 0-100% probability. 


*

*It is often worth checking model calibration using a test set

*if you want to explore this visually, you can use the AUC curve and a family of cost lines. The gradient of the cost lines = (1-p)CFP/pCFN with p=prevalence. So if fraud 1% of time, cost FP to FN 1:5 then gradient = 19.8. The optimal point is the point of the AUC where a line with this gradient forms a tangent with curve. Relatively low prob cut-off with low sensitivity and high specificity

*In practice usually use crossvalidation or test data to optimize probability cut-point to a given ratio of FP/FN (I think there is a formula to derive this cut-point but can never find it when I'm looking for it)

