I have a logistic regression model (fit via glmnet in R with elastic net regularization), and I would like to maximize the difference between true positives and false positives. In order to do this, the following procedure came to mind:
- Fit standard logistic regression model
- Using prediction threshold as 0.5, identify all positive predictions
- Assign weight 1 for positively predicted observations, 0 for all others
- Fit weighted logistic regression model
What would be the flaws with this approach? What would be the correct way to proceed with this problem?
The reason for wanting to maximize the difference between the number of true positives and false negatives is due to the design of my application. As part of a class project, I am building a autonomous participant in an online marketplace - if my model predicts it can buy something and sell it later at a higher price, it places a bid. I would like to stick to logistic regression and output binary outcomes (win, lose) based on fixed costs and unit price increments (I gain or lose the same amount on every transaction). A false positive hurts me because it means that I buy something and am unable to sell it for a higher price. However, a false negative doesn't hurt me (only in terms of opportunity cost) because it just means if I didn't buy, but if I had, I would have made money. Similarly, a true positive benefits me because I buy and then sell for a higher price, but a true negative doesn't benefit me because I didn't take any action.
I agree that the 0.5 cut-off is completely arbitrary, and when I optimized the model from step 1 on the prediction threshold which yields the highest difference between true/false positives, it turns out to be closer to 0.4. I think this is due to the skewed nature of my data - the ratio between negatives and positives is about 1:3.
Right now, I am following the following steps:
- Split data intto training/test
- Fit model on training, make predictions in test set and compute difference between true/false positives
- Fit model on full, make predictions in test set and compute difference between true/false positives
The difference between true/false positives is smaller in step #3 than in step #2, despite the training set being a subset of the full set. Since I don't care whether the model in #3 has more true negatives and less false negatives, is there anything I can do without altering the likelihood function itself?