# Logistic regression: maximizing true positives - false positives

I have a logistic regression model (fit via glmnet in R with elastic net regularization), and I would like to maximize the difference between true positives and false positives. In order to do this, the following procedure came to mind:

1. Fit standard logistic regression model
2. Using prediction threshold as 0.5, identify all positive predictions
3. Assign weight 1 for positively predicted observations, 0 for all others
4. Fit weighted logistic regression model

What would be the flaws with this approach? What would be the correct way to proceed with this problem?

The reason for wanting to maximize the difference between the number of true positives and false negatives is due to the design of my application. As part of a class project, I am building a autonomous participant in an online marketplace - if my model predicts it can buy something and sell it later at a higher price, it places a bid. I would like to stick to logistic regression and output binary outcomes (win, lose) based on fixed costs and unit price increments (I gain or lose the same amount on every transaction). A false positive hurts me because it means that I buy something and am unable to sell it for a higher price. However, a false negative doesn't hurt me (only in terms of opportunity cost) because it just means if I didn't buy, but if I had, I would have made money. Similarly, a true positive benefits me because I buy and then sell for a higher price, but a true negative doesn't benefit me because I didn't take any action.

I agree that the 0.5 cut-off is completely arbitrary, and when I optimized the model from step 1 on the prediction threshold which yields the highest difference between true/false positives, it turns out to be closer to 0.4. I think this is due to the skewed nature of my data - the ratio between negatives and positives is about 1:3.

Right now, I am following the following steps:

1. Split data intto training/test
2. Fit model on training, make predictions in test set and compute difference between true/false positives
3. Fit model on full, make predictions in test set and compute difference between true/false positives

The difference between true/false positives is smaller in step #3 than in step #2, despite the training set being a subset of the full set. Since I don't care whether the model in #3 has more true negatives and less false negatives, is there anything I can do without altering the likelihood function itself?

• Before asking what would be the flaws with this approach, maybe you should write why this approach should work, in your opinion. Why do you think that the steps 2-4 improve the outcome? – user31264 Oct 21 '13 at 14:40
• Also, am I right that at the end you drop the model from the step 1 and use only model from the step 4? – user31264 Oct 21 '13 at 14:47
• Yes, I was planning on using the model fitted with the entire data set, but it doesn't make sense to do so because it's under-performing the model fitted with the training set. – tmakino Oct 21 '13 at 15:50
• I don't have a source on this right now... but are you aware you can optimize a logistic regression model to maximize the Area Under The (Receiver Operating Characteristic) Curve (or AUC)? No need to reinvent the wheel. – AdamO Oct 21 '13 at 16:11
• What I don't quite understand here is why you haven't included anything about the predicted future price in your model, nor have you included the magnitude of profit/loss into the optimisation. Surely a decision to "buy" that leads to a 99% loss is much worse than a decision to "buy" that leads to a 1% loss, even though both are false positives. – probabilityislogic Oct 21 '13 at 20:04

You don't seem to want logistic regression at all. What you say is "I would like to maximize the difference between true positives and false positives." That is a fine objective function, but it is not logistic regression. Let's see what it is.

First, some notation. The dependent variable is going to be $Y_i$:
\begin{align} Y_i &= \left\{ \begin{array}{l} 1 \qquad \textrm{Purchase $i$ was profitable}\\ 0 \qquad \textrm{Purchase $i$ was un-profitable} \end{array} \right. \end{align}

The independent variables (the stuff you use to try to predict whether you should buy) are going to be $X_i$ (a vector). The parameter you are trying to estimate is going to be $\beta$ (a vector). You will predict buy when $X_i\beta>0$. For observation $i$, you predict buy when $X_i\beta>0$ or when the indicator function $\mathbf{1}_{X_i\beta>0}=1$.

A true positive happens on observation $i$ when both $Y_i=1$ and $\mathbf{1}_{X_i\beta>0}=1$. A false positive on observation $i$ happens when $Y_i=0$ and $\mathbf{1}_{X_i\beta>0}=1$. You wish to find the $\beta$ which maximizes true positives minus false positives, or: $$max_\beta \; \sum_{i=1}^N Y_i\cdot\mathbf{1}_{X_i\beta>0} - \sum_{i=1}^N (1-Y_i)\cdot\mathbf{1}_{X_i\beta>0}$$

This is not an especially familiar objective function for estimating a discrete response model, but bear with me while I do a little algebra on the objective function: \begin{align} &\sum_{i=1}^N Y_i\cdot\mathbf{1}_{X_i\beta>0} - \sum_{i=1}^N (1-Y_i)\cdot\mathbf{1}_{X_i\beta>0}\\ = &\sum_{i=1}^N Y_i\cdot\mathbf{1}_{X_i\beta>0} - \sum_{i=1}^N \mathbf{1}_{X_i\beta>0} + \sum_{i=1}^N Y_i\cdot\mathbf{1}_{X_i\beta>0}\\ = &\sum_{i=1}^N Y_i\cdot\mathbf{1}_{X_i\beta>0} - \sum_{i=1}^N \mathbf{1}_{X_i\beta>0} + \sum_{i=1}^N Y_i\cdot\mathbf{1}_{X_i\beta>0} \\ & \qquad + \sum_{i=1}^N 1 - \sum_{i=1}^N 1 + \sum_{i=1}^N Y_i - \sum_{i=1}^N Y_i\\ = &\sum_{i=1}^N Y_i\cdot\mathbf{1}_{X_i\beta>0} + \sum_{i=1}^N (1-Y_i)(1-\mathbf{1}_{X_i\beta>0}) - \sum_{i=1}^N 1 + \sum_{i=1}^N Y_i \\ \end{align}

OK, now notice that the last two terms in that sum are not functions of $\beta$, so we can ignore them in the maximization. Finally, we have just shown that the problem you want to solve, "maximize the difference between true positives and false positives" is the same as this problem: $$max_\beta \; \sum_{i=1}^N Y_i\cdot\mathbf{1}_{X_i\beta>0} + \sum_{i=1}^N (1-Y_i)(1-\mathbf{1}_{X_i\beta>0})$$

Now, that estimator has a name! It is named the maximum score estimator. It is a very intuitive way to estimate the parameter of a discrete response model. The parameter is chosen so as to maximize the number of correct predictions. The first term is the number of true positives, and the second term is the number of true negatives.

This is a pretty good way to estimate a (binary) discrete response model. The estimator is consistent, for example. (Manski, 1985, J of Econometrics) There are some oddities to this estimator, though. First, it is not unique in small samples. Once you have found one $\beta$ which solves the maximization, then any other $\beta$ which makes the exact same predictions in your dataset will solve the maximization---so, infinitely many $\beta$s close to the one you found. Also, the estimator is not asymptotically normal, and it converges slower than typical maximum likelihood estimators---cube root $N$ instead of root $N$ convergence. (Kim and Pollard, 1990, Ann of Stat) Finally, you can't use bootstrapping to do inference on it. (Abrevaya & Huang, 2005, Econometrica) There are some papers using this estimator though---there is a fun one about predicting results in the NCAA basketball tournament by Caudill, International Journal of Forecasting, April 2003, v. 19, iss. 2, pp. 313-17.

An estimator that overcomes most of these problems is Horowitz's smoothed maximum score estimator (Horowitz, 1992, Econometrica and Horowitz, 2002, J of Econometrics). It gives a root-$N$ consistent, asymptotically normal, unique estimator which is amenable to bootstrapping. Horowitz provides example code to implement his estimator on his webpage.

• Thank you for including the algebra to equate my cost function with the maximum score estimator. With the given indicator function for $\beta^Tx > 0$, does this mean that I will always classify $p>0.5$ as a positive and $p<=0.5$ as negative? Also, is p (the model output) calculated using the logistic function with input $\beta^T x$? The current approach I'm using is the AUC cost function, then optimizing on the prediction threshold to find the value with the highest difference between true positives and false positives. I understand that your answer explicitly finds the maximum difference – tmakino Oct 25 '13 at 14:28
• (continued) by defining it in the cost function (and fixing the prediction threshold at 0.5), thus skipping the intermediate step I've taken. However, the AUC already exists in the regression package I'm using (glmnet) while the maximum score esimator doesn't. Do you think my approach is reasonable given my objective? – tmakino Oct 25 '13 at 14:29
• Unfortunately, I'm not very familiar with the AUC method, so I can't say how appropriate it is here. In the maximum score estimator, there really isn't a $p$, because you are not assuming a logistic model. You are just deciding to predict 1 when $X_i\beta>0$ and then finding the best $\beta$. – Bill Oct 25 '13 at 19:23

There are several things wrong with that approach, including:

• Seeking a cutoff for a continuous probability
• Using an arbitrary cutoff of 0.5
• Assuming that the cost of a "false positive" and a "false negative" are the same for all subjects
• Using weights that are not fractional
• Using weights that are estimated
• Overriding maximum likelihood estimation
• Not utilizing optimum Bayes decision theory, which dictates that optimum decisions are based on full information (not on whether something exceeds something else) and utility/loss/cost functions
• Thank you, is there a way to achieve this while sticking to logistic regression (i.e. without touching the likelihood function)? – tmakino Oct 21 '13 at 12:41
• It depends on what "this" is. What is the ultimate goal and how will the model be used? – Frank Harrell Oct 21 '13 at 13:00
• I edited my question to provide detail on what I'm trying to achieve. – tmakino Oct 21 '13 at 16:03
• Unless I'm missing something, nothing you added would imply the use of a cutpoint. Note that a predicted probability provides its own error rate. – Frank Harrell Oct 21 '13 at 21:05

The best approach to achieving what you're trying to describe is probably to directly optimize the logistic regression parameters with an AUC loss function. The textbook "Statistical Methods in Diagnostic Medicine" by Zhou describes this method.

The AUC (area under the receiver operating characteristic curve-- or ROC) is roughly interpreted as the probability that a randomly sampled "case" has a higher marker value than a "control". This is a measure of model discrimination, or its ability to correctly classify the outcome. The ROC is a curve in the unit plane which shows the sensitivity versus 1 - specificity for all possible marker values (fitted outcomes) in a regression model.

By using the traditional formulation of the logistic regression model,

$$\mbox{logit Pr}(Y = 1 | X) = \alpha + \beta X$$

with log odds ratios for model parameters, you can roughly define an AUC based loss function to obtain optimal parameters. Unlike likelihood based logistic regression, AUC regression is not regular and can converge to local maxima in the parameter space.

• I would have thought AUC is not best here because there is small loss for false negative, but large loss for false positive. – probabilityislogic Oct 21 '13 at 19:50
• Well, the real problem is that OP has a continuous outcome (ROI) and is dichotomizing it as a loss/gain. But splitting hairs aside, with ROC regression in general "stupid" marker cut-off regions indeed count toward the AUC. You can use the partial AUC if you prespecify what counts as meaningful versus stupid marker values, and partial AUC regression has all the same performance capabilities (and issues). – AdamO Oct 21 '13 at 20:21

I would like to add something to the discussion on the maximum score estimator of Manski. In case somebody is interested in computing this estimator exactly, an algorithm in R using the Mixed Integer Programming formulation of the paper Florios, K., & Skouras, S. (2008). Exact computation of max weighted score estimators. Journal of Econometrics, 146(1), 86-91. (https://doi.org/10.1016/j.jeconom.2008.05.018) is available in the Homepage of Prof. Rohit Patra (http://users.stat.ufl.edu/~rohitpatra/Code/MSEcompute) and in Github (https://github.com/kflorios/maxscore-estimator-mip-inR). Both algorithms use the library CPLEX. Nevertheless, keep in mind that the computation of the Maximum Score estimator is an NP-hard problem, so instances with moderate sample size and moderate number of predictor variables can be solved exactly.