Logistic regression: maximizing true positives - false positives

Question

I have a logistic regression model (fit via glmnet in R with elastic net regularization), and I would like to maximize the difference between true positives and false positives. In order to do this, the following procedure came to mind:

1. Fit standard logistic regression model
2. Using prediction threshold as 0.5, identify all positive predictions
3. Assign weight 1 for positively predicted observations, 0 for all others
4. Fit weighted logistic regression model

What would be the flaws with this approach? What would be the correct way to proceed with this problem?

The reason for wanting to maximize the difference between the number of true positives and false negatives is due to the design of my application. As part of a class project, I am building a autonomous participant in an online marketplace - if my model predicts it can buy something and sell it later at a higher price, it places a bid. I would like to stick to logistic regression and output binary outcomes (win, lose) based on fixed costs and unit price increments (I gain or lose the same amount on every transaction). A false positive hurts me because it means that I buy something and am unable to sell it for a higher price. However, a false negative doesn't hurt me (only in terms of opportunity cost) because it just means if I didn't buy, but if I had, I would have made money. Similarly, a true positive benefits me because I buy and then sell for a higher price, but a true negative doesn't benefit me because I didn't take any action.

I agree that the 0.5 cut-off is completely arbitrary, and when I optimized the model from step 1 on the prediction threshold which yields the highest difference between true/false positives, it turns out to be closer to 0.4. I think this is due to the skewed nature of my data - the ratio between negatives and positives is about 1:3.

Right now, I am following the following steps:

1. Split data intto training/test
2. Fit model on training, make predictions in test set and compute difference between true/false positives
3. Fit model on full, make predictions in test set and compute difference between true/false positives

The difference between true/false positives is smaller in step #3 than in step #2, despite the training set being a subset of the full set. Since I don't care whether the model in #3 has more true negatives and less false negatives, is there anything I can do without altering the likelihood function itself?

Answer 1

You don't seem to want logistic regression at all. What you say is "I would like to maximize the difference between true positives and false positives." That is a fine objective function, but it is not logistic regression. Let's see what it is.

First, some notation. The dependent variable is going to be $Y_i$:
\begin{align} Y_i &= \left\{ \begin{array}{l} 1 \qquad \textrm{Purchase $i$ was profitable}\\ 0 \qquad \textrm{Purchase $i$ was un-profitable} \end{array} \right. \end{align}

The independent variables (the stuff you use to try to predict whether you should buy) are going to be $X_i$ (a vector). The parameter you are trying to estimate is going to be $\beta$ (a vector). You will predict buy when $X_i\beta>0$. For observation $i$, you predict buy when $X_i\beta>0$ or when the indicator function $\mathbf{1}_{X_i\beta>0}=1$.

A true positive happens on observation $i$ when both $Y_i=1$ and $\mathbf{1}_{X_i\beta>0}=1$. A false positive on observation $i$ happens when $Y_i=0$ and $\mathbf{1}_{X_i\beta>0}=1$. You wish to find the $\beta$ which maximizes true positives minus false positives, or: \begin{equation} max_\beta \; \sum_{i=1}^N Y_i\cdot\mathbf{1}_{X_i\beta>0} - \sum_{i=1}^N (1-Y_i)\cdot\mathbf{1}_{X_i\beta>0} \end{equation}

This is not an especially familiar objective function for estimating a discrete response model, but bear with me while I do a little algebra on the objective function: \begin{align} &\sum_{i=1}^N Y_i\cdot\mathbf{1}_{X_i\beta>0} - \sum_{i=1}^N (1-Y_i)\cdot\mathbf{1}_{X_i\beta>0}\\ = &\sum_{i=1}^N Y_i\cdot\mathbf{1}_{X_i\beta>0} - \sum_{i=1}^N \mathbf{1}_{X_i\beta>0} + \sum_{i=1}^N Y_i\cdot\mathbf{1}_{X_i\beta>0}\\ = &\sum_{i=1}^N Y_i\cdot\mathbf{1}_{X_i\beta>0} - \sum_{i=1}^N \mathbf{1}_{X_i\beta>0} + \sum_{i=1}^N Y_i\cdot\mathbf{1}_{X_i\beta>0} \\ & \qquad + \sum_{i=1}^N 1 - \sum_{i=1}^N 1 + \sum_{i=1}^N Y_i - \sum_{i=1}^N Y_i\\ = &\sum_{i=1}^N Y_i\cdot\mathbf{1}_{X_i\beta>0} + \sum_{i=1}^N (1-Y_i)(1-\mathbf{1}_{X_i\beta>0}) - \sum_{i=1}^N 1 + \sum_{i=1}^N Y_i \\ \end{align}

OK, now notice that the last two terms in that sum are not functions of $\beta$, so we can ignore them in the maximization. Finally, we have just shown that the problem you want to solve, "maximize the difference between true positives and false positives" is the same as this problem: \begin{equation} max_\beta \; \sum_{i=1}^N Y_i\cdot\mathbf{1}_{X_i\beta>0} + \sum_{i=1}^N (1-Y_i)(1-\mathbf{1}_{X_i\beta>0}) \end{equation}

Now, that estimator has a name! It is named the maximum score estimator. It is a very intuitive way to estimate the parameter of a discrete response model. The parameter is chosen so as to maximize the number of correct predictions. The first term is the number of true positives, and the second term is the number of true negatives.

This is a pretty good way to estimate a (binary) discrete response model. The estimator is consistent, for example. (Manski, 1985, J of Econometrics) There are some oddities to this estimator, though. First, it is not unique in small samples. Once you have found one $\beta$ which solves the maximization, then any other $\beta$ which makes the exact same predictions in your dataset will solve the maximization---so, infinitely many $\beta$s close to the one you found. Also, the estimator is not asymptotically normal, and it converges slower than typical maximum likelihood estimators---cube root $N$ instead of root $N$ convergence. (Kim and Pollard, 1990, Ann of Stat) Finally, you can't use bootstrapping to do inference on it. (Abrevaya & Huang, 2005, Econometrica) There are some papers using this estimator though---there is a fun one about predicting results in the NCAA basketball tournament by Caudill, International Journal of Forecasting, April 2003, v. 19, iss. 2, pp. 313-17.

An estimator that overcomes most of these problems is Horowitz's smoothed maximum score estimator (Horowitz, 1992, Econometrica and Horowitz, 2002, J of Econometrics). It gives a root-$N$ consistent, asymptotically normal, unique estimator which is amenable to bootstrapping. Horowitz provides example code to implement his estimator on his webpage.

Answer 2

There are several things wrong with that approach, including:

• Seeking a cutoff for a continuous probability
• Using an arbitrary cutoff of 0.5
• Assuming that the cost of a "false positive" and a "false negative" are the same for all subjects
• Using weights that are not fractional
• Using weights that are estimated
• Overriding maximum likelihood estimation
• Not utilizing optimum Bayes decision theory, which dictates that optimum decisions are based on full information (not on whether something exceeds something else) and utility/loss/cost functions

Answer 3

The best approach to achieving what you're trying to describe is probably to directly optimize the logistic regression parameters with an AUC loss function. The textbook "Statistical Methods in Diagnostic Medicine" by Zhou describes this method.

The AUC (area under the receiver operating characteristic curve-- or ROC) is roughly interpreted as the probability that a randomly sampled "case" has a higher marker value than a "control". This is a measure of model discrimination, or its ability to correctly classify the outcome. The ROC is a curve in the unit plane which shows the sensitivity versus 1 - specificity for all possible marker values (fitted outcomes) in a regression model.

By using the traditional formulation of the logistic regression model,

$$ \mbox{logit Pr}(Y = 1 | X) = \alpha + \beta X$$

with log odds ratios for model parameters, you can roughly define an AUC based loss function to obtain optimal parameters. Unlike likelihood based logistic regression, AUC regression is not regular and can converge to local maxima in the parameter space.

Answer 4

I would like to add something to the discussion on the maximum score estimator of Manski. In case somebody is interested in computing this estimator exactly, an algorithm in R using the Mixed Integer Programming formulation of the paper Florios, K., & Skouras, S. (2008). Exact computation of max weighted score estimators. Journal of Econometrics, 146(1), 86-91. (https://doi.org/10.1016/j.jeconom.2008.05.018) is available in the Homepage of Prof. Rohit Patra (http://users.stat.ufl.edu/~rohitpatra/Code/MSEcompute) and in Github (https://github.com/kflorios/maxscore-estimator-mip-inR). Both algorithms use the library CPLEX. Nevertheless, keep in mind that the computation of the Maximum Score estimator is an NP-hard problem, so instances with moderate sample size and moderate number of predictor variables can be solved exactly.