# Decision rule as a hyper-parameter in LASSO

I have a question that is related to the following: Is decision threshold a hyperparameter in logistic regression? but would like some clarification.

The general consensus is that the decision rule is not a hyper-parameter in the strictest sense that it won't affect the flexibility of the logistic regression.

I have a related question in the context of LASSO logistic regression. In the LASSO we optimize lambda and create a model, then we optimize the decision rule. My question is: wouldn't this be similar to optimizing say: mtry in a random forest and then using that static value of mtry and optimizing the number of trees after (which is the incorrect method)? What if we were to optimize lambda and arrive at lambda=0.01 and then optimize our decision rule at 0.6, however if we optimized them at the same time we may have arrived at a lambda=0.05 and a decision rule of 0.65 given us better performance.

I suppose the overall point and question is, in the face of optimizing hyper-parameters as well as the decision rule, is the decision rule a hyper-parameter in the strict sense?

Start with the decision rule in its ideal sense. It represents the tradeoff of costs between false-positive and false-negative classifications. In that sense the decision rule isn't a function of the data; it's a function of how you want to use your model of the data. In that sense it's not a hyper-parameter; it's a prior choice of a critical parameter value.

This report explains this relationship in Section 7. With 0 cost of true classification, say that the costs of false positives and false negatives are scaled to sum to 1. Call the cost of a false positive $$c$$ so that the cost of a false negative is $$(1-c)$$. Then the optimal probability classification cutoff for minimizing expected cost is at $$c$$.

When you specify a decision rule at 0.6, you are in effect specifying $$c = 0.6$$, saying that false positives are 1.5 (0.6/0.4) times as costly as false negatives. Changing the decision rule is just changing your estimate of the relative costs. So the decision rule in that sense represents your choice about how to use the data and your model, not something to be learned from the data independent of that choice.

This relationship is, however, based on having the true probability model in hand, notated as $$\eta(\boldsymbol{x})$$ as a function of the covarates $$\boldsymbol{x}$$ in the linked report. Instead, what you have is an estimated model, notated as $$q(\boldsymbol{x})$$. Section 7 of the above report states:

While $$\eta(\boldsymbol{x})$$ may not be well-approximated by the model $$q(\boldsymbol{x})$$, it may still be possible for each cost $$c$$ to approximate $$\{\eta(\boldsymbol{x})> c\}$$ well with $$\{q(\boldsymbol{x})> c\}$$, but each $$c$$ requiring a separate model fit $$q(.)$$.

So you wish to tune the parameters for the model fit $$q(.)$$ to come close to $$\eta(\boldsymbol{x})$$ in a sense that they have similar behaviors with respect to the (ideal) decision-rule value $$c$$. One way to do something like that is instead to find a cutoff probability value for the mis-specified model $$q(\boldsymbol{x})$$ to be other than $$c$$, say $$c^\dagger$$, to provide the desired model performance (e.g., accuracy) on your data. That is, you try to approximate $$\{\eta(\boldsymbol{x})> c\}$$ well with $$\{q(\boldsymbol{x})> c^\dagger\}$$ in a way that suits your purpose. I'll leave to others to decide whether one should call such a modification of a mis-specified model a "hyper-parameter" choice and, if so, whether that would be "in the strict sense."

One could argue that the choice of decision rule (in the first sense above) should instead be used to tune the modeling approach. A standard logistic regression, with coefficient values determined by maximum likelihood, represents only one of many ways to fit a linear model to data with binary outcomes. Its solution is equivalent to minimizing a log-loss function. Log-loss is a strictly proper scoring rule in the sense that it is optimized at the true probability distribution.

There is, however, a wide universe of strictly proper scoring rules from which one might choose; see Sections 2 and 3 of the report linked above. These rules differ in terms of their weighting along the probability scale. The log-loss rule puts high weight near the extremes. If you have a false-positive cost of $$c$$ in the above formulation, you might want instead to choose a scoring rule that puts more weight on probabilities around $$c$$.

The report linked above describes these issues extensively, and shows in Section 9 how to use iteratively weighted least squares to fit a linear model based on any proper scoring rule. This approach can be extended to penalization methods like LASSO; Section 15 of the report suggests that shrinkage of coefficients (as provided by LASSO and other penalization methods) can improve performance with some choices of weight function.

That said, I suspect that mis-specification of a linear model typically poses more of a problem than the choice of proper scoring rule in practical applications.

Optimizing your model near the probability cutoff associated with your choice of relative false positive/negative costs is nevertheless something to consider seriously. For example, that is the approach used in targeted maximum likelihood estimation, in which models are tuned to focus on a particular prediction region of interest. Combining multiple such models can minimize the dangers posed by any one model being mis-specified.

• What if the only thing I care about is maximizing accuracy (realizing that accuracy is not a proper scoring rule). If all I care about is maximizing accuracy, then false positives and false negatives have equal miss-classification costs. However, for a given data set, if I go about choosing the decision rule through cross validation as in a hyper-parameter, maximizing accuracy, I likely wouldn't end up with a threshold at 0.5, suggesting that the cost c is in fact a function of my data. I think this is where I am still a bit confused by your answer. Dec 20 '19 at 18:41
• @astel the choice of $c$ as the probability cutoff threshold is admittedly based on having the true probability model rather than a model one has estimated. Section 7 of the linked report extends the discussion to cutoffs based on probabilities estimated from a model. I have added a bit to the answer in that context, which is perhaps more in line with your idea of a decision rule as a hyper-parameter.
– EdM
Dec 20 '19 at 19:57