# Alternatives to minimizing loss in regression

We know that loss (error) minimization originated with Legendre and Gauss in the early 18th c.

More recently Friedman* extolled its virtues for use in predictive modeling:

“The aim of regression analysis is to use the data to construct a function f(x) that can serve as a reasonable approximation of f(x) over the domain D of interest. The notion of reasonableness depends on the purpose for which the approximation is to be used. In nearly all applications however accuracy is important...if the sole purpose of the regression analysis is to obtain a rule for predicting future values of the response y, given values for the covariates (x1,...,xn), then accuracy is the only important virtue of the model...”

But is accuracy the only important virtue of a model?

My question concerns whether or not loss minimization is the only way to evaluate a regression function whether OLS, ML, gradient descent, quadratic or least absolute deviation, whatever.

Clearly, it's possible to evaluate multiple metrics beyond loss a posteriori, i.e., after models have been built. But are there multivariate functions which evaluate multiple metrics a priori or during the optimization phase?

For example, suppose one wanted to use a regression function which optimized fit based not just on loss minimization but which also maximized nonlinear dependence or Shannon entropy?

Are such functions available and/or have these issues been researched and published?

What software exists which implements these methods?

*Friedman, J.H., Multivariate Adaptive Regression Splines, The Annals of Statistics, 1991, vol. 19,No. 1 (March), pp. 1-67.

• I think your question is missing a crucial detail. Friedman's statement has the "if ... then" format. You ask about the "then" portion, but never state whether you are accepting the "if" portion as a premise. That is, are we assuming for the purpose of writing an answer as given that "the sole purpose of the regression analysis is to obtain a rule for predicting future values of the response $y$, given values for the covariates $(x_1,\dots ,x_n)$"? Because a model might forego that premise in exchange for, e.g., interpretability of some phenomenon. – Reinstate Monica Jul 30 at 17:53
• @Sycorax Apologies for any ambiguity. The question is more general than Friedman's "if". To your point, I'm interested in learning about 'model(s) that might forego that premise in exchange for, e.g., interpretability..." Thx. – user332577 Jul 30 at 20:25
• Gauss is surely not the sole claimant to developments of optimization in science. Gauss solved a very specific problem: least squares. I would think Lagrange would be very close because a) they were contemporaneous b) Lagrange worked on the same astronomical problem and c) he arguably did more than Gauss for optimization and mechanics. RA Fischer is arguably the father of statistical optimization. – AdamO Jul 31 at 16:33

Rational choice theory says that any rational preference can be modeled with a utility function.

Therefore any (rational) decision process can be encoded in a loss function and posed as an optimization problem.

For example, L1 and L2 regularization can be viewed as encoding a preference for smaller parameters or more parsimonious models into the loss function. Any preference can be similarly encoded, assuming it's not irrational.

For example, suppose one wanted to use a regression function which optimized fit based not just on loss minimization but which also maximized nonlinear dependence or Shannon entropy?

Then you would adjust your utility function to include a term penalizing those things, just as we did for L1/L2 regularization.

Now, this might make the problem computationally intractable; for example, 0/1 loss is known to result in an NP-hard problem. In general, people prefer to study tractable problems so you won't find much off-the-shelf software that does this, but nothing is stopping you from writing down such a function and applying some sufficiently generalized optimizer to it.

If you retort that you have a preference in mind which cannot be modeled by a loss function, even in principle, then all I can say is that such a preference is irrational. Don't blame me, that's just modus tollens from the above theorem. You are free to have such a preference (there is good empirical evidence that most preferences that people actually hold are irrational in one way or another) but you will not find much literature on solving such problems in a formal regression context.

• "People prefer to study tractable problems so you won't find much off-the-shelf software that does this" Such software is exactly what I'm looking for. Suggestions? Thx. – user332577 Jul 30 at 20:34
• It all depends on the form your modified loss function takes. If your modified loss function is still differentiable, you can use an stochastic gradient descent optimizer like adam, for which many implementations exist. If you don't have a gradient but believe the function is still smooth, you can use Bayesian optimization. And for really intractable metrics like accuracy (0/1 loss function,) you would need to do something like branch-and-bound. – olooney Jul 30 at 21:04

But is accuracy the only important virtue of a model?

The practical aspects of what a model's for is too nuanced for a theoretical discussion. Interpretation and generalizability come to mind. "Who will use this model?" should be a top line question in all statistical analyses.

Friedman's statement is defensible in a classical statistics framework: we have no fundamental reason to object to black-box prediction. If you want a Y-hat that's going to be very close to the Y you observe in the future, then "build your model as big as a house" as John Tukey would say.

This does not excuse overfitting, unless your question is ill defined [plural "you" being the proverbial statistician]. Overfitting is too often the result of analysts who encounter data rather than approach it. By going through hundreds of models and picking "the best", you are prone to build models that lack generality. We see it all the time.

Inadvertantly you also ask a different question: "Alternatives to minimizing loss in regression". Minimax estimators, estimators that minimize a loss function, are a subset of the class of method of moments estimators: estimators that give a zero to an estimating function. The goal of the MOMs is to find an unbiased estimator, rather than one that minimizes a loss.

• Thanks. "Alternatives" wasn't an inadvertent word, it is the point of the question. Apologies for any imprecision in specification. Based on your comment, it still sounds like MOM estimators optimize a single metric whether loss or unbiasedness, correct? I'm interested in hearing about functions that optimize multiple metrics. Thx. – user332577 Jul 30 at 20:29
• That's a pretty rule-bound theoretical rationale. For instance, optimize a single loss of any linear combination...why linear combinations? What about nonlinear combinations? Are you really claiming that there is absolutely no way to break out of the straitjacket you're imposing? – user332577 Jul 30 at 22:23
• Statements that something cannot be done imposes a rule that begs to be broken. I'm interested in learning about heuristics and workarounds to academic concerns. – user332577 Jul 31 at 15:44
• My rejoinder to your comment(s) remains clear enough to me. I'm neither an academic nor a PhD in statistics. Your responses are helpful while being limited, imho, by their academic and theoretical focus. My experience with PhD statisticians has frequently resulted in skepticism wrt claims of fool's errands that are rooted in appeals to theoretical precedent. Are you really stating that heuristics and workaround algorithms to my request for alternatives are impossible to develop? Frankly, I don't believe that, not for one minute. Thx again for your response and comments. – user332577 Jul 31 at 17:12