We know that loss (error) minimization originated with Legendre and Gauss in the early 18th c.
More recently Friedman* extolled its virtues for use in predictive modeling:
“The aim of regression analysis is to use the data to construct a function f(x) that can serve as a reasonable approximation of f(x) over the domain D of interest. The notion of reasonableness depends on the purpose for which the approximation is to be used. In nearly all applications however accuracy is important...if the sole purpose of the regression analysis is to obtain a rule for predicting future values of the response y, given values for the covariates (x1,...,xn), then accuracy is the only important virtue of the model...”
But is accuracy the only important virtue of a model?
My question concerns whether or not loss minimization is the only way to evaluate a regression function whether OLS, ML, gradient descent, quadratic or least absolute deviation, whatever.
Clearly, it's possible to evaluate multiple metrics beyond loss a posteriori, i.e., after models have been built. But are there multivariate functions which evaluate multiple metrics a priori or during the optimization phase?
For example, suppose one wanted to use a regression function which optimized fit based not just on loss minimization but which also maximized nonlinear dependence or Shannon entropy?
Are such functions available and/or have these issues been researched and published?
What software exists which implements these methods?
*Friedman, J.H., Multivariate Adaptive Regression Splines, The Annals of Statistics, 1991, vol. 19,No. 1 (March), pp. 1-67.