# Are there any supervised learning methods that do NOT boil down to optimizing a loss function?

All of supervised learning methods I can think of amount to optimizing a loss function (RMSE, AIC, Cross-Entropy,...) against a labeled data set. One would think that "learning = optimizing loss functions".

Are there any learning methods that don't amount to an optimization problem?

• k nearest neighbors? – shimao Oct 14 '18 at 19:46
• @shimao: I have some doubts about that... How do we choose $k$ after all? – usεr11852 says Reinstate Monic Oct 14 '18 at 22:27
• @Alex: Can you please define what you mean by learning a bit more specifically? For example, why would we consider unsupervised learning as not an optimisation problem? Clustering, anomaly detection, blind signal separation, all of them quintessential unsupervised techniques, entail loss functions which we optimise against (eg.g. intra-cluster variability, reachability densities, mutual information, etc.). What is the element of "unsupervision" that distinguishes the two learning paradigms in your question regarding their connection to optimisation? (Fun question, +1) – usεr11852 says Reinstate Monic Oct 14 '18 at 22:48

In the dark ages before statistical methods were in widespread use, qualitative assessments and expert judgements were the only source of the kinds of risk assessment that supervised learning provided. Indeed, the "supervised learning" process was not a metaphor for optimization but the core task of the analyst: become familiar with the subject, learn from history and experiments where possible, and make reasoned analysis.

• Are you describing the Delphi method? (Dead piece of knowledge this one, but I remember it as it was part of one of the first ever courses I took, "Technology Systems"; being taught introductory Operational Research in CS by a lecturer who himself was a civil engineer by training; those were the days...) (+1) – usεr11852 says Reinstate Monic Oct 16 '18 at 23:57
• I wasn't describing the Delphi method in particular, although it certainly seems to fit this description. I just mean to say that qualitative assessments play an important role that is not necessarily completely superseded by statistics. For example, some hedge funds still operate by having some financial expert make an assessment of how the market might move in the future and taking positions based on that assessment. Social sciences make widespread use of qualitative methods (although there has been a shift in the last 20 years to use more quantitative research). – Sycorax says Reinstate Monica Oct 17 '18 at 1:36

In one sense, the answer is tautologically "no". This is because the point of supervised methods is to have optimal predictions, so they are all an attempt to minimize a predictive loss function.

However, not all methods are using optimization routines (i.e., L-BFGS, gradient descent, etc.) to achieve this. The most striking example is Bayesian methods, in which integration, rather than optimization, is used to make inference. Other examples are anything in which the solution is in closed form; linear regression and random forests are two such examples.

• Bayesian methods might not be used for inference at all. Maybe you mean estimation? – AdamO Oct 15 '18 at 13:37

Estimating equations solve systems of linear equations. While, you could take the antiderivative of the estimating function and say that's a loss function, they're not entirely theoretically or practically equivalent.

$$0 = \sum_{i=1}^n w_i (\theta X_i - Y_i))$$

For instance, it's true OLS minimizes $$\sum_{i=1}^n (Y_i - \beta \mathbf{X}_i)^2$$

But it also solves: $$0 = (\mathbf{X}^T\mathbf{X}^{-1})\mathbf{X}^T(Y - \mathbf{X}\beta)$$ and it turns out this gives us a nice interpretation of the regression coefficient.

Also Bayesian methods update subjective evidence with data. This also has some ties to optimization routines, but like EEs that is not theoretically or practically equivalent.