3
$\begingroup$

The prolific combinatorialist Paul Erdős used to say that God keeps the most elegant proof of each theorem in "The Book". This inspired the following attempt to capture what some of the book might look like.

Currently I'm getting into machine learning, and I was wondering what people think would be the most elegant algorithms and proofs in this area that one should absolutely know (if for noting else than for their pure aesthetic appeal)? All sorts of suggestions are welcome, both classical results and cutting-edge concepts (though for the latter it may be tricky to evaluate their eventual relevance at this point).

$\endgroup$

closed as primarily opinion-based by SmallChess, Firebug, Nick Cox, Carl, whuber Apr 25 '17 at 22:06

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ about putting this on hold: here is a similar question that worked out well - why wouldn't that work here? $\endgroup$ – amakelov Apr 27 '17 at 0:31
3
$\begingroup$

I'll nominate the Lasso. Imposing an $\ell_1$ constraint or penalty on a linear regression forces some parameter estimates to be exactly zero. The proof is a simple geometrical argument about the shape of balls under the $\ell_1$ norm, e.g., Figure 2.2 in Statistical Learning with Sparsity by Hastie, Tibshirani & Wainwright (2015). Sparsity has all kinds of advantages, from regularization to easier interpretability.

Related is of course the LARS algorithm, which efficiently computes the entire parameter estimate path for various values of the shrinkage parameter $\lambda$.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.