3
$\begingroup$

The prolific combinatorialist Paul Erdős used to say that God keeps the most elegant proof of each theorem in "The Book". This inspired the following attempt to capture what some of the book might look like.

Currently I'm getting into machine learning, and I was wondering what people think would be the most elegant algorithms and proofs in this area that one should absolutely know (if for noting else than for their pure aesthetic appeal)? All sorts of suggestions are welcome, both classical results and cutting-edge concepts (though for the latter it may be tricky to evaluate their eventual relevance at this point).

$\endgroup$
  • $\begingroup$ about putting this on hold: here is a similar question that worked out well - why wouldn't that work here? $\endgroup$ – amakelov Apr 27 '17 at 0:31
3
$\begingroup$

I'll nominate the Lasso. Imposing an $\ell_1$ constraint or penalty on a linear regression forces some parameter estimates to be exactly zero. The proof is a simple geometrical argument about the shape of balls under the $\ell_1$ norm, e.g., Figure 2.2 in Statistical Learning with Sparsity by Hastie, Tibshirani & Wainwright (2015). Sparsity has all kinds of advantages, from regularization to easier interpretability.

Related is of course the LARS algorithm, which efficiently computes the entire parameter estimate path for various values of the shrinkage parameter $\lambda$.

| cite | improve this answer | |
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.