I understand the argument for how training with an L1/L2 regularizer is the same thing as finding the MAP estimate when the prior is Gaussian/Laplace. But there's a crucial difference. In Bayes' theorem, the prior must not be influenced by the data, while in practice ML people tend to tune the regularizer to maximize the validation score. This seems to contradict the Bayesian interpretation. In fact, it sounds closer to "Empirical Bayes". How would someone in both the ML and Bayesian communities respond to this?

  • 2
    $\begingroup$ I think when people mention this equivalence, they assume a fixed regularization parameter. Tuning the regularization parameter using validation data is similar to "tuning" your choice of prior using the validation data $\endgroup$ – Jonny Lomond Apr 30 '18 at 16:30
  • $\begingroup$ Possible duplicate of How is empirical bayes valid? $\endgroup$ – jld Apr 30 '18 at 16:56

If you have a hyperprior $P(\alpha)$ (where $\alpha$ is the regularization parameter), then $P(\alpha|x) = \frac{P(x|\alpha)P(\alpha)}{P(x)}$. If we make the assumption that $P(\alpha)$ is quite wide and nearly constant within a wide range of values, then $P(\alpha|x) \propto P(x|\alpha)$, so we can select our regularization based on whatever works best on the validation set.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.