Intuitive explanation behind the statistical interpretation of regularization I understand that using regularization is equivalent to finding a MAP estimate. I am wondering why using a Gaussian prior (for example) is better at preventing overfitting than using the uniform prior. I can somewhat follow the maths but I cannot form an intuitive reasoning for why this happens.
 A: Short answer: the improper uniform prior does not put more probability density close to $0$ and thus does not introduce regularization, while the Gaussian and the Laplace prior do that. Moreover, intuitively, the improper uniform prior does put more density mass in any point of the support of the parameter, making the MLE equal to the MAP.
Long answer and comment: I don't think it is always better to use regularization with respect to classical MLE estimate. For simplicity, let's focus on linear regression with gaussian errors.
A clear advantage of regularization is when you have the number of parameters $p$ larger than the number of observations $n$.
In such a case, due to the identifiability issue, the classical MLE is not well defined, while regularized versions can be computed (e.g. Lasso, Ridge regression).
The intuition is that the regularization introduces a hard constraint that makes the solution to the optimization problem unique.
Such a solution is equivalent to the MAP in the Bayesian paradigm (e.g. under Laplace, Gaussian prior). The intuition is that the prior does not put a hard constraint, but a soft constraint, assigning different probability densities to the different values of the parameter. Thanks to the soft constraint induced by the prior term the MAP is unique.
Regularization can be used to prevent overfitting also in the case of $p$ smaller than $n$. The idea is that putting some parameters equal to zero, or shrinking them to zero, prevents too complex/flexible models. This can produce a better joint estimate of the parameters in terms of MSE thanks to the fact that even if regularization introduces a bias it can allow to reduce the variance of the estimator. See Stein's paradox for an interesting example.
The same can be done under the Bayesian paradigm, where the posterior distribution (and so the MAP) is a compromise between your prior opinion and the likelihood (Bayes theorem).
