Is MLE with regularization a bayesian method? It is usually said that priors on bayesian statistics can be regarded as regularization factors since they penalize solutions where the prior places low density of probability.
Then, given this simple model whose MLE parameters are:
$$
argmax_{\mu} \text{ } \mathcal{N}(y; \mu, \sigma)
$$
and I add a prior:
$$
argmax_{\mu} \text{ } \mathcal{N}(y; \mu, \sigma) \mathcal{N}(\mu; 0, \sigma_0)
$$
the parameters are not the MLE parameters but the MAP parameters. 
Question: Does this mean that if I introduce some regularization in my model I am doing a bayesian analysis (even if only use point-estimates)? 
Or this just makes no sense making this "ontological" distinction at this point since the method to find either MLE or MAP is the same (isn't it?)?
 A: If you use the L2 norm, i.e., quadratic penalty on the log likelihood function, penalization is very similar to a Bayesian procedure with a Gaussian prior with mean zero for the non-intercept regression coefficients.  But unlike the full Bayesian procedure that factors in the uncertainty about the the amount of penalization (analogous to treating the variance of random effects as if it were a known constant), the penalized maximum likelihood procedure pretends that the optimum penalty was pre-specified and is not an unknown parameter.  So it results in confidence limits that are a bit too narrow.
A: It means that the analysis has a Bayesian interpretation, but that doesn't mean that it might not also have a frequentist interpretation as well.  The MAP estimate might be viewed as being a partialy Bayesian approach, with a more complete Bayesian approach being to consider the posterior distribution over the parameters.  It is still a Bayesian approach though, as the definition of probability would be a "degree of plausibility", rather than a long run frequency.
