I have been struggling with this for a while. A typical optimisation problem can be viewed as optimising some cost function which is a combination of a data term and a penalty term which encourages certain solutions. And normally there is a weighting term between the two.
In the bayesian setting, this can be interpreted with the usual prior and the likelihood function. In the current problem that I am trying to understand, I model the prior as a multivariate normal with zero mean and the precision matrix equal to $\lambda \Lambda$ where $\lambda$ can be thought of this regularisation weighting and $\Lambda$ is some appropriate precision matrix structure that encodes the plausible solutions somehow. In my particular example, the precision matrix encodes some smoothness constraints on the estimated parameters i.e. the prior encourages smooth solutions. In this case, $\lambda$ denotes the strength of this smoothness penalty term. A $\lambda$ of zero would mean the ML estimate where we only optimise the cost function i.e. the likelihood function. This is because as $\lambda$ decreases, the precision decreases and hence the variance of each of the parameter in the prior increases. So, low values of $\lambda$ will move towards the unregularized solution.
Now, a typical thing I have seen is that there is some sort of an iterative scheme, where we first start with an approximation to $\lambda$ and compute the distribution over the other parameter of interest using some approximate scheme like variational Bayes or Expectation Propagation and then use this approximation to update our estimate of $\lambda$ (assuming priors over $\lambda$ are of the conjugate form, usually done with a Gamma distribution which also keeps it positive).
Now, my question is that if I start with a very low value for $\lambda$ as my approximation, then the prior term would hardly have any effect. Would this not push the estimated distribution towards solutions that are less plausible i.e. basically give high probabilities to unregularized solutions? I am having a lot of trouble understanding how this update scheme can actually find good values for $\lambda$ i.e. finding the value of $\lambda$ that is optimal with respect to the observed data. So, basically what I have trouble understanding it is what is stopping the inference to drive this value of $\lambda$ down to zero or close to zero to prefer the unregularized maximum likelihood estimate? I really do not see how this value of $\lambda$ is being driven by the data or the evidence term.