Optimal regularization for non linear optimization

Question

What would be the optimal (bayesian?) solution for fitting a model f(x) to data h, given the following assumptions:

• h is a vector with N elements
• h has gaussian noise with known covariance $\sigma_h I$
• x is known to have an a-priori gaussian distribution, with covariance $C_x$
• the first and second order derivatives of f(x) are known

My understanding is that:

• The maximum likelihood solution is given by minimizing $\sum_{i=1}^N (f(x,i)-h_i)^2$.

• The MAP estimate is given by: $\sum_{i=1}^N \frac{1}{\sigma_c}(f(x,i)-h_i)^2 + \lambda x^t C_x^{-1} x$

Both can be optimized using non-linear optimization techniques such as BFGS. So the question comes down to:

Is there a way to determine the optimal value for $\lambda$ ?

Answer 1

In all similar problems I dealt with there was no "optimal" value for regularisation coefficient.

And now it seems to me that the only solution in your case is using Cross Validation procedure.

Answer 2

The proper Bayesian solution would be to treat the regularisation parameter as a nuisance parameter and marginalise it out of the analysis, using an appropriate hyper-prior. In may be possible to do this analytically, which I have found to be reasonably effective, see section 2 of my paper

G. C. Cawley and N. L. C. Talbot, Preventing over-fitting during model selection via Bayesian regularisation of the hyper-parameters, Journal of Machine Learning Research, volume 8, pages 841-861, April 2007 (www)

Alternatively you could choose the regularisation parameter by maximising the Bayesian evidence for the model, which is a common approach in neural networks, see e.g. this paper.

Cross-validation is also a reasonable non-Bayesian approach as Dmitri suggests (+1).