# Optimal regularization for non linear optimization

What would be the optimal (Bayesian?) solution for fitting a model $f(x)$ to data $h$, given the following assumptions:

• $h$ is a vector with $N$ elements
• $h$ has Gaussian noise with known covariance $\sigma_h I$
• $x$ is known to have an a-priori Gaussian distribution, with covariance $C_x$
• the first- and second-order derivatives of $f(x)$ are known

My understanding is that:

• The maximum likelihood solution is given by minimizing $\sum_{i=1}^N (f(x,i)-h_i)^2$.

• The MAP estimate is given by: $\sum_{i=1}^N \frac{1}{\sigma_c}(f(x,i)-h_i)^2 + \lambda x^t C_x^{-1} x$

Both can be optimized using non-linear optimization techniques such as BFGS.
So the question comes down to:

Is there a way to determine the optimal value for $\lambda$ ?

-

In all similar problems I dealt with there was no "optimal" value for regularisation coefficient.

And now it seems to me that the only solution in your case is using Cross Validation procedure.

-

The proper Bayesian solution would be to treat the regularisation parameter as a nuisance parameter and marginalise it out of the analysis, using an appropriate hyper-prior. In may be possible to do this analytically, which I have found to be reasonably effective, see section 2 of my paper

G. C. Cawley and N. L. C. Talbot, Preventing over-fitting during model selection via Bayesian regularisation of the hyper-parameters, Journal of Machine Learning Research, volume 8, pages 841-861, April 2007 (www)

Alternatively you could choose the regularisation parameter by maximising the Bayesian evidence for the model, which is a common approach in neural networks, see e.g. this paper.

Cross-validation is also a reasonable non-Bayesian approach as Dmitri suggests (+1).

-