2
$\begingroup$

What would be the optimal (Bayesian?) solution for fitting a model $f(x)$ to data $h$, given the following assumptions:

  • $h$ is a vector with $N$ elements
  • $h$ has Gaussian noise with known covariance $\sigma_h I$
  • $x$ is known to have an a-priori Gaussian distribution, with covariance $C_x$
  • the first- and second-order derivatives of $f(x)$ are known

My understanding is that:

  • The maximum likelihood solution is given by minimizing $\sum_{i=1}^N (f(x,i)-h_i)^2$.

  • The MAP estimate is given by: $\sum_{i=1}^N \frac{1}{\sigma_c}(f(x,i)-h_i)^2 + \lambda x^t C_x^{-1} x$

Both can be optimized using non-linear optimization techniques such as BFGS.
So the question comes down to:

Is there a way to determine the optimal value for $\lambda$ ?

$\endgroup$
2
$\begingroup$

In all similar problems I dealt with there was no "optimal" value for regularisation coefficient.

And now it seems to me that the only solution in your case is using Cross Validation procedure.

$\endgroup$
2
$\begingroup$

The proper Bayesian solution would be to treat the regularisation parameter as a nuisance parameter and marginalise it out of the analysis, using an appropriate hyper-prior. In may be possible to do this analytically, which I have found to be reasonably effective, see section 2 of my paper

G. C. Cawley and N. L. C. Talbot, Preventing over-fitting during model selection via Bayesian regularisation of the hyper-parameters, Journal of Machine Learning Research, volume 8, pages 841-861, April 2007 (www)

Alternatively you could choose the regularisation parameter by maximising the Bayesian evidence for the model, which is a common approach in neural networks, see e.g. this paper.

Cross-validation is also a reasonable non-Bayesian approach as Dmitri suggests (+1).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.