# Gaussian process box constrained optimization

I am have been trying to optimize the Gaussian process likelihood function (multivariate Gaussian likelihood) in R (optimx and nloptr) while having some box constraints for my hyper parameter estimates.

I am facing the problem of the solution rushing to the edges of the parameter space. Why this happening is? And what is a good optimizer in R that is good for constrained R optimization?

• It depends on your objective function and the values of the bound (box) constraints. How did you determine what values to use for those bounds? The explanation could be that the variables on the bounds wind up being optimal in your problem. Stupid question: Are you maximizing or minimizing the likelihood? Many optimizers have a default or exclusive "setting" (behavior) to only do minimization. If the optimizer only does minimization, you need to multiply the objective value provided to the optimizer by -1, so that it will maximize what you want. So, are you actually maximizing likelihood? – Mark L. Stone Aug 29 '16 at 19:15
• Assuming your optimization is implemented correctly and box constraints are reasonable -- the best approach to solve this problem is to use HMC sampling of model hyperparameters, and averaging predictions over alternative hyperparameter configurations. – Sycorax Aug 29 '16 at 20:09
• There are also purpose-build GP parameter optimizers for R. One is in the DICE family (Deep Inside Computer Experiments). I think there are at least 2 others but I can't recall their names atm. – Sycorax Aug 29 '16 at 20:11
• @GeneralAbrial Thank you for your comment, The HMC sampling is extremely time consuming so sometimes it is not an option in my case. I tried to use teh Dice family however they do not offer to insert your own covariance function. I want to know how to just to use the optimization techniques implemented in Dice in R ? Do you have any experience with that – Wis Aug 30 '16 at 17:08
• @MarkL.Stone I am doing maximization thus I did multiply by -1. thank you for yoru comment – Wis Aug 30 '16 at 17:10

A strong and relatively straightforward routine to use for box-constraints is BOBYQA; it is available in R through the minqa package. It is the default optimiser for the lme4 package when it comes to box-constraints for the evaluation of (generalised) linear mixed models deviances ($-2$ log-likelihoods) so I suspect it will work well with the GP likelihood evaluations too. (I have used for this GP optimisation tasks successfully at some point but I actually found that re-writing my problem in an unconstrained form was more beneficial.)
• @GeneralAbrial: Local minima can be due to functionally identical models being derived by permuting/exchanging terms or input weighting. If the local optimum is not too different from the global one, probably the difference in performance might even go unnoticed. This is quite noticeable for NN for example. For what is worth, from the packages you comment about: gptk uses gradient information within its own CG solver, GPFDA uses nlminb and GPfit and tgp use R's optim (usually with L-BFGS-B); so clearly derivative-based algorithms for GPs are not a priori poor choices... – usεr11852 Aug 30 '16 at 22:39