2
$\begingroup$

I want to apply gradient descent for parameter estimation in a Gaussian Process. The parameters have to be > 0. How can I prevent gradient descent to find parameters that are below zero?

My code for gradient descent (in Octave) looks like this:

function re = get_max(theta_0, epsilon, f1, k, X, y, noise, maxit=100)
  for i = 1:maxit
    direction = epsilon * f1(X, y, k, theta_0, noise);
    if(norm(direction) <= 1e-6)
      re = theta_0;
      return;
    endif
    theta_0 = theta_0 + direction;
  endfor
  re = theta_0;
endfunction

X and y is the training data of the gaussian process and k is the covariance function. theta_0 is the vector of parameters that I want to estimate, epsilon is the step size for gradient descent and f1 is the derivative of the function (here the marginal log likelihood) that should be optimized.

My problem is that often gradient descent reaches parameters that are < 0. But I am only searching for positive parameters. I even cannot continue gradient descent, because calculation of the marginal log likehood or its derivative do not work for parameters < 0 (because of non positive definite covariance matrix).

So is there a good way to prevent gradient descent from finding parameters that are < 0?

$\endgroup$
3
$\begingroup$

Some options:

  1. Exponentiation of the constrained variable will coerce it to be positive for all real inputs.

  2. Don't use gradient descent. Using an optimizer that's built to deal with constraints will respect the constraints. L-BFGS-B is a strong, general-purpose choice.

$\endgroup$
  • 1
    $\begingroup$ Both good options; there's also barrier functions, where a term is added to the objective function that makes it increasingly worse as the parameter approaches the boundary. This is usually not as good an approach as the two suggested by user777, but I mention it more for completeness. There are occasions where it's reasonable and it's had a revival recently because of the connection to interior point methods. $\endgroup$ – Glen_b Nov 24 '15 at 2:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.