3
$\begingroup$

I'm interested in adding basis functions to a Gaussian Process. In particular, following Section 2 of Rasmussen's book, I have $$g(x)=f(x) + h(x)^\top\beta,\qquad f(x)\sim\mathcal{GP}\left(0,k\left(x,x'\right)\right).$$ In the book, $\beta\sim\mathcal N\left(b,B\right)$ and therefore, everything can be nicely derived by completing squares.

Is there a robust way to estimate this specification with a non-negativity constraint on the parameters? In other words, with $\beta\ge 0$?

Improve this question
$\endgroup$
1
  • $\begingroup$ Which equation does this correspond to in the linked book? $\endgroup$
    – user225256
    Commented Apr 3 at 13:21

2 Answers 2

Reset to default
3
+25
$\begingroup$

First, note that they explicitly specify that under the 2.39 formulation of $g(x)=h(x)^T\beta+f(x)$ that the GP only models the residuals. Under this case, there's no sense in placing constraints on $\beta$ just like with any other regression model.

Next, the Gaussian prior $\beta\sim\mathcal{N}(b,B)$ goes together with the 2.40 formulation

$$g(x)\sim\mathcal{GP}\left(h(x)^Tb, k(x,x')+h(x)^TBh(x)\right)$$

In this model, as $\beta$ is Gaussian, a non-negativity constraint means either

  1. Changing the $\beta$ distribution from Gaussian to multivariate folded normal, which is something you should very carefully consider.

  2. You know beyond all doubt that $b$ values are all positive and far from 0, (might be even something like $\forall j:b_j\ge 3$) because then the probability of getting a negative coefficient should be very low and the nonnegativity constraint should have little effect. Just like the previous, this option requires very careful consideration.

In general, I don't understand why one should place such a constraint on regression coefficients (but that's just me), you might want to elaborate this point.

Improve this answer
$\endgroup$
3
  • $\begingroup$ In the special (yet not so rare) case where the trend function $h$ belongs to the RKHS of the Kernel, there is clearly a confusion between the prior on $\beta$ and that on the smooth GP part $f(x)$. The constraint $\beta \geq 0$ then has no clear meaning. $\endgroup$
    – Yves
    Commented Apr 4 at 7:44
  • $\begingroup$ „I don't understand why one should place such a constraint on regression coefficients“ It happens in models with non negative coefficients What are some motivations for using nonnegative least squares? $\endgroup$
    – Sextus Empiricus
    Commented Apr 4 at 8:49
  • $\begingroup$ But the question was not about constrained Bayesian regression using GP, just the simple setting of Rasmussen's book $\endgroup$
    – Spätzle
    Commented Apr 8 at 11:32
2
$\begingroup$

You can use an algorithm for non negative least squares fitting and replace the least squares fitting step with a kriging step.

For an algorithm of non negative least squares see How do Lawson and Hanson solve the unconstrained least squares problem?.

For examples of kriging see Why are Gaussian Processes valid statistical models for time series forecasting?.

In the nnls algorithm you can replace the least squares step with kriging and the least squares cost function with a likelihood function based on a covariance matrix. The algorithm of the link is an active set method that adjusts the active set based on gradients, for this you need to compute an alternative, or the simplest (but most computation intensive) way is to try every possible variable.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.